108 points mbac32768 1 hour ago 43 comments

hackermailman 50 minutes ago | parent

In the article he talks about living within a few minutes walk to work and telling too many people so they made him the emergency office guy. Same thing happened to me, foolishly told everyone hey I can see my apartment from here so I would get panicked calls to run in and shut down servers because the information security guy got a tip of an impending exploit about to be released on a holiday weekend with no way to do it remotely. Everyone else lived a ferry ride or commuter train away.

NooneAtAll5 21 minutes ago | parent

did you at least ask for pay increase?

tombert 10 minutes ago | parent

Not the person you asked, but I've learned that if you're not careful about what you tell your employers and/or coworkers that it will almost invariably come back to haunt you in the form of unpaid labor.

It took me an embarrassingly long time to realize that (my 30's), and now I keep my conversations with my coworkers very utilitarian and divulge minimal information about myself. I try not to be a dick or anything, I'm polite enough, but pretty much everything I talk about is either directly relevant to work and/or so surface level as to border on "vapid".

rshnotsecure 46 minutes ago | parent

Surprised they even had sprinklers set up. Historically, things that avoid moisture like Halon Gas [1] have been used.

[1] - https://en.wikipedia.org/wiki/Bromotrifluoromethane

mbac32768 41 minutes ago | parent

Other datacenters we built had them (which scared the shit out of me tbh). I don't remember why this one had to stay on sprinklers.

dklsf 38 minutes ago | parent

Maybe because it was in a normal office building? The gas extinguishers might be too hazardous there

hvb2 40 minutes ago | parent

Or CO2 but that's a life or death situation if there's any people still inside

thechao 39 minutes ago | parent

Halon is not exactly fun to breathe, either.

wolvoleo 32 minutes ago | parent

Especially after it's been in contact with fire, the resulting products are even worse.

But it's only used in military and aviation these days anyway.

dklsf 39 minutes ago | parent

If those go off, they end up destroying hard drives (the spinning kind), due to the sound pressure....

So fire suppression and data centers don't mix too well

shagie 33 minutes ago | parent

Ahh... the Vax War (bit hard to search for it now... gotta search for the first line). VAXen, my children, just don't belong some places.

https://www.hack.org/~mc/texts/vaxen-joke.txt

Realizing that I'm spoiling some of it... but we're here, so some of it is inherently spoiled...

           The fire alarm klaxon went off and the siren warning
           of imminent halon gas release was screaming.  We
           started to panic but the data center manager shouted
           over the din, "Don't worry, the halon system failed
           its acceptance test last week. It's disabled and
           nothing will happen."
 
           He was half right, the primary halon system indeed
           failed to discharge. But the secondary halon system
           observed that the primary had conked and instantly
           did its duty, which was to deal with Dire Disasters.
           It had twice the capacity and six times the discharge
           rate.

wolvoleo 29 minutes ago | parent

They don't have to blow that fast. It can be regulated.

But yes, I've seen low-O2 installations also. These have other issues though, like no protection when humans need to be in there for maintenance. And the need to be very very careful when reducing it again, making sure everyone is out. It happened to my colleague once that this went wrong and while it's not low enough that you'll die, he wasn't exactly feeling great for the next few hours.

gosub100 38 minutes ago | parent

Probably not safe to use on a floor that has people on it.

wolvoleo 37 minutes ago | parent

We had it at work at a datacenter right beside offices and reception.

You just have to reinforce the ceiling panels and the windows so they don't get blown out, but it's not really an issue otherwise. It was all certified and regularly checked. Water is indeed a bit counterproductive as it's not really much better than fire for the equipment in there.

PS: Halon itself is of course gone but it's other less harmful gases these days.

What also happens is low-oxygen environments (they only raise the O2 when maintenance is being done).

advisedwang 26 minutes ago | parent

Likey codes or building management wouldn't let them make the necessary modifications to safely install a Halon system

garciasn 16 minutes ago | parent

Halon systems are banned, at the international and federal level, and have been since the 1990s; so, modern DCs don't and can't use them.

wolvoleo 38 minutes ago | parent

How is it the only datacenter?

33 Thomas St is basically one huge datacenter, that's why it doesn't have any windows. https://en.wikipedia.org/wiki/33_Thomas_Street

dklsf 36 minutes ago | parent

This was pre-Hurricane Sandy, so before 2012. I'm guessing data centers have grown since then

lelandfe 36 minutes ago | parent

Work nearby, it looks just as evil in person.

kccqzy 30 minutes ago | parent

Jane Street’s only NYC data center. Not the only data center in general.

advisedwang 27 minutes ago | parent

It's Jane Street's only datacenter. not the only one in NYC. As they explain, they chose to specially locate their datacenter.

maronato 27 minutes ago | parent

The headline is confusing. It’s Jane Street’s only data center in NYC, not NYC’s only data center.

reticulates 36 minutes ago | parent

The article reminds me of when Hurricane Sandy took out a bunch of New York DCs with 3+ feet of water and people were carrying diesel up stairs to keep generators running.

“The superhuman folks at the data center are hauling 55 gallon drums of diesel fuel up 17 flights of stairs. Assuming they're able to sustain a rate of transfer that exceeds the rate of consumption, the data center should run until electricity is restored. The latest estimate from Con Edison for lower Manhattan is 4 days. (The New York Stock Exchange will be open tomorrow, also running on generator power.)”

https://news.ycombinator.com/item?id=4720388 (https://web.archive.org/web/20121101194314/http://status.fog...)

https://arstechnica.com/information-technology/2012/11/how-o...

skinfaxi 32 minutes ago | parent

Diesel is ~7lbs/gallon. That is truly superhuman damn.

altcognito 10 minutes ago | parent

385 lbs, with a dolly, not too bad.

Forgeties79 8 minutes ago | parent

>17 flights of stairs

HighGoldstein 28 minutes ago | parent

I wonder how well these "superhuman folks" were paid

timacles 26 minutes ago | parent

It was a bunch of quants so very well

bigbinary 15 minutes ago | parent

I’d haul 55 gallon drums of diesel for measly 1.5m a year

Forgeties79 8 minutes ago | parent

You’ll need that money for the back surgery

mvcosta91 26 minutes ago | parent

TIL: wabi-sabi. I’ve been trying to find a way to express the beauty of imperfection in the age of AI, and this fits perfectly.

kridsdale1 19 minutes ago | parent

It’s extremely common and cliche for people to use that term for imperfect stuff these days.

philipwhiuk 20 minutes ago | parent

Almost too close to 'Pool on the roof must have a leak' to be real ;)

rdl 18 minutes ago | parent

Odd that they didn't have at minimum 1) a physical Emergency Power Off (EPO) button inside the datacenter, such that one button could de-energize everything -- usually required as part of building a datacenter due to code variance for the "low voltage" (<4000V in this context...) wiring. 2) EPO interlink to the smoke alarms, such that it would de-energize equipment automatically (maybe with delay) if smoke detected.

arjgoel30 13 minutes ago | parent

Wow crazy I didn't know they had such simple systems

defmetrix 13 minutes ago | parent

He earned his bonus that year.

sciurus 11 minutes ago | parent

I wonder why the answer was to turn off the servers instead of turn their cooling back on. Would it not be able to stabilize the temperature fast enough?

demillort 10 minutes ago | parent

Man, I wish MOC were still alive. I’d love to know his take on this story.

LPisGood 9 minutes ago | parent

I am shocked to hear they do not use foam.

lostmsu 9 minutes ago | parent

Alarm firmware wasn't programmed in OCaml

phplovesong 7 minutes ago | parent

Is this the the same jane street that did the indian manipulation scam that stole 4 billion from invostor? That jane street?

stephbook 4 minutes ago | parent

> a false smoke alarm, a datacenter with no cooling, and a sprint to pull every power cord before the sprinklers went off

I smell AI