N/A A day in the life...

AuntPeggy

Final Approach
PoA Supporter
Joined
May 23, 2006
Messages
8,479
Location
Oklahoma
Display Name

Display name:
Namaste
This morning, the radio warned of an accident on the parkway just beyond our exit that was backing up traffic in both directions. Alerted, we watched for the expected traffic jam and were quite relieved to speed along normally to our exit. However, an additional report indicated that "Hope Street in Stamford is shut down due to a car hitting a pole."

Our office is just one block off Hope Street. Although we saw traffic diverting down a side street, we continued on and drove along Hope Street wondering how the radio could conclude the street was closed by someone hitting a pole. Then, when we got to the turnoff to work, the traffic light hung blackly over the intersection. Ah, someone hit a POWER pole. Another block further, red, blue, and white strobe lights flashed randomly from police cruisers as cars were diverted and directed away from their intended destinations.

Gratefully, we glided into a parking spot and braved the cold through doors that were mysteriously propped open into a formerly secure lobby -- dark and cool. No elevators, so we trudged up to the third floor and after a couple of attempts found a security door that would let us enter our offices.

UPS' cried for attention from the server room. UPS' chirped and bleeped from under desktops in cubicles. My UPS and my computers were dark and silent. People had gathered in the kitchen to stare at the non-functioning coffee maker. I poured a cup of tepid tap water and drowned a teabag in it. A co-worker reported that the electrical flushing mechanism in the bathroom wouldn't... Let's not go there.

"I can't do anything without a computer," my friend said. "Let's get some coffee down the street." The traffic light had been working just a half-mile before our turnoff, at the strip mall where "Dunkin Donuts" was. I grabbed my jacket and headed toward the door just as flourescent lights blinked, flickered, and then lit the rooms. UPS' fell silent and, with a mighty groan, the heating system came to life.

One of the IT guys went past us to caress keyboards in the server room. Its going to be a long day for him.

People trickled back to their offices and life appeared to go back to normal. There was a email from a remote worker asking us to find her server and turn it on. It wasn't until after noon that someone remembered that her server's labeled name was about 2 years out of date, found it and got it started for her.

"I smell smoke," someone said. The faint odor permiated the floor, coming from the heating vents. "It's just the heater starting up," another said. The smell dissipated and once again things went back to normal.

By 9:30 we started to notice the heat. My little fan simply blew hot air around. People were threatening to take off their shirts. The usually frigid lab and server rooms were becoming over-populated. After fiddling with my desk clock, it began counting up the degrees, starting at 89. I wasn't able to get it into the hot part of the cubicle where my feet dangled amid three computers. To get some respite, we grabbed coats and walked through freezing drizzle to the cafeteria for lunch.

When we returned, the cubicle temperature was 91. I sent an email to the building engineers. At 93, I phoned. The IT guys, in sweat-soaked t-shirts were still bringing up servers and trying to accommodate the growing crowds in the labs. There has been the sound of boots tramping on the roof for over an hour and the temperature is 94.5. I have taken off my shoes and wool sox, put on sandals, and rolled up my corduroy pants to my knees. The wet paper towel I use to wipe down my arms, face, and neck keeps drying out.
 
What are you doing messing around with computers, you shoulda' been a writer.

That's where the real money is....
 
The Vice President of Engineering, USA just came zipping around the offices and told our managers to send us home. Meanwhile, guys in hard hats and workboots have been tramping around pointing at ventilation returns. The temperature has dropped to 92.
 
Looks like almost the same calamity happened to Rackspace in Dallas. They didn't lose power but they lost cooling.

http://www.datacenterknowledge.com/archives/2007/Nov/13/additional_details_on_rackspace_outages.html

The datacenters are not my area. I don't know that I've ever discussed keeping the cooling working, but we have diesel generator backup in the big ones which I guess will keep everything cool and powered up.

What's scary here is they would have needed somebody to make the decision to stay on generator power as the utility power came back - intermittently.
 
When I came in on Monday morning, the building engineer was fiddling with one of the thermostats. Later in the day when he was fiddling with another one, I asked him about the heat and response. He told me two interesting things.

First, the building management made the decision that he should calibrate all the thermostats in the building. Until that time, they just had the initial factory settings. He was calibrating the sensors to the actual temperature at the position of the thermostat.

Second, when the power came back up on Friday, the compressor(s?) burned out. As a safety feature, the HVAC system defaults to "heat" when the compressor is not working, so the heaters just turned on full blast and didn't cycle.

So, if your data center is working under the same scheme, a power failure that takes out the air compressor could cause severe damage to your servers by overheating them. We had a separate circuit and separate air-conditioning service for the servers, which is why they were in a cool room while everyone else was sweltering.
 
Another potential data-center glitch is water pressure. If your AC uses cooling towers filled with water, they may not be able to keep the water levels correct if the electrical outage results in lost water pressure. Plenty of power in the data center thanks to generators, but no AC.

During the electrical outage in the eastern us a couple of years ago, our Detroit data center could have run for days on it's backup generator. The city couldn't supply water though, so all we got out of the generators was an orderly shutdown.
 
Another potential data-center glitch is water pressure. If your AC uses cooling towers filled with water, they may not be able to keep the water levels correct if the electrical outage results in lost water pressure. Plenty of power in the data center thanks to generators, but no AC.

During the electrical outage in the eastern us a couple of years ago, our Detroit data center could have run for days on it's backup generator. The city couldn't supply water though, so all we got out of the generators was an orderly shutdown.

Yeahbut, the chiller water is recirculated. I know it gets cooled in the open air so a lot must be last to evaporation to the outside world, but I wonder how much city water has to added back in a day.
 
We had a similar weekend of fun here. On Sunday there was snow all day right around freezing temperature, which meant that the snow fell and landed more as ice. Roads were a whole lot of fun to drive on. Visibility was at bare minimums for IFR flight, even. My instructor flew his Navajo, but he was the only person to land at IPT on Sunday.

Yesterday morning I drove to work and found a tree in the middle of the road on my backroad, meaning I had to turn around and backtrack almost back to my house (5 miles or so) and take the alternate route. Roads were garbage until I hit the highway, at which point they just turned to mostly garbage. :)

Power went out for 10 hours yesterday. Good reason to take the Navajo (still at IFR minimums) to fly my fiancee home. Got home 5 minutes after the power came back on - my stove was blinking 12:08.

Yay for winter! :)
 
Back
Top