Fire at FAA tech center

wsuffa

wsuffa

Touchdown! Greaser!
Joined
Feb 22, 2005
Messages
23,615
Location
DC Suburbs
Display Name
Display name:
Bill S.
Apparently there was a fire at FAA tech center at Atlantic City today. Aside from weeaking havoc with data feeds (flightaware has no data) the FAA ATCSCC site is down, and it appears that IFR reroutes are impacted.

I found out the hard way today when ATC put a ground hold on DC area departures + denied all reroutes and had no update time. They got me out with an altitude limit after 35 minutes....
 
There was a transformer fire. I had lunch with an FAA contractor and he had to get back to deal with possible issues.
 
wsuffa said:
Apparently there was a fire at FAA tech center at Atlantic City today. Aside from weeaking havoc with data feeds (flightaware has no data) the FAA ATCSCC site is down, and it appears that IFR reroutes are impacted.

I found out the hard way today when ATC put a ground hold on DC area departures + denied all reroutes and had no update time. They got me out with an altitude limit after 35 minutes....
Click to expand...

I saw Flight Aware's announcement at the top of the page. Fun. Was wondering 'root cause'. Failover systems didn't work?
 
TangoWhiskey said:
I saw Flight Aware's announcement at the top of the page. Fun. Was wondering 'root cause'. Failover systems didn't work?
Click to expand...

Sounds like a single site power outage.

You're assuming they have multi-site redundancy for all systems, critical and non-critical?

I doubt whatever feeds FlightAware is considered mission critical, for example.
 
Hey Troy... your company sells stuff that would have helped them have backups, and I know how to bring whole data-centers online really really fast on a deadline.

Think they'll call either of us? ;)

I feel bad for whoever's cramming stuff in racks somewhere this weekend and restoring stuff from backups. If they even bother.

Last time I had a datacenter drop like that, we got permission from the FD to go get three critical machines, drove them across town, and we were 100% back online in a couple of hours. (Everything else was site-redundant.)

Current employer would be screwed. Can't move telco circuits that fast. Thus, big push to move to 100% SIP. That can be moved. With some pain.
 
Ooh one more point.

Proof positive that confiscating nail clippers at airports is worthless. A very small explosive device will take out a transformer.

Break the infrastructure the system halts or overloads.
 
Interesting to know where that transformer was. When I worked ZBOS, we had our own diesel generators for when commercial power went down. If that transformer was before the generators, then it should have been transparent to switch over to self power.
 
denverpilot said:
Hey Troy... your company sells stuff that would have helped them have backups, and I know how to bring whole data-centers online really really fast on a deadline.

Think they'll call either of us? ;)

I feel bad for whoever's cramming stuff in racks somewhere this weekend and restoring stuff from backups. If they even bother.

Last time I had a datacenter drop like that, we got permission from the FD to go get three critical machines, drove them across town, and we were 100% back online in a couple of hours. (Everything else was site-redundant.)

Current employer would be screwed. Can't move telco circuits that fast. Thus, big push to move to 100% SIP. That can be moved. With some pain.
Click to expand...

Government budgets. Lots of money spent, but probably not in the right places. Now, they'll get reactionary, when planning would have been preventative.
 
BillTIZ said:
Interesting to know where that transformer was. When I worked ZBOS, we had our own diesel generators for when commercial power went down. If that transformer was before the generators, then it should have been transparent to switch over to self power.
Click to expand...

The copper bus bar explosion at the datacenter I know of, was between the transfer switches (city power or diesel genset) and the UPS room.

When the UPS batteries died, down she went. UPS didn't power cooling so everything was heat-soaked in the few hours prior to failure. Drives and machines continued to fail for a year afterward at a higher than normal rate.

A good friend's desk was underneath the conduit that carried the buss bar that exploded. If he'd have stayed late at work that night, they found 1/2" wide balls of melted copper embedded an inch in his desk and his chair was shredded. Those would have been in his head.

(Manufacturing defect, air bubble in the bar that was over an inch thick and about seven inches wide), mixed with water introduced into the conduit by a roof leak that went undetected... Short to conduit, should have just popped protection devices... But instead, flash heated bus bar weakened at the air bubble and superheated the air trapped inside... boom.)

Fire Department pulled the occupancy permit for 48 hours while it was rebuilt. Took three days total.
 
You must log in or register to reply here.
Back
Top