server outage, 7/8/21
Posted: 09 Jul 2021, 15:39
Most of you may not have noticed but I know Softball/Staark and Gator did (while getting Softball prepped to join us in DCS!).
The datacenter where the servers are hosted had a power outage yesterday starting at about 8PM Pacific. It was quite an outage in a business area, and the generators did not kick in like they should have (personally I think when adding a 3rd generator recently they messed things up and did not properly test).
I was the first one in my company to know about it thanks to Softball!
We started working immediately to assess the situation and plan the bring-up once power came back online. Power came back online about 10PM and myself and another starting working remotely to evaluate the damage while a 3rd waiting with an angry mob outside the data center door to get in (they did not let anyone in as all cabinet doors were open due to the chilled water plant being offline and too warm to have closed cabinets). We did have everything up successfully and running mostly by 1AM with some issue systems and switches taking the rest of the night to repair/bring up. Needless to say I worked all night with no sleep until it was time to prep for the Naval Retirement Ceremony I was invited to today.
The datacenter where the servers are hosted had a power outage yesterday starting at about 8PM Pacific. It was quite an outage in a business area, and the generators did not kick in like they should have (personally I think when adding a 3rd generator recently they messed things up and did not properly test).
I was the first one in my company to know about it thanks to Softball!
We started working immediately to assess the situation and plan the bring-up once power came back online. Power came back online about 10PM and myself and another starting working remotely to evaluate the damage while a 3rd waiting with an angry mob outside the data center door to get in (they did not let anyone in as all cabinet doors were open due to the chilled water plant being offline and too warm to have closed cabinets). We did have everything up successfully and running mostly by 1AM with some issue systems and switches taking the rest of the night to repair/bring up. Needless to say I worked all night with no sleep until it was time to prep for the Naval Retirement Ceremony I was invited to today.