Seattle City Data Center under Full Power
Seattle’s Emergency Operations Center (EOC) was activated yesterday, Friday, August 15th, for a downtown Seattle power emergency – several banks of transformers failed at City Light’s Union Street substation, one of two substations serving the downtown core. City Light (Seattle’s municipal electric utility – a department of City Government) shut down power to some portions of the waterfront, and asked downtown buildings to significantly reduce their electrical use in order to avoid a complete failure of the downtown grid. All this on one of the hottest days of the year (95 degrees) for Seattle.
There was no exensive power failure, so the headline “City Light Avoids 90 Degree Outage” in the Seattle PI was buried in the local section, and the problem was just a footnote in the Seattle Times.
How do City government information technology workers respond to emergencies like this? We’ve had lots of practice – WTO riots, Nisqually earthquake, electrical vault fires, windstorms, actual cyberattacks – and we intentionally conduct emergency operations drills both as a Department of Information Technology (DoIT) and as part of City-wide or region-wide drills such as Soundshake.
Yesterday we went through our well-drilled disaster response: directors and managers alerted all employees. On our own – even before the EOC was activated – we sent desktop and server technical staff plus telephone, data communication and radio system technicians to the EOC to prepare it for activation. The EOC was activated using a DoIT-maintained “community notification” or telephone call-out system to all critical City government executives. When the EOC was activated, we sent an executive there to support the City’s leadership in making crucial decisions about the event. (See also my previous blog entry about the City’s new EOC facility.)
Because this was a power emergency – and because our data centers are a major consumer of downtown power – we activated a long-standing protocol to reduce power consumption. All servers and equipment in the data center are color coded based on their importance to government operations. We shut those systems down in an orderly fashion as rapidly as possible. We also have uninterruptible power supplies and a one-megawatt backup generator for the main data center, plus many other backup generators for critical technology services throughout the City of Seattle.
Yesterday was a hot day in Seattle. It didn’t get hotter, thanks to well-practiced disaster drills and pre-planning!
the Chief Seattle Geek blog 