Complete Failure

Fairfax County says Verizon “failed completely” during June 29 derecho.

— The eerie silence is what Steve Souder, Fairfax County’s 911 director, remembers about the night of June 29, when the derecho hit Fairfax County.

“The derecho was fierce and sudden, a lot of things most storms aren’t,” Souder said in an interview on Friday. “We were busy as all get-out.”

The fast-moving storms slammed Fairfax County at about 10:20 p.m., resulting in a record number of emergency calls — a 415 percent jump for 911 dispatchers and a 2,000 percent jump for Fire and Rescue. Approximately 30 minutes later, power flickered and then died in the McConnell Public Safety and Transportation Operations Center (MPSTOC), where 46 911-dispatchers were fielding hundreds of calls.

“We can’t afford to be crippled, so our generators came on, and we’re rocking and rollin’ in the blink of an eye,” Souder said.

At 1:30 a.m., the first wave of calls started to subside.

“We’re taking a deep breath and looking ahead to sunrise, when we fully expected another wave of calls as people wake up and take a look around their homes at the damage,” Souder said.

THE NEXT MORNING, as hundreds of thousands of county residents awoke to smashed cars, split fences and downed power lines from uprooted trees, the county’s 911 operators braced themselves for another onslaught of calls.

But the next wave never came.

“The phones just stopped ringing. We were dead in the water … Never, ever, ever has this happened to us,” said Souder, who has been in emergency operations for 44 years.

It would be several more hours before Souder and other county officials learned that no calls were getting through due to a problem with Verizon. According to Souder, all calls to Fairfax County’s 911 go through the Verizon network, regardless of the commercial carrier service. From 7:36 a.m. until 3 p.m. on June 30, 911 service was completely down and for the next three days service was sporadic.

Souder said signs of trouble with Verizon came around 7 a.m., when Verizon sent a cryptic email to Fairfax County staff saying that the Arlington central office was without power or backup battery/generator. The references to Arlington, according to Souder, suggested that 911 service was affected only in Arlington County, so Fairfax County’s 911 staff continued with their normal operations.

“We are completely unaware that incoming 911 call service from Verizon is slowly dying … We don’t get officially notified until about 10 hours after [the] whole mess begins, which is completely unacceptable,” Souder said, adding:

“It’s like the captain of the Titanic telling passengers the ship hit an iceberg when the bow is on the bottom of the ocean floor.”

As a result of this critical outage, Fairfax County submitted official comments to the Federal Communications Commission (FCC) detailing what happened, suggestions for Verizon to improve its service and exhibits to show supporting documentation and actions.

“During and after a storm, and in any emergency or disaster, the loss of the public’s ability to contact emergency responders is most profoundly felt,” according to County officials. “Families in darkened homes crushed by fallen trees, motorists unable to get through roadways blocked by downed electric power lines, elderly residents in care facilities without power in temperatures over 90 degrees, and any other citizens in need of emergency services must be able to call 911 to seek assistance.”

“Fairfax County is, and will stay, on top of this,” said Fairfax County Board of Supervisors Chairman Sharon Bulova (D-at large). The county’s response to the FCC, released last Thursday, lays the blame for the outage squarely at the feet of Verizon, and Verizon acknowledged some responsibility in its Aug. 13 report to the MWCOG.

After first denying it had major problems with Arlington County’s 911 service, Verizon officials admitted they did not know 911 emergency service was out in Fairfax County until alerted by county officials.

In the Verizon report, officials said two major generator failures — one in Fairfax and one in the central Arlington office, which routes 911 calls to multiple centers — caused “multiple failures cascading from these specific generator problems.”

Verizon also reported that it lost visibility over its own network, so that technicians did not know that power was draining from battery-operated generators until it was too late.

“When the batteries in the Arlington central office were being depleted … why was there a delay in deploying additional resources to remedy the situation before the Arlington facility went dark?” asked county officials, in concluding statements in the FCC report. “Verizon’s 911 service in Fairfax County failed completely during the June 29, 2012, derecho. Both immediate and longer-term changes are needed to improve 911 service in the metropolitan Washington, D.C., area.”

THE CURRENT FCC REPORT is one of several investigations launched into the 911 fiasco. According to officials, Fairfax County is pursuing the issue from many angles, including the report to the FCC and regional work through the Metropolitan Washington Council of Government.

“We are very fortunate that no one died as a result of the 911 outage,” said Supervisor Pat Herrity, (R-Springfield). Herrity is a member of the statewide 911 panel tasked with looking into the outage.

“I have every confidence that Fairfax County is up to whatever emergency comes our way — in as much as we have control. I’m more worried about third parties such as Verizon based on our 911 experience. Frankly, that was wholly unacceptable and we are going to do everything necessary to hold their feet to the fire,” said Supervisor Jeff McKay (D-Lee).

Souder said it’s gratifying to know Verizon is taking some responsibility, and attempting to fix its communication problems.

“The public should know that, seven weeks after this storm, we’re still a long way from being done from making sure this [911 failure] doesn’t happen again,” Souder said.