CenturyLink weaknesses faulted in Minnesota 911 outage

After a major outage in August that led to almost 700 dropped 911 calls in Minnesota, regulators have found weaknesses in CenturyLink's administration of the state's emergency call system.

While the outage was caused by human error at a CenturyLink vendor, state safety officials and the public were not notified in a timely manner that the system had partly failed, according to a report by the Minnesota departments of Commerce and Public Safety.

"No automatic alarms were activated and sent to [emergency call centers] by this service disruption," the report said. Most emergency call centers weren't formally notified the 65-minute 911 outage had taken place until after it was over.

The outage, the report said, "indicates that CenturyLink did not provide safe and adequate service."

Monroe, La.-based CenturyLink, the largest landline phone company in Minnesota, administers the state's private 911 network, which delivers calls to all 97 of Minnesota's emergency call centers. CenturyLink is in the third year of a five-year $29 million contract with the state.

CenturyLink said in a statement that it "places a high priority on public safety, and as a result we have taken several steps to address this issue." Those steps include software updates, additional monitoring and alarms, "enhanced call testing" and procedures to "isolate problems quicker."

The public-safety department said it received no reports of adverse outcomes because of the 911 interruption on Aug. 1. Still, it was a landmark of sorts.

Dana Wahlberg, director of the department's emergency-communications division, noted in an e-mail that the outage "was the most widely impacted 911 service disruption" in Minnesota that she has experienced in her 30 years in the business.

The departments are recommending that the Minnesota Public Utilities Commission (PUC) beef up CenturyLink's reporting requirements for the Minnesota 911 system. The PUC is expected to consider those recommendations Thursday.

The PUC commissioned an investigation three weeks after the widespread outage, which affected most of Minnesota.

CenturyLink has contracts with Colorado-based West Safety Services to fulfill its 911 obligations. Every 911 call in Minnesota routes through West's "emergency communications management centers" in either Englewood, Colo., or Miami — primarily the latter. From there, the calls zip to 911 centers back in Minnesota.

The two routers are intended to be redundant hubs in diverse geographic locations. On Aug 1., a technician at West made a mistake while doing maintenance, disrupting service at the Miami hub.

The system's redundancy then partly failed. While 356 calls to 911 in Minnesota were successfully routed through West's Colorado hub, another 693 calls were dropped before they could get to an emergency call center.

Neither CenturyLink nor the public-safety department were aware of the work at West "since it was erroneously considered to be routine and not service affecting," the state report said. Maintenance work by West also resulted in two smaller 911 failures in Minnesota last year — on July 9 and Aug. 21, which together led to 90 dropped calls.

The 911 outage on Aug. 1, which affected at least three-quarters of Minnesota's emergency call centers, began at 3:47 p.m. The state report said CenturyLink confirmed that it learned of the issue about 11 minutes later. But the state's emergency call centers didn't get official notification until 4:59 — more than an hour later and after the outage had ended.

CenturyLink said it sent an alert to all emergency call centers at 4:24 p.m. The company said in a PUC filing that it was not aware of the full scope of the outage until it was resolved.

In a filing with the PUC, CenturyLink said West made software improvements last fall to prevent the type of error that caused the August outage.

West also implemented "enhanced alarming" for 911 calls that don't go through.

The public-safety and commerce departments were irked at CenturyLink's initial response to the outage.

CenturyLink did not provide a statement that the Department of Public Safety could release to the public until 2:20 p.m. on Aug. 2, even though the state had asked for updates before a 1:30 news conference — "an unacceptable and unnecessary delay," according to the report.

CenturyLink in the PUC filing agreed "this is an area that needs improvement, and [the company] has taken steps to that end." Century­Link said in a statement that it's meeting regularly with emergency-response groups in Minnesota to make sure "we are quickly addressing any concerns."

Minnesota has since been affected by another CenturyLink-related 911 interruption. On Dec. 27, the company experienced a nationwide outage that affected all sorts of calls, including an unknown number of 911 calls.

That outage was not on a particular state 911 system. Instead, it occurred on CenturyLink' "transport network," essentially a transmission line carrying calls from several carriers. The Federal Communications Commission (FCC) is investigating the outage.

In 2015, the FCC slapped CenturyLink with a $16 million fine for a 911 failure in seven states — including Minnesota — that lasted for six hours and led to 6,600 missed emergency calls. Verizon and a forerunner company to West Safety Services were also respectively fined $3.4 million and $1.4 million for that outage.

Mike Hughlett • 612-673-7003