How a data system glitch spiraled into a holiday crisis for Sun Country

Early Tuesday morning, more than 30 hours after Sun Country Airlines lost connection to a key data center, and with no sign of the link coming back, a technology manager told top executives that an immediate software upgrade would solve everything.

But there was a catch. Upgrading software in the middle of the day would require Sun Country to cancel more flights, just a day after the data outage forced cancellations and delays that added up to the worst disruption at the airline in years.

"I had a lot of questions about that, a lot of concerns," Greg Mays, Sun Country's chief operating officer, said in a lengthy interview near the end of one of the most difficult weeks in the Minneapolis-based airline's recent history.

The holiday season coming to a close this weekend has been a troubling one for the nation's airlines. The rapid spread of the latest coronavirus variant has sickened plane crews and other staff, leading to flight cancellations of around 7 to 8% of U.S. airlines' passenger capacity every day since Christmas. Before the COVID-19 pandemic, about 2% of flights were canceled on a typical day.

Sun Country's woes were even bigger — 16% of its passenger flights Monday were canceled — and stemmed from a pedestrian technology problem rather than omicron infections. Nearly 2,000 people were scheduled to be on those flights.

The airline offered to refund tickets, provide $200 vouchers for future travel and accept applications for additional expense reimbursement, such as for hotels and meals.

Still, the day turned nightmarish as the cancellations, which happened on the first set of flights Monday morning, created a cascade of delays across its entire schedule of 90 flights. Displaced passengers flooded Sun Country's customer service website and call center. Dozens took to Twitter and other social media to complain.

"I'm never going to fly Sun Country again," Cory Williams of Woodbury said Friday. He first spoke with the Star Tribune on Tuesday to describe how the airline's actions led his family to cancel a weeklong trip to Arizona.

Though Sun Country has grabbed headlines for customer service problems in the past — infamously during a late winter storm in 2018 when it left 250 passengers stranded in Mexico — the airline rarely cancels flights. Its completion factor, a measure of scheduled flights that are finished, is one of the highest in the airline industry at 99.9%.

"This pains us anytime we have to cancel flights," said Mays, who joined the company in 2019 after two decades in operations roles at Delta and Alaska airlines. "I've never seen anything like this here."

With still no access to the data center Tuesday morning and no clarity when it would be recovered, the question facing Sun Country executives was whether to stage a repeat of Monday's cancellations in hopes that updated software running in a different data center would bring everything back to normal.

"Typically, you'd do this on a quiet. early Saturday morning in an off-peak period of the year," said Robert Mann, an independent airline analyst in Port Washington, N.Y. "But you simply cannot run the airline manually. My guess is their options were slim and none."

The problem began Sunday night, at the end of the most heavily booked day of the three-week holiday period for Sun Country. While the last few flights of the day were in the air, the airline's crew management system was down.

Over the past 30 years, crew management systems evolved to do far more than what's suggested by their name and original purpose of scheduling pilots and flight attendants. The systems are now the operational backbone for many airlines, tracking planes, ensuring crew schedules comply with government restrictions and even monitoring pilots' exposure to cosmic radiation.

After holding virtual meetings every two hours all night long, Mays left home at 3 a.m. to join a handful of operations managers at headquarters. The company's top officers, Chief Executive Jude Bricker and President Dave Davis, were on holiday trips and joined on meetings remotely over the next two days.

About that same time, Scott and Mary Baumgartner of Rogers left home to catch a 5:30 a.m. Sun Country flight at Minneapolis-St. Paul International Airport to Fort Myers, Fla.

When they got through check-in and security, they noticed none of the Sun Country flights had gate assignments.

Snow was falling, the roads had been slick, and the couple thought weather might affect their trip. "I thought there might be something bigger going on," Scott Baumgartner said. But when he checked his smartphone, "everything looked good."

Sun Country learned from the provider of the crew management system — Dubai-based AIMS International — that its main data center was having network difficulties. Sun Country dealt with a similar outage involving AIMS in July.

Without access to the data on its AIMS system, the handful of people in Sun Country headquarters would not be able to verify that the half-dozen people in each crew assigned to the first round of flights were properly rested and scheduled. There was no choice but to cancel.

"Our passengers assume safety. They can only do that because we don't," Mays said. "If it's not right, I tell our teams to stop the operation and I'll have their back."

As more people arrived at the office, they directly checked crews about work hours. The airline worked with AIMS to access Sun Country's database on a single computer. A camera was then put in front of that computer's screen, so information on it could be seen via videochat by other people making decisions on the airline's operations.

Outside, snow continued to fall. Over in Terminal 2, passengers scrambled for new flight options or simply went home. Three Sun Country charter flights carrying the University of Minnesota Marching Band to Phoenix for the Guaranteed Rate Bowl took off on time, but many other scheduled flights were delayed.

The Baumgartners used a smartphone app to find a flight on American Airlines, but to Tampa instead of Fort Myers and with a stop in Washington, D.C. They arrived at 9 p.m. Minnesota time, and their son-in-law picked them up for a two-hour drive to Cape Coral.

"We lost a whole day of visiting our grandkids," Scott Baumgartner said.

By 8 a.m. Tuesday, Mays said he knew the airline could not endure another day with such limited access to data. Sun Country's IT staff planned to make the software upgrade to the AIMS system at a quiet time in the first quarter and had done some prep work.

"Ultimately, we made the decision to make that transition," Mays said. "We had our first pushout [of flights], and we set a final decision for 10:30."

Over the next two hours, Mays talked to executives at airlines in Spain and Mexico that were also AIMS customers, who explained how they got through the upgrade successfully. Two new backup systems were prepared — one simply paper sheets tacked on a wall denoting each aircraft by its tail number — and staffers quickly pored over training materials that showed the new look of the AIMS software.

At 10:45, Mays agreed that the update should begin at 11. Cancellation notices went out to passengers on 14 afternoon flights, seven at MSP and seven returning to MSP.

Another nearly 2,000 people's holiday vacations were being interrupted. At least this time, executives hoped, passengers would get the news before they started for airports.

"We handled this the absolute best we could in the circumstances. We'll see how our passengers judge us, and that'll occur over time," Mays said.

The upgrade worked, but Sun Country will eventually review its work with AIMS, Mays said. This weekend, subzero weather in the Twin Cities seemed more likely to be the main challenge to on-time performance. Sunday will be second-busiest day of the holidays for the carrier.

"I want to make sure our local community here knows how seriously we take our commitment to our passengers," Mays said. "We don't get it right all the time. Our first thought is safety. But we always want to do our best for passengers. We're very proud to be the hometown airline."

On Friday, some of the displaced passengers reported getting refunds. Others, like Williams, said they received partial refunds and were seeking more information.

Some, like Scott and Mary Baumgartner, were still waiting. The couple are scheduled to fly home Monday from Fort Myers on Sun Country.