Blue Cross Blue Shield of Minnesota is working rapidly to shore up its cybersecurity defenses after an internal whistleblower raised alarm that the state's largest health insurer had long neglected thousands of important updates.

Internal documents show that Minnesota Blue Cross allowed 200,000 vulnerabilities classified as "critical" or "severe" to linger for years on its computer systems, despite stark warnings to executives. Software patches were available to fix most of the weak points.

The top cybersecurity executive at Minnesota Blue Cross says the insurer has been working diligently in recent weeks to bring the number of security vulnerabilities as low as possible by year's end.

"We certainly understand that our members expect us to protect their most sensitive data, and we want them to know that we are committed every single day to doing just that," Minnesota Blue Cross Chief Information Security Officer Amy Eklund said in an e-mailed statement.

Minnesota Blue Cross insures 2.8 million people, including about 1 million outside Minnesota, and brings in $6.7 billion in annual revenue. Its computer systems contain members' demographic information, medical billing codes and financial records — prized data for identity thieves and other cybercriminals.

Pam Dixon, executive director of the World Privacy Forum, a consumer-rights group, said an insurer harboring many thousands of vulnerabilities on its computers is enough to make an IT expert "break out in a cold sweat."

"The speed and the level of sophistication at which the attackers are operating today is extraordinary," Dixon said. "It is a foolish person who is running security at a large-scale organization with a lot of PHI, personal health information, without absolutely up-to-date, pristinely managed technology."

Minnesota Blue Cross has never reported a data breach of its own systems. In 2015 the personal data of 11,000 members of Minnesota's Supervalu Group Health Plan were breached after Minnesota Blue Cross stored their information on vulnerable computers owned by another Blue Cross licensee, now called Anthem Inc.

Attackers have breached more health care records across the country in 2019, 40.8 million so far, than in the previous three years combined. Most exploit weaknesses that could have been repaired with available software patches, but weren't.

At Minnesota Blue Cross, documents obtained by the Star Tribune show that cybersecurity engineer Tom Yardic met with executives as early as August 2018 to raise alarm that important patches weren't getting done. On Sept. 16 Yardic e-mailed the board of trustees in what the e-mail describes as a last-ditch effort to push for change.

"I am sending this e-mail because I have been unable to impact the situation within the avenues the organization provides," Yardic wrote to the trustees and CEO Dr. Craig Samitt. Although the seriousness of the situation had been acknowledged in meetings going back over a year, Yardic wrote, "what has not happened is a serious attempt to remedy the situation."

Scans of the Minnesota Blue Cross network show the number of software vulnerabilities classified as critical or severe peaked at around 200,000 inside roughly 2,000 important computers called servers, according to records obtained by the Star Tribune and confirmed by the insurer. At least 89,000 of those vulnerabilities were more than three years old as of the end of last year, and some 24,000 dated to 2010 or earlier.

There were an additional 2 million vulnerabilities on Minnesota Blue Cross' 6,000 employee workstations, in part because IT staff had deployed thousands of machines that contained hundreds of unpatched vulnerabilities apiece, Blue Cross documents show.

In some cases, the same security flaw may be counted hundreds of times because it's on hundreds of machines.

Minnesota Blue Cross did not dispute the accuracy of the number of past vulnerabilities. But a spokesman said the current totals are lower — much lower in the case of workstations.

Eklund declined to reveal exactly where the "managed volume" of vulnerabilities stands today. Responding to written questions, she also said it would be "misleading" to suggest that the raw number of vulnerabilities provides a full picture of overall risk.

"Protecting our members' information is our top priority, and our efforts are ongoing," Minnesota Blue Cross officials said via e-mail. "As with all companies holding sensitive information, we remain vigilant in our security systems and testing, but we will always strive to do more."

Patching is important

There are many ways to protect vulnerable computers connected to the internet, and Minnesota Blue Cross uses many of them. But cybersecurity consultants and engineers say it's unusual to avoid the most basic step, which is to regularly install software patches, especially critical ones.

"I don't know of anyone who would say that patching isn't important," said Ryan Elmer, a Minneapolis-based technology risk manager at accounting and consulting firm Boulay. "It's like a dentist telling you not to brush your teeth."

A software patch is a piece of computer code that rewrites part of an older program to fix a security vulnerability or improve performance. Since new vulnerabilities are constantly discovered, installing patches is an ongoing job at large companies. Last year companies took an average 34 days to install the most serious ones, classified "critical" patches, and 38 days for less-severe patches, according to an analysis by cybersecurity firm Rapid7.

Unpatched computers can be vulnerable to "ransomware" attacks, in which a hacker turns an organization's information into gibberish until the victim pays a ransom. Unpatched systems can also leak sensitive data to the dark web, by allowing identity thieves to create fake user accounts on a network and export sensitive data. Attacks may compromise a single employee's workstation or can spread "laterally" across an entire network, even reaching into servers containing massive databases.

Such risks are not abstract — Blue Cross affiliates in California and Idaho reported breaches of health data just this year.

The largest-ever health data breach happened at the Blues plan in Indiana now known as Anthem Inc. Forensic analysis concluded that an attacker penetrated Anthem's network through a "phishing" e-mail to an employee in Virginia, causing the loss of nearly 79 million Anthem health records over an 11-month period in 2014 and 2015. The second-worst U.S. health breach, of 11 million records from Premera Blue Cross in Washington, also hit in 2015. A federal audit found the insurer had failed to fix known problems, including not installing software patches.

In September, Yardic told trustees that Minnesota Blue Cross risked something similar.

"Today we have approximately 2,000 servers containing confidential information that are missing a large number of critical security updates, many for several years," he wrote. "Like Premera Blue Cross, who was recently penalized for not protecting member data, we have not 'installed software updates and security patches on a timely basis' or in many cases, at all."

Larry Ponemon, who founded independent IT research firm the Ponemon Institute 17 years ago, said many companies don't patch vulnerabilities because the work is time-consuming and often complex. Patches must be tested to make sure they don't create new problems. Installing new software may require taking important computers offline.

"It takes a lot of effort. So companies just don't patch," Ponemon said. "It happens all the time."

Yet most data breaches are preventable with patches. In a survey of 2,900 IT professionals by Ponemon Institute last year, 60% of respondents said the data breaches at their companies could have happened because of a known vulnerability for which the patch was not installed.

Blues plans nationally are licensed by the Blue Cross Blue Shield Association, which says it maintains a broad security program requiring its members to meet "cybersecurity related standards and policies." The national association didn't release the specific requirements, including any requirements for handling security-related patches.

U.S. Sen. Mark Warner, D-Va., a tech investor who co-founded the Senate Cybersecurity Caucus in 2016, said many health care organizations struggle to balance patient care with the need to invest in cyber-preparedness activities.

"Health organizations should work to minimize vulnerabilities by keeping software up to date, constantly scanning for weaknesses across their entire IT infrastructure, and patching vulnerabilities as soon as they are detected," Warner said in a statement to the Star Tribune.

A push from the top

Insurers such as Minnesota Blue Cross are covered by the federal health care privacy law known as HIPAA, which requires covered entities to "identify and protect against reasonably anticipated threats" to the security or integrity of patients' electronic health information.

The law does not require organizations to install every software patch. However, it does require HIPAA-covered organizations to mitigate risks from unpatched vulnerabilities, either by installing the patch or establishing other compensating controls, like restricting network access or disabling network services that could be exploited remotely, federal officials said last year.

Minnesota Blue Cross officials say their servers undergo rigorous "penetration testing" on a quarterly basis, and the company's network is protected by many layers of security to prevent and detect intrusions.

"We have invested heavily in our security program, which comprises both prevention and detection capabilities," a company statement said. "These capabilities are supported by advanced detection [tools], third party testing, and 24/7 monitoring."

Minnesota Blue Cross switched to a new vulnerability scanning tool last year for its server network. Blue Cross documents show that during the rollout, the implementation consultant noted, "something might be wrong, these numbers seem really high" as he was looking at the volume of vulnerabilities.

Yardic's Sept. 16 e-mail to Minnesota Blue Cross' trustees said the company was failing to take reasonable steps to protect its members' information, because of what he saw as "a long-standing cultural indifference to computer and network security."

"It will take a sustained push from the top to permanently change this culture," he wrote.

Three months later, Eklund, the top IT security officer at Minnesota Blue Cross, said in her statement that the insurer had a strong focus on its volume of vulnerabilities:

"Through ongoing focus, collaborative efforts and opportunity afforded by migration and upgrade projects, our managed volume continues to decrease and should be considerably reduced by the end of the year."