360 deaths or 22,000? Why Minnesota's COVID-19 models are so different

If Minnesotans were still having dinner parties, the hottest topics might be sourdough bread and epidemiological models.

Complex disease models, while perhaps previously a niche interest, are far from abstract these days — they're a major reason many of us are stuck at home and whole sectors of the economy are shut down. And we turn to models because we all want concrete answers about when life can go back to normal.

Unfortunately for those seeking clarity, the models don't seem to agree. At all.

One model, commissioned by Minnesota's Department of Health and designed by University of Minnesota researchers, projects the state could see 22,000 deaths over the next six months even with "stay at home" orders in place into the first week of May.

Another, favored by the White House and produced by the Institute for Health Metrics and Evaluation at the University of Washington, says it expects only 360 deaths statewide.

How can scientists looking at the same data come up with such different estimates? And who's right?

We talked with members of both the Minnesota and Washington modeling teams and other epidemiologists to sort out the science.

Where the models agreeThe Washington and Minnesota models actually agree on some key factors:

• Social distancing: Relaxing social distancing guidelines too soon would lead to a spike in cases and deaths.

• ICU beds and ventilators: Exceeding the state's capacity of ICU beds and ventilators will lead to deaths that could have been avoided.

• Rolling changes: Both are trying to figure out how coronavirus works on the fly, and have made major changes in their projections repeatedly as our understanding of the virus grows.

• Deaths, not cases: Both assume real-world data on deaths is a more useful metric to ground their predictions than estimates of total cases because there hasn't been sufficient testing to measure how many people have actually contracted COVID-19.

So why do the models project such different death totals and ICU demands? There are three big reasons.

DIFFERENCE NO. 1They make different assumptions about when social distancing measures will be relaxedThe current version of the University of Minnesota model includes several different scenarios, each reflecting a different social distancing strategy. None of those scenarios matches exactly what Gov. Tim Walz has implemented with his executive orders, but the closest, (it's "Scenario 4," for those who have been watching the state's briefings)" looks like this:

1. "Stay at Home" order: A 6-week order for Minnesotans to stay home, from March 27 until May 8. (Walz's current "stay at home" period ends May 4.)

2. Extended requirements for those at risk: A continued stay-at-home requirement for the elderly and other "vulnerable" people into August, when the model estimates deaths will be about a month past their peak.

3. Social distancing: Even after the "stay at home" order is lifted, everyone would continue to practice some social distancing through at least May 30.

In this scenario, the model projects that ICU demand could overwhelm hospitals by July:

In all the Minnesota-built models, the vast majority of infections and deaths here occur in a second, much bigger peak in mid-July, after extreme social distancing measures have been relaxed. The curve we're currently seeing, they say, may not be the worst of what Minnesotans can expect from COVID-19.

The Washington model currently projects only a single peak in deaths, a scenario they say is possible as long as governments don't relax social distancing until new infections are so rare (1 per million residents) that public health officials can isolate every single case. Officials would then need to track down everyone they interacted with, known as contact tracing.

The Washington model even projects a date — the week of May 31 — that infections should be low enough that public health workforces "should be able to handle aggressive case finding and contact tracing," said Dr. Theo Vos, an epidemiologist on the Washington team.

Though the Washington model is generally more optimistic than the Minnesota models, Washington actually projects a later date than the state's current assumption of when social distancing would be relaxed — May 8. And if the Washington model was adjusted to relax social distancing too early, a second spike would also appear.

DIFFERENCE NO. 2Minnesota's model assumes we wouldn't lock down againThe potentially catastrophic death tolls projected in the Minnesota model's second peak come with a Big If. These scenarios assume that even in the face of truly horrific daily death totals — 1,000 a day at the top of the curve — that state officials would not re-institute stay-at-home orders.

"Are we going to do that? Maybe, maybe not. We think it's important to emphasize the consequences of not doing that," said Dr. Eva Enns, a researcher at the University of Minnesota who is helping design the model.

Without additional stay-at-home orders or a vaccine, the virus would eventually wane only due to the state reaching a point of herd immunity. And that means that a lot more Minnesotans would have to face the disease.

"The pathway to herd immunity without a vaccine is infection," Enns said. "and it means some people don't do well."

If the governor intervened quickly in case of a renewed spike, the state could flatten the curve again, and potentially avoid many deaths — but waiting until the peak would be too late.

In fact, a series of on-again, off-again stay-at-home policies was the recommended option in an early paper on coronavirus released by scientists at Imperial College, London. The Minnesota team has said their model is based on the Imperial College model.

In this scenario, rather than one giant second peak, the Imperial College paper predicted a series of ongoing, smaller peaks.

(The chart below, from the Imperial College report, shows projections for Great Britain, not Minnesota.)

Though avoiding infections and reducing deaths sounds like an easy call to make, this scenario would prevent Minnesota from reaching a state of herd immunity for a long time, which the Imperial College researchers projected would mean repeatedly disrupting lives and the economy for perhaps 18 months, or until a vaccine is available.

"We want zero deaths, right? But that economic component is important, too, which clearly this model doesn't capture," said Dr. Shalini Kulasingam, another University of Minnesota researcher and model designer.

DIFFERENCE NO. 3The models themselves work totally differentlyWhile the models both are trying to understand how coronavirus works and how it is affected by human behavior, they set off from nearly opposite starting points.

HOW THE MINNESOTA MODEL WORKSThe Minnesota model starts by trying to mathematically replicate how the infection and human interactions with it work. It then check that understanding against what has actually happened so far. Their approach is known as a SEIR model, an acronym that stands for how people in the model are grouped into phases of the disease: Susceptible, Exposed, Infected and Recovered.

To make a reasonable model of how the disease will progress, the modelers must make correct assumptions about:

• How many people an infected person is likely to come into contact with

• How likely it is each one of those people will be infected

• How likely that person is to be hospitalized

• How deadly the disease is for those who catch it

The Minnesota team has touted that their model is the only one with inputs customized to the unique age, racial and health demographics of Minnesotans, and that this should allow them to more accurately make Minnesota-specific projections.

But since the coronavirus is so new, none of those parameters are well-understood by scientists, said Dr. Maria Sundaram, an epidemiologist at Emory University.

"We know a lot more about flu than we do about COVID-19," Dr. Sundaram said, "and it's still very challenging for us to predict what's going to happen with flu."

(Sundaram earned her PhD at the University of Minnesota but is not part of the Minnesota team.)

So the modelers run their simulation many times, each time with a different set of parameters inside what they consider to be a reasonable range. If the virus is more infectious, the state could be in for a steeper, taller peak. But if it's less, the projected second wave wouldn't peak as high, but would take longer to play out. The different simulations predict a wide range of possible death tolls: 9,000 to 36,000, with 22,000 being what the modelers consider the most reasonable estimate.

HOW THE WASHINGTON MODEL WORKSThe Washington model starts by taking what we know about how many deaths an area has had so far, and then tries to compare the number and timing of those deaths to outbreaks in other places.

That assumes that the "shape of the curve" will match roughly with that seen in places like China and Italy, which inherently comes with other assumptions about how cases are tested and treated, whether communities are good at social distancing, and how deaths are counted.

The Washington model is updated often to incorporate new data. And that new data (and some methodological shifts) have led to significant variations in the model's projections.

Vos said the model has gotten increasingly accurate as more data comes in.

"The earlier you are in an epidemic, the harder it is to make sense of what it will look like, because you have really small numbers to work with," he said. "So as we go along, we become more confident about what it will look like."

So which one is better?It depends. Neither are "wrong," and there's a chance neither end up being "right." The words "models are not a crystal ball" were uttered by nearly every epidemiologist we spoke with, sometimes more than once.

"We can't know which one, if either will be right, because that's predicting the future which we just cannot do as humans," Sundaram said. "We can just make an educated guess."

Still, both modeling teams defended their chosen approach and had concerns about others.

The Minnesota team joins a large cadre of scientists who believe the Washington model is too optimistic.

"I don't know if it's really all that insightful," said Stefan Gildemeister, Minnesota's State Health Economist, who is supervising the effort for the Minnesota Department of Health. "The notion that they are assuming deep isolation, that there's testing capability that just doesn't exist at the moment, and they're kind of ignoring the fact that the virus will continue to perpetuate, is just not very helpful."

Gildemeister said the virus isn't likely to go quietly, and that even Minnesota's intense social distancing and targeting of outbreaks at long-term care facilities hasn't stopped it.

"To assume one more month and we would be scot-free is just not realistic," Gildemeister said. "I hope folks are really thoughtful before they open up on the basis of the [Washington] evidence. Lives are at stake. To say we're done with a curve in various jurisdictions, that is incredibly dangerous."

Sundaram, the Emory epidemiologist, said the Washington model could be considered a best-case scenario.

"That maybe is a strong assumption to make, that we do everything right the first time around, or that everything goes exactly the way that we expected it would go," Sundaram said.

Sundaram said the Washington model is right that contact tracing would be needed for containment of a second wave without the help of a vaccine, but that many places in the U.S. aren't there yet.

Vos, with the Washington team, said he wasn't familiar with the Minnesota model specifically, but said that generally, traditional SEIR models have not performed well during this pandemic.

"Those models can produce very large numbers," Vos said. "You've seen a number of examples predicting that within no time, millions and millions of deaths would occur. That has not happened."

To Vos, starting with theory — assumptions that are "pluck[ed] out of the air," as he put it — when so much about coronavirus is murky — could lead to "wild" projections. Vos said the Washington model's close ties to observed data make it more reliable.

"I think we're lower than most of those models, but we're much closer to actual observations," he said. "I think we're making better estimates than any of the other prediction models I've seen around."

Gildemeister said the Minnesota model is more sophisticated than a traditional SEIR model, and has been specifically designed to avoid out-of-control overprediction.

And despite his reservations about the accuracy of SEIR modeling, Vos said the team at Washington is working to turn their model into a hybrid that would add a SEIR model to the curve-fitting, tightly tied to the number of actual deaths observed, in order to project out farther into the future.

While the Washington team aims to update their projections three times a week, it has taken much longer for each version of the Minnesota model to be released. That means that in the short-term, the Minnesota model can appear to be out of sync with actual deaths and ICU demand.

That's a bad sign, Vos said.

"I would say it's much easier to predict in the short term than it is in the long term," Vos said. "So if a model does poorly at predicting in the short term, then I would have much less faith in the long term predictions."

Sundaram said SEIR models can be "computationally intensive" to update, so it's understandable that the Minnesota team has taken longer so far. But speed is not the only consideration, she said. "There's absolutely a trade off between having an answer quickly and having an answer that's the most accurate possible answer."

The update frequency is also related to the different goals each modeling team is pursuing, Gildemeister said. The Minnesota model is designed primarily to inform state officials about which combinations of social distancing measures would have the best chance of saving lives, rather than making precise predictions about when to expect different milestones, he said.

And the Minnesota modelers have expressed concern that frequent updates imply to casual observers that the Washington model's predictions have always been spot on, imparting a false sense of precision.

Recent updates to the Washington model have included significant changes in its Minnesota projections. Earlier this week, the model indicated that Minnesota was nearly a week past its peak, with Wednesday's update saying that peak would come today. The date for when the state might be able to relax social distancing has also been pushed later in each recent update.

So which modelers are on the right track? Sundaram said the answer is "it depends."

"And that's such an unsatisfying answer," Sundaram said. "If you were to ask any epidemiologist what the most common response to any question is, they would say 'it depends.'"

In the short term, that means the best way to look at models is to understand the assumptions and limitations each has.

And either way, Gildemeister said it's just too early to know.

"You don't grade halfway through a test," he said. "and I feel we're not even a third of the way into this test."