Accuracy of COVID-19 tests coming to market uncertain

U.S. regulators have opened the floodgates for millions of COVID-19 test kits to be used on Americans in coming months, but there is little evidence yet that the tests work well and no evidence that one works better than another.

Tests that deliver false results can worsen the spread of disease and cloud researchers' understanding of its viral properties. A Star Tribune review of hundreds of pages of regulatory filings from the first 30 rapid tests to detect COVID-19 found that most test makers are doing the bare minimum amount of validation work before putting their wares in clinicians' hands.

Manufacturers are rushing tests to market because of nationwide test shortages and the belief that widespread testing is necessary to help identify those infected with COVID-19 and their contacts. Such use of testing has been crucial in countries that have had the most success limiting outbreaks and the toll.

Mayo Clinic researchers warned Thursday that inaccurate test results may drive a "second wave" of infections involving people who spread the virus after a test falsely indicates they don't have it.

Asked how well the new tests work, Mayo Clinic internist and biostatistician Dr. Colin West said: "The most honest answer is, we don't really know. At least outside of some very controlled lab environments, the data on how many false negatives or false positives these tests might yield has not been disseminated."

Joanne Bartkus, director of the Minnesota Department of Health's public lab, said differences in accuracy between testing platforms have stood out, in some cases. The state recently stopped using the MagNA Pure LC platform and the QIAcube platform because of concerns about accuracy rates, she said. Retesting with another platform found the MagNA Pure LC missed about 3% of positive samples, she said.

The lack of any definitive, widely available test to diagnose COVID-19 has caused states including Minnesota to impose sweeping orders to shut down schools and workplaces, because it's not possible to know who needs to be quarantined. Manufacturers have rushed new tests into production, and the Food and Drug Administration is allowing them onto the market with minimal oversight.

Most of the 30 or so lab-based rapid-detection tests that detect genetic traces of the virus were validated using 30 "contrived" samples of the novel coronavirus that causes COVID-19 — not a clinical trial with hundreds of specimens from real patients, federal filings show. Even less information is available about the tests that look for the presence of antibodies in blood samples.

"This is the Wild West right now," said Michael Osterholm, director of the Center for Infectious Disease Research and Policy at the University of Minnesota. "Everybody is focused on how many tests will be out there. No one is really focusing on quality. ... We need to have the right data, not just more data."

Some of these tests are already in widespread production and use.

Mesa Biotech in San Diego delivered its first 13,000 COVID-19 test kits for its palm-sized Accula system last week, while Illinois' Abbott Laboratories is shipping 50,000 tests per day for its toaster-sized ID Now system. Both tests are triggered by the presence of genetic material unique to the virus that causes COVID-19.

Eagan in-home urgent care provider LX Medical is offering a different kind of COVID-19 test, made jointly by Becton Dickinson (BD) and BioMedomics. That test looks for specific antibodies in a blood sample after an infection has been present for a number of days. The manufacturers are planning to ship a million antibody tests in the next six weeks.

Federal records show that the Mesa and Abbott Labs point-of-care tests were validated by scoring 100% accuracy on 30 positive samples and 30 negative samples. Each system has a validated "limit of detection" that requires fewer than 200 genetic copies to be in the sample, after amplification. The filings aren't required to disclose any estimates of true diagnostic accuracy right now.

Sonia Lecce, a senior regulatory consultant with medical research and consulting firm NAMSA, said FDA "is stressing complete transparency" as companies show their tests reach the expected result in 19 of 20 positive samples, among other requirements.

As of Thursday, the FDA had issued 35 emergency-use authorizations (EUAs) to the makers of these tests, most of which are molecular tests restricted to use in clinical labs, not point-of-care systems. The FDA said that its accelerated review process for COVID-19 tests reflects the agency's desire to balance "the benefit of increasing testing capacity with the risk of less certainty."

"Because we are under a unique situation now, what the FDA is doing with the EUA is kind of putting it in the manufacturers' hands to be completely transparent and to assure the validity of the test," Lecce said.

For rapid blood tests that deliver results at the point of care by detecting antibodies, there is no requirement to have the FDA review the test data before marketing the device, if the manufacturer meets certain requirements.

The BD-BioMedomics test has no public data listed on the FDA website, and no decision has been made on whether to seek an EUA for it.

A company spokesman pointed to a February article in the Journal of Medical Virology that reported results from 525 blood tests performed in China, where the virus was first detected.

The article said the test had about 89% sensitivity in detecting samples with antibodies, and about 91% specificity in not reacting to samples without antibodies.

"BioMedomics has validated the test and filed it with FDA, which meets the requirements under the March 16 guidance, and we are able to ship tests to customers," BD spokesman Troy Kirkpatrick said in an e-mail.

The BioMedomics kit is among a wave of more than 70 serology tests for COVID-19 antibodies moving quickly into the market with minimal FDA oversight. The FDA issued a warning last week that some serology test makers were falsely claiming they had FDA approval, or claiming their tests can diagnose COVID-19 when they can't.

Dr. James Riggen, a certified physician assistant and co-founder of LX Medical, said the state of the evidence today about the COVID-19 tests means the test result is just one piece of information to consider when deciding whether someone has it.

The tests "are meant to be tools. There can be false negatives and false positives with all these tests, and you need to be able to interpret these results in the context of the patient's medical history," Riggen said.

False-positive results can happen if the test detects genetics of a virus similar to but different from the one that causes COVID-19. They're driven largely by how the test was designed to detect snippets of genetic code. The risk in a false positive is a strict quarantine imposed on someone who is not contagious.

False negatives can happen if the sample was taken incorrectly, or if the specimen and testing chemicals weren't stored properly, or if the specimen was collected before the virus was detectable.

On Thursday, a paper published in the medical journal Mayo Clinic Proceedings said false-negative test results could trigger a second wave of cases if people with false negatives relax social distancing and hand-washing rules, or try to go back to work. A front-line clinician with false test results could expose patients and other clinicians to the virus.

In Spain, where more than 15,000 people have died after getting COVID-19, authorities recently returned thousands of test kits to their Chinese manufacturer after it was found that the tests were correctly detecting positive cases only 30% of the time, Business Insider reported.

"We are aware of the Spanish situation," Dr. Timothy Stenzel, director of in vitro diagnostics at the FDA, said during an April 1 conference call.

"I would urge those that buy these tests to do some sort of verification of the performance. That's not required, but I think it's only wise."

Stenzel said the FDA is in the early days of inviting antibody test manufacturers to undergo a voluntary validation program through a third party.

"Once that program is up and running and we are able to verify the performance that is stated in their package inserts we will make that information publicly known," he said.

Joe Carlson • 612-673-4779