Health systems are using AI to predict severe Covid-19 cases. But limited data could produce unreliable results

Limited data in Covid-19 AI tools could produce unreliable results Some institutions have not published any results showing whether their models to predict a Covid-19 patient's trajectory actually work. Erin Brodwin STAT

As the United States braces for a bleak winter, hospital systems across the country are ramping up their efforts to develop AI systems to predict how likely their Covid-19 patients are to fall severely ill or even die. Yet most of the efforts are being developed in silos and trained on limited datasets, raising crucial questions about their reliability.

Dozens of institutions and companies — including Stanford, Mount Sinai, and the electronic health records vendors Epic and Cerner — have been working since the spring on models that are essentially designed to do the same thing: crunch large amounts of patient data and turn out a risk score for a patient’s chances of dying or needing a ventilator.

In the months since launching those efforts, though, transparency about the tools, including the data they’re trained on and their impact on patient care, has been mixed. Some institutions have not published any results showing whether their models work. And among those that have published findings, the research has raised concerns about the generalizability of a given model, especially one that is tested and trained only on local data.

A study published this month in Nature Machine Intelligence revealed that a Covid-19 deterioration model successfully deployed in Wuhan, China, yielded results that were no better than a roll of the dice when applied to a sample of patients in New York.

Several of the datasets also fail to include diverse sets of patients, putting some of the models at high risk of contributing to biased and unequal care for Covid-19, which has already taken a disproportionate toll on Black and Indigenous communities and other communities of color. That risk is clear in an ongoing review published in the BMJ: After analyzing dozens of Covid-19 prediction models designed around the world, the authors concluded that all of them were highly susceptible to bias. ...