Racial Bias in Mortality Prediction Scores

The COVID-19 pandemic has forced us to face many uncomfortable realities. As critical care physicians, my colleagues and I work in intensive care units (ICUs) that have grown to accommodate the large numbers of patients who have become seriously ill from COVID-19 during the past year. However, the specter of scarcity has been looming. We know that a day may come when there are not enough ICU beds or ventilators for all the patients who need them. In such a situation, how will hospitals fairly choose who gets access to these scarce resources, knowing that the patients who are turned away may die?

Ethicists have developed crisis standards of care (CSCs) to answer exactly this question during pandemics like COVID-19 or other health crises like natural disasters or mass casualty situations. CSCs generally promote two main ethical goals: 1) save the most lives, and 2) do so fairly. To achieve the first goal, CSCs suggest prioritizing patients with the best chances of survival to receive scarce resources. Most CSCs operationalize this by using a mortality prediction model, the Sequential Organ Failure Assessment (SOFA) score, to rank patients based on their predicted probability of surviving their hospital stay. However, whether using the SOFA score allows us to achieve the second goal—allocate resources fairly—is not known. Therefore, against the backdrop of a pandemic that has exposed the structural disadvantages experienced by racial and ethnic minorities, my co-authors and I set out to determine whether the SOFA score is equally good at predicting mortality among Black and white patients.

We found that the SOFA score is “miscalibrated” and racially biased. Specifically, it overestimates mortality among Black patients and underestimates mortality among white patients. In other words, using the SOFA score would lead us to believe that Black patients are sicker or more likely to die than they actually are. This matters because, within the framework of CSCs, patients who are most likely to die are last in line to receive scarce resources. Therefore, using the SOFA score could systematically and unfairly divert scarce critical care resources away from Black patients. In a rudimentary simulation, we estimated how many Black patients would be affected by this miscalibration. We found that 81% of Black patients from lower priority CSC categories, and 9% of all Black patients, would be improperly excluded from the highest priority CSC category. In our paper, we also test the accuracy of other scores, and share general principles to consider when choosing a mortality prediction model to include in CSCs.

This work has clear and urgent policy implications. We must develop equitable models that accurately predict mortality among all patients lest we exacerbate the unacceptable racial disparities that have become so evident during the COVID-19 pandemic. Thankfully, although many hospitals continue to operate on the brink of scarcity, none have had to deploy their CSCs yet. We can hope that day will not come during this pandemic or future health crises, while also working to ensure that we are prepared to make fair decisions if it does.

The study, Equitably Allocating Resources During Crises: Racial Differences in Mortality Prediction Models, was published online in the American Journal of Respiratory and Critical Care Medicine on March 21, 2021.  Authors include Deepshikha Charan Ashana, George L. Anesi, Vincent X. Liu, Gabriel J. Escobar, Christopher Chesley, Nwamaka D. Eneanya, Gary E. Weissman, William Dwight Miller, Michael O. Harhay, and Scott D. Halpern.