From assessing a patient's cancer risk to setting criminal sentences, AI and algorithms are increasingly used to make life-altering decisions in health care and law. But how can we ensure that these algorithms are designed to remove the biases inherent to human decision-making, rather than reproducing them?
This is a question Emma Pierson, an assistant professor of computer science at UC Berkeley, grapples with every day.
"Algorithms have a lot of potential advantages over people for being fair decision-makers, but those advantages have not always been realized - and it can be quite hard to realize them," Pierson said.
In her work, Pierson is developing AI and machine learning methods for medicine and the social sciences, with the goal of improving health care and reducing social inequality. Recently, she developed a new test for detecting racial discrimination in policing. She is also working to reduce racial and gender bias in algorithms, particularly those that are used to assess disease risk and make critical decisions about patient care.
"I am trying to use AI to make a healthier and fairer world," Pierson said.
Pierson was recently named the Zhang Family Endowed Professor and is affiliated with the Berkeley AI Research Lab, Computational Precision Health and the Center for Human-Compatible AI.
People over particles
Pierson was an undergraduate studying physics at Stanford University when she learned that her mother, who had been diagnosed with breast cancer a few years earlier, was a carrier of the BRCA1 gene mutation. Pierson had a 50% chance of also having the mutation, which confers a high risk of breast and ovarian cancers. Genetic testing a few years later confirmed she was a carrier.
"It was very scary and upsetting, but also quite focusing in terms of finding what I wanted to do with my life," Pierson said.
Pierson, who had been using statistical methods to study galaxies, already had the notion that she was "more interested in people than particles." She upended her studies, deciding to focus instead on how she could apply computer models to the fields of health care and the social sciences.
Soon after beginning a graduate program in computer science, also at Stanford, Pierson began working on police traffic stop data collected as part of the Stanford Open Policing Project. The data revealed undeniable racial disparities, galvanizing Pierson to focus her work not only on improving health care, but also on identifying and addressing racial inequalities.
"If you looked at the very basic statistics of the data - the racial disparities in how likely people were to be searched or arrested after a stop - the gaps were huge," Pierson said. "It was apparent that something catastrophic was going on."
However, her work on police traffic stop data also highlighted the importance of designing precise statistical tools for understanding the how and why of inequalities. In a study published late last year, Pierson and her collaborator Nora Gera, a graduate student at Cornell University, devised a new way to test whether inequities are caused by discrimination rather than other factors.
I am trying to use AI to make a healthier and fairer world.
Emma Pierson
In traffic stop data from Texas, Colorado and Arizona, they noticed that there were a number of individuals who police reported as white in some encounters and Hispanic in others. In the study, they compared how these individuals were treated when they were perceived as white versus Hispanic, finding that drivers were significantly more likely to be searched and arrested when the police perceived them as Hispanic.
The findings provide strong evidence that racial bias was likely a cause of the disparities.
"The reason we looked at white and Hispanic drivers is because those were the most commonly confused racial groups in the particular data that we looked at, but there's no reason you couldn't apply it to other categories," Pierson said.
This same approach could be applied in other situations where a person's racial or gender category is determined by how they are perceived, rather than being self-reported, and where there are multiple perceptions of the same person over time.
"For instance, you could look at the same person over time in different health care settings when their race is inconsistently perceived and see how they're treated differently," Pierson said.
Counteracting human biases
Using data to pinpoint the root causes of racial and gender disparities is just the first step in addressing these gaps. Pierson is also working to design AI algorithms that can help counteract the biases inherent in human thinking.
But creating and implementing fair and unbiased algorithms in the real world comes with a host of challenges. Like humans, algorithms trained on biased data will also make biased decisions. In addition, many of the algorithms used in health care and law are created by corporations and are proprietary, meaning it is difficult for the public to scrutinize or critique how they were designed and trained.
"If an algorithm is unfair, it can also reproduce unfairness on a much vaster scale than any single human decision maker," Pierson said.
However, Pierson points to a New York Times article by MIT professor Sendhil Mullainathan, which argues that "discrimination by algorithm can be more readily discovered and more easily fixed" than discrimination by humans. Experiments can be designed to reveal bias even in proprietary algorithms, and researchers like Pierson are finding ways to address these biases once they have been uncovered.
Recently, Pierson has been studying how to design algorithms that will perform well in the presence of missing data. This question is especially pertinent in medicine, where algorithms that predict disease risk often omit many important factors - such as a patient's genetic history or exposure to pollutants - because these factors are unknown. Instead, the race of the patient is often used as a variable.
"Race is a socially constructed variable, not biological," Pierson said. "And historically, race has been included in medical decision-making and in algorithms in racist ways. By using race in these algorithms, we might worry about further entrenching health disparities."
If an algorithm is unfair, it can also reproduce unfairness on a much vaster scale than any single human decision maker.
Emma Pierson
However, simply removing race from medical algorithms doesn't necessarily help reverse medical disparities. In a 2024 paper, Pierson and her team found that removing race from a cancer risk prediction algorithm caused it to under-predict risk for Black patients, potentially reducing access to colorectal cancer screens. And in a survey study published last month in JAMA Internal Medicine, Pierson and her collaborators found that many patients are comfortable including race in medical algorithms, as long as doctors are transparent about how the information is being used.
"I think it's fair to say that in a world where we had access to all data for all people and everything was perfectly equitable, we would not need to use race in algorithms," Pierson said. "But it is also important to consider the world we are actually in and design algorithms that help us make the best decisions for patients today with the data we actually have."
Pierson says that doctors decided to test her mother for the BRCA1 gene because she was Ashkenazi Jewish, an ethnicity 10 times more likely than the general population to carry these cancer-causing mutations. Without the testing, Pierson's mother might not have lived to see her children grow up, and Pierson might not have learned this critical information about her medical risk.
"My experience [with the BRCA1 mutation] has really crystallized for me that these risk tools are not just abstractions or interesting objects of study, but things that concretely affect people's fundamental health and life decisions," Pierson said. "They are vitally important to get right."
