In the criminal justice system, decisions about when and how long to detain people have historically been made by other people, like judges and parole boards. But that process is changing: Decision-makers increasingly include artificial intelligence systems in a variety of tasks, from predicting crime to analyzing DNA to recommending prison sentences. The use of AI in these domains raises pressing questions about how these computing systems use data to make predictions and recommendations, as well as larger questions about how to safeguard fairness in an AI age.
Notably, many AI systems are "black boxes," which means their behavior and decision-making processes are opaque to scrutiny. This poses a problem in the justice system, in which public trust and the accountability of key players like judges are tied to an understanding of how and why life-changing decisions are made. In addition, even if a black box system is statistically fair and accurate, it may not meet standards of procedural fairness required by our constitutional system.
In April 2024, the National Institute of Justice (NIJ) issued a public request for information that could help inform future guidelines on safe and effective ways to use AI in the criminal justice system. The Computing Research Association — a large organization focused on innovative computing research related to timely challenges — responded by convening a team of experts from academic institutions and industry to crystallize a comment to submit to the NIJ. SFI Professor Cris Moore and External Professor Stephanie Forrest (Arizona State University) were among the submission's authors. The group's argument was clear: Where constitutional rights are at stake, critical decisions shouldn't be made using AI with hidden processes.
"The idea that an opaque system — which neither defendants, nor their attorneys, nor their judges understand — could play a role in major decisions about a person's liberty is repugnant to our individualized justice system," the authors noted. "An opaque system is an accuser the defendant cannot face; a witness they cannot cross-examine, presenting evidence they cannot contest."
This August, the group followed up with an opinion published in the Communications of the ACM. While the original Executive Order 14110 that prompted the NIJ's query has been rescinded, a new Executive Order 13859 calls for safe testing of AI and to "foster public trust and confidence in AI technologies and protect civil liberties, privacy, and American values in their application."
In a criminal-justice setting, AI technologies would only fit this bill if they improve both the fairness and transparency of the current system, says Moore. This is part of what makes AI appealing. Human decision-making processes, after all, aren't always transparent either.
"We should use AI if it makes the judicial system more transparent and accountable," Moore says. "If it doesn't, we shouldn't use it."
He and his collaborators submitted their remarks to the NIJ in May, 2024. They highlighted key arguments that the Justice Department should consider as it develops and implements new guidelines about the fair and beneficial use of AI in sentencing and other cases. Many of those arguments emphasized the need for transparency: everyone who either uses AI or is affected by an AI-produced recommendation should have a clear understanding of the data it used, and how it came up with its recommendations or risk scores. In addition, the experts advised, the procedure by which a judge uses guidance from an AI system should be clear.
Some researchers have warned that increasing transparency can reduce the usefulness of an AI system, but in the last few years, researchers in the field of "explainable AI" have developed approaches that help illuminate how these models process information and produce inputs.
Explainable AI systems may help, but Moore notes that there is a range of ways to define transparency. Transparency doesn't have to mean that everyone understands the computer code and mathematics under the hood of a neural network. It could mean understanding what data were used, and how. He points to the Fair Credit Reporting Act (FCRA), which requires credit-rating companies to disclose consumer information used to make credit decisions and set ratings. The companies can keep their process hidden, says Moore, but a consumer can easily download the information used in the algorithm. It also gives consumers the right to contest those data if they're not accurate. On the other hand, he points out that the FCRA doesn't let consumers question whether the algorithm is doing the right thing with their data. "It's important to be able to look at an AI's inner workings, not just its inputs and outputs," he says.
In addition to recommendations about transparency, the researchers advised that output from AI systems should be specific and quantitative — reporting a "7% probability of rearrest for a violent felony," for example, rather than describing a suspect with a label like "high risk." Qualitative labels, Moore says, leave too much room for misinterpretation.
"If the judge understands what the system's output means, including what kinds of mistakes they can make, then I think they can be useful tools," Moore says. "Not as replacements for judges, but to provide an average or baseline recommendation."
Critically, the authors warned that AI systems should never completely replace human decision-makers, especially in cases where detention and the constitutional rights of a person are at stake. In the optimal scenario, AI systems might become a kind of digital consultant that produces output taken into consideration by a judge or other decision-maker, along with other factors related to the case. "But we should always be prepared to explain an AI's recommendation, and to question how it was produced," says Moore.
