UNIVERSITY PARK, Pa. — Critical decisions are increasingly being made by artificial intelligence (AI) systems in high-stakes industries like healthcare, criminal justice and finance, ushering in a wave of ethical and legal concerns over how these systems can be implemented fairly. Now, a new study by a Penn State researcher and collaborators proposes a holistic theory of AI fairness while providing an applicable framework that organizations can use to manage this complicated issue.
According to co-author Jing Tian , assistant professor of information systems at Penn State's Smeal College of Business, fairness in AI-automated decision making is an evolving issue that must consider many different stakeholder perspectives, including balancing equitable services with economic realities. The design theory proposed in the study, published in the journal Management Information Systems Quarterly , addresses these conflicting concerns through regular human-AI collaboration, treating AI fairness as a continuously managed problem similar to cybersecurity.
In the following Q&A, Tian discussed the complexities of improving AI fairness and how the proposed design theory — titled Fairness Adaptation through AI-augmented Responsiveness, or FAIR — can provide a flexible framework to address these issues across a wide range of industries.
Q: How are organizations using AI systems to make automated decisions?
Tian: Individuals and organizations are using AI systems to make automated decisions in many industries, such as triaging patients in an emergency room setting and evaluating mortgage applications. These tools can make decisions faster and more consistently, but they can also reproduce the inequalities and biases found in their training data. Algorithms used in healthcare have shown racial disparities, for example, and hiring systems have disadvantaged women. When software influences who receives medical treatment or qualifies for a loan, we need to ask not only whether it is accurate, but also whether its decisions are fair.
Q: In your recent paper, how do you frame the issue of AI fairness?
Tian: My colleagues and I argue that AI fairness is not a problem organizations can fix once and forget. Fairness means different things to different people, and these diverse perspectives often conflict in ways that have no simple solution. We frame AI fairness as a "sociotechnical paradox" — a set of persistent, interconnected and evolving tensions that must be managed actively as they develop.
We identify three underlying sources of these tensions. First, different conceptions of fairness can conflict. For example, if an organization were to treat everyone equally, that may not produce equitable outcomes for historically disadvantaged groups like women or people of color. Second, fairness can conflict with organizational goals, including predictive accuracy of the AI systems, operational efficiency and profitability. Finally, what is fair for an individual may not be fair for a group — or society as a whole — and vice versa. These competing demands create difficult trade-offs and situations where every available option has a downside.
Consider an AI system that prioritizes patients in a hospital emergency department. A physician might favor applying the same criteria to everyone, but a patient advocate might prioritize underserved communities. A hospital administrator, meanwhile, might focus on treating as many patients as possible. These goals are all legitimate, yet they frequently pull in opposite directions. These tensions go beyond biased data and into how AI systems are designed. As regulations, organizational priorities and even societal expectations continue to evolve, these tensions will resurface, requiring ongoing adaptation rather than a one-time fix.
Q: FAIR proposes a two-part solution. How does the first part — a system for finding and resolving fairness issues — work?
Tian: We break each AI decision-making system into three connected layers: the data it uses, the model that generates predictions and the rules that translate those predictions into decisions. Organizations can use this framework to identify where fairness problems arise and make targeted improvements rather than treat the entire system as a black box whose inner workings are abstract and intangible.
At each layer, teams can manage fairness tensions through a continuous adaptive cycle of surfacing, or identifying, problems and resolving them. The collaboration between AI agents and humans is a crucial part of this process, combining AI's speed and scale with human judgment and accountability. During surfacing, AI agents monitor the system and flag emerging disparities, while human experts interpret these findings and determine which issues require attention. AI agents then propose and test possible improvements, while humans evaluate the trade-offs and approve appropriate changes.
Q: The second part of the solution sets standards for addressing AI-fairness problems. What would such a system look like?
Tian: In order to manage fairness across an entire organization's AI systems, we propose shared standards with room for local judgment. At the firm-wide level, a central governance team would establish baseline requirements for fairness monitoring, documentation, accountability and human approval. The individual teams responsible for specific AI applications could then adapt these processes to the decisions they handle.
An important principle is that the higher the stakes, the greater the need for human oversight. For example, organizations may allow an AI tool that recommends products to make routine adjustments with minimal human review. A system that helps determine who receives medical treatments, however, should operate under much stricter oversight. Organizations should also seek feedback from employees, users and other stakeholders so these standards remain responsive to real-world experiences.
Q: What would it take for an organization to successfully implement FAIR in their AI systems?
Tian: When putting this kind of approach into practice, an organization must start with a shift in mindset. Just as they cannot address cybersecurity or safety concerns with a one-time inspection, organizations cannot treat AI fairness as a box to check at launch. To improve fairness in their AI systems, organizations need ongoing monitoring, clear responsibility for responding to problems and a process for learning as technology and expectations change.
That also requires investment. Employees will need to be trained to work with AI agents and interpret fairness concerns, and organizations will need to devote resources to set up effective governance teams that can coordinate across departments. The people affected by AI decisions must also have clear and accessible avenues through which they can share their experiences. We expect these changes to involve a learning period. The benefits may be limited at first, but as organizations develop the necessary capabilities, we believe this approach could provide a meaningful path towards fairer AI systems.
Co-authors on the paper include Arun Rai of Georgia State University and Ling Xue of the University of Georgia.
