PITTSBURGH - Health systems and hospitals are widely adopting AI solutions in administrative and clinical workflows, but there are significant gaps in strategy, validation, testing environments and success measurement, according to a new report from the Center for Connected Medicine at UPMC and KLAS Research.
The report examines how health systems are adopting and governing AI and is based on a survey of more than two dozen health system leaders. The research reveals that while AI adoption has accelerated rapidly across the industry, many organizations are still working to establish the infrastructure, governance frameworks and strategic foundations needed to support long-term success.
"The health care industry has moved remarkably quickly from discussing the potential of AI to actively deploying solutions across the enterprise," said Rob Bart, M.D., chief medical information officer at UPMC. "What's emerging from this research is a clear recognition that implementation is only the first step. Health systems are now focused on building the governance structures, testing capabilities and organizational strategies necessary to ensure AI delivers meaningful and measurable value."
The findings paint a picture of an industry moving quickly to embrace AI technologies, particularly in administrative and clinical workflows, while simultaneously grappling with the complexities of implementation, validation and oversight.
Among the report's key findings:
- More than 90% of health systems have deployed third-party AI solutions, demonstrating that AI has moved beyond experimentation and into mainstream use.
- Clinical documentation (52%) is the most commonly cited area for AI deployment, followed by revenue cycle, coding and billing (36%) applications.
- Testing of AI solutions is nearly universal, with 92% of organizations reporting that they evaluate third-party AI tools before deployment. However, validation methods vary considerably, ranging from formal vendor testing protocols to limited pilot programs and other informal approaches.
- Less than half of respondents (44%) report having a dedicated data platform or environment for testing AI solutions, highlighting potential challenges in evaluating performance, safety and scalability.
- Most organizations' AI strategies are evolving, with 63% indicating their approach to AI is developing or ad hoc rather than fully established or advanced.
- Leading barriers to AI adoption include limited resources, insufficient time and shortages of specialized talent, reflecting the realities of implementing emerging technologies in complex healthcare environments.
- Respondents reported no strong consensus on the metrics used to measure AI success, underscoring the need for greater alignment around how organizations evaluate return on investment, operational impact and clinical value.
The report explores how health system leaders are balancing the urgency to adopt AI-powered solutions with the responsibility to manage risk, validate performance and establish processes that support trustworthy and effective use of the technology.
For example, UPMC developed a real-world data platform called Ahavi that can validate and improve third-party AI models before deployment at UPMC. Ahavi leverages de-identified patient data to test AI solutions "in silico" to see their effect on wide-ranging applications without disruption to patient care.
"As AI adoption continues to expand, health care providers are recognizing that success depends on more than selecting the right technology," said
Ken Howard, vice president at UPMC Enterprises, the innovation, commercialization and venture capital arm of UPMC. "It also requires the right data infrastructure, governance processes, workforce capabilities and evaluation frameworks to support responsible innovation."
The report is intended to help healthcare executives, technology leaders, clinicians and operations teams benchmark their organization's AI maturity and better understand the challenges and opportunities facing the industry.
The full report, "Validation and Trust: How Health Systems Are Testing and Governing AI Solutions," is available for download at upmcenterprises.com.
