Why Healthcare Organizations Must Prioritize AI Governance

AI governance can also be viewed as an extension of data governance. In the end, AI is composed of data and advanced analytics at its core. AI solutions need the data governance principles and practices of data stewardship, data quality, metadata management, data privacy and security just as much or more than traditional data and analytics solutions need them. It is important to not reinvent governance for AI solutions from the ground up without considering the critical foundational elements of data governance.

Part of AI governance should address concerns around bias, decision-making and the economic and cultural impact of the technology. It’s critical that healthcare organizations build on a foundation of data governance and additional areas specific to AI governance if they hope to have sustained, organizationwide adoption of AI solutions.

Patients Are Concerned About Insufficient AI Governance

Every year, nonprofit healthcare research organization ECRI releases a list of the most significant threats to patient safety. For 2025, “insufficient governance of artificial intelligence in healthcare” claimed the No. 2 spot.

And patients have every right to be concerned about how their providers use AI in their care delivery. If there is a virtual nursing solution that relies on computer vision to monitor a room, have patients been notified about the camera and microphone, and have they given their explicit consent? Do patients know that their primary care physicians are using ambient listening tools for clinical documentation during annual exams? How is the organization protecting patient healthcare data if an AI solution is being used?

These are questions healthcare organizations need to answer as they explore the roles AI can play within their clinical workflows. But according to a 2023 survey from the Center for Connected Medicine at UPMC and KLAS Research, just 16% of health systems have governance structures in place to specifically address AI and data.

A University of Minnesota School of Public Health analysis of 2023 American Hospital Association survey data found that while a majority of U.S. hospitals were using AI-assisted predictive models, only 44% of them had evaluated their models for bias. This is detrimental to the equitable delivery of care, and if healthcare organizations want to achieve the Quintuple Aim, which includes advancing health equity, they need to seriously address AI bias.

READ MORE: Take advantage of data and AI for better healthcare outcomes.

AI Governance Is Not Just a Technology Change

Beneath the consistent interest around AI are concerns about healthcare’s workforce shortage and persistent clinical burnout. Physicians and nurses are stretched thin, and they don’t always have the support that they need. On top of that, the U.S. will continue to see a steady increase in healthcare demands with an aging baby boomer population.

Provider organizations are looking for solutions to reduce the administrative burden on clinicians and help them do more with less. They’re also hoping to adopt solutions that can attract talent and perhaps extend the careers of nurses or specialists who may otherwise retire in the next few years.

That’s why, as healthcare organizations keep piloting and expanding AI-powered applications, there must be a strong governance structure to ensure providers and patients are protected. And it’s not enough to appoint a senior leader to oversee AI and call it a day. AI governance should include multiple departments, with organizationwide training and education.

“We need a level of AI literacy so that the physicians using the device say, ‘Whoa, wait a minute. Has it gone through our AI evaluation process? What AI are you talking about?’” said Dr. Brett Oliver, chief medical information officer at Louisville, Ky.-based Baptist Health Medical Group, during an industry conference in 2024. “Relatively small conversations like that can have a big impact to make sure that the AI that’s coming in gets evaluated and is monitored.”