How AI’s Growing Role in Nursing Raises Questions About Safety, Ethics, and Human Care
Penn Nursing Report Cautions That AI Systems May Add Rather Than Reduce Costs and Workflow Burdens
In Their Own Words
Proposing a Framework to License Autonomous Clinical AI
As Autonomous Clinical AI Continues to Evolve, We Need Uniform Regulatory Standards, Says LDI Fellow
Print:
This First Opinion originally appeared in STAT News on May 11, 2026.
Late last month, Utah’s Medical Licensing Board called for the immediate suspension of the state’s pilot program with the AI company Doctronic. The program lets a chatbot evaluate patients and recommend prescription renewals for nearly 200 chronic condition drugs, with the state planning to phase out physician review of each case.
The board said that it only learned about the pilot after it had launched. Its warning was blunt: proceeding without proper clinical oversight “potentially places Utah citizens at risk.”
The backlash was predictable, and avoidable. Utah is one of at least 47 states now considering more than 250 bills governing clinical AI, producing a patchwork of rules on bias audits, payment policy, and patient consent. All this while the federal government’s main tool for regulating medical software, the Food and Drug Administration’s device-approval process, is structurally unfit for regulating autonomous clinical AI. It was built for static products like imaging algorithms, not for adaptive systems that keep improving.
This comes at a time where the country faces a worsening physician shortage. National projections show shortfalls of tens of thousands of doctors over the next decade, especially in primary care and rural areas. Traditional fixes such as more medical school seats and more residency slots will take years to make a meaningful difference. Artificial intelligence offers a faster path, but it requires a regulatory framework that fits.
Read the full First Opinion here.
Author
Alon Bergman, PhD
Assistant Professor, Medical Ethics and Health Policy, Perelman School of Medicine; Assistant Professor, Health Care Management, Wharton School
More on AI in Health Care
Penn Nursing Leaders Speak Out on AI’s Growing Role in Patient Care
Q&A Looks at Concerns Over Patient Trust, Workforce Pressures, Unreliable Outputs, and Hospitals Rushing AI Adoption
AI Prescribing Medications In Utah: A Flawed Regulatory Playbook
A First-of-Its-Kind Pilot Raises Safety Risks and Questions About Who Should Regulate Machines Acting Like Doctors
U.S. AI Boom Could Cause Health Care Costs to Soar Without Changes to Payment Policy
Federal Officials Factor in Health Gains to Control Spending, LDI Fellow Says
AI Device Firms Pay $60M to U.S. Doctors, Raising Oversight Concerns
In a First-of-Its-Kind Analysis, Researchers Track Industry Payments Tied to AI Medical Devices and Call for Stronger FDA Oversight, Transparency, and Aid for Underserved Hospitals
Can AI Hear When Patients Are Ready for Palliative Care?
Researchers Use AI to Analyze Patient Phone Calls for Vocal Cues Predicting Palliative Care Acceptance