News

AI Scribes: Can Technology Do More Than Free Doctors From Data Entry?

A Penn Lab Takes a New Approach to Unpacking the Doctor-Patient Encounter

March 17, 2025

By:

Hoag Levins

Print:

A team led by LDI Senior Fellow Kevin B. Johnson, MD, MS, is dramatically rethinking how to reinvent the clinician-patient exam room encounter

Since the widespread adoption of electronic health record (EHR) systems in the 1990s, the health care industry has been on a relentless quest to digitally optimize the doctor-patient interaction. Today, many predict this pursuit will be dramatically changed by the integration of artificial intelligence (AI) into nearly all aspects of EHR systems, transforming patient care and clinical workflows in ever more revolutionary ways.

AI-powered EHRs are now theoretically capable of automating administrative tasks, providing real-time clinical insights, personalizing treatment plans, and allowing health care providers to focus more on patient care and less on data entry. Implementing these capabilities into a coherent clinical practice has the potential to not just enhance efficiency. It promises to fundamentally change how medical professionals interact with patients to make clinical decisions in what is becoming, in effect, a true collaboration of doctor and computer code. However, while the idea of such a system is popular, its full implementation remains a maddeningly complicated puzzle, particularly regarding the documentation of the physician-patient encounter that is such a key event.

Buried in Clerical Work

The current EHR systems have not only failed to solve that puzzle, but they’ve also made it far worse and become a major driver of clinician burnout. A 2024 study funded by the American Medical Association (AMA) and published in the Journal of the American Medical Association (JAMA) looked at EPIC Systems data from 200,081 physicians across 396 organizations. It found that physicians averaged 5.8 hours of active EHR use per eight-hour patient schedule. That means, at 57.8%, clerical EHR tasks ate up the majority of physicians’ time.

“Since the 1960s, the only change in the paradigm we use for clinical documentation has really been the conversion from handwritten notes to typed or electronically entered notes, and it’s time for us to be thinking about what we can do differently to get the provider out of the secretary role,” said LDI Senior Fellow and Penn Professor Kevin B. Johnson, MD, MS. In 2023 he received a National Institutes of Health (NIH) Director’s Pioneer Award grant to explore how a broader range of technology might be brought to the doctor-patient encounter in a way that freed the physician from clerical duties and provided patients with a more intense and diagnostically meaningful connection to the clinician.

Expanding Vision of AI Scribes

That investigation—Re-imaging Documentation Using Computational Enhancement (REDUCE)—is now at the heart of Johnson’s AI for Ambulatory Care Innovation (AI-4-AI) Lab. It’s focused on greatly expanding and enhancing what are now known as ambient AI scribe systems.

The system Johnson’s team has been building and studying might be thought of as a window into the interaction between the physician, the patient, and the computer. The exam room includes three video cameras, one of which is worn on the head of the physician. Another records the physical movements of the doctor and patient and a third records the computer screen. Aside from being transcribed, the data from the audio and video feeds goes into a central repository where it is further processed by a variety of AI analytics. These include concept extraction algorithms, interaction analysis algorithms, activity labeling algorithms, and other algorithms that assess cognition, indicate nonobvious frailty, and identify gaps in the patient care. These data, combined with data from the electronic health record, provide a new resource that Johnson hopes to release to the public to catalyze research.

Along the way, Dr. Johnson has become a lightning rod for evaluations using AI in clinical care. The first, early-phase study of real-world results of some aspects of reimagining visit documentation was published in February in JAMA Network Open. It found that 46 participating physicians from 17 different medical specialties reported reduced EHR engagement time per visit and a significant time savings to complete documentation.

Quantifying the Entire Visit

“The goal of the project is to think not just about documentation, but to get beyond it and to think about ways in which the entire visit can be quantified, documented, and reused in support of patient care and quality,” said Johnson.

Johnson came to Penn as a Penn Integrated Knowledge (PIK) professor in 2021. Created in 2004, the PIK program recruits renowned scholars whose work draws from multiple disciplines and whose achievements demonstrate a rare ability to thrive at the intersection of multiple fields of research. He is simultaneously a Professor of Biostatistics, Epidemiology and Informatics, and Pediatrics at the Perelman School of Medicine; of Computer and Information Science and Bioengineering at the School of Engineering and Applied Science; and of Science Communication at the Annenberg School for Communication.

“Our approach is to leverage advances in areas like AI perception and language understanding to re-envision the output of the medical encounter,” said Johnson. “We want to use those technologies to summarize what has been spoken and unspoken, to assess cognition, to detect things like conscious or unconscious bias, to identify gaps in symptoms, and identify ways to quantify signs and symptoms such as those of a patient stuttering, or a patient with word-finding issues, or one exhibiting a gait disturbance that may increase the risk of falls.”

Anonymized Data’s Further Uses

To provide broad access to sensitive video data, Johnson’s team developed tools to de-identify the videos. Even the video of the doctor and patient movements during the encounter is anonymized into stick figures that can, in association with the audio, provide useful visual information on the clinician and patient affects and interactions.

Aside from freeing the doctor from input tasks and allowing for greater focus on the patient’s words and body cues, this type of system can aggregate data from thousands of doctor-patient encounters. This collection creates a rich, anonymized resource that can provide valuable insights for researchers studying the doctor-patient interaction itself as well as for educating medical students about the nonobvious nuances of the patient encounters that are the core of their future medical practice.