Consumer feedback surveys have become ubiquitous. When people withdraw money from their bank, buy paper from Staples, or undergo a medical test at Labcorp, they frequently receive a short questionnaire soon afterward by text, email.

Hospital patients also often receive surveys. These tend to run longer than the usual surveys, ever since the launch of the Hospital Consumer Assessment of Healthcare Providers and Systems (HCAHPS) in 2006, and typically arrive in paper form by mail within 48 hours to six weeks after a hospital stay. Created by the Centers for Medicare and Medicaid Services (CMS) and the Agency for Healthcare Research and Quality (AHRQ), the survey is sent to randomly selected patients, and it contains 29 close-ended questions about their hospital experience and their satisfaction with their care. For hospitals the responses can affect their finances. “Medicare and other payers make reimbursements to hospitals and clinics contingent in part on how these organizations perform on HCAHPS and related patient surveys,” said LDI Senior Fellow Shivan Mehta, Associate Chief Innovation Officer at Penn Medicine.

The survey is rigorously validated and has proven valuable in helping hospitals improve patient experiences. “The HCAHPS survey tells us something that a lot of other quality measures don’t tell us because we get direct information straight from patients,” said LDI Senior Fellow Paula Chatterjee, Director of Health Equity Research at LDI.

In addition, “for research conducted across large numbers of hospitals, HCAHPS has been the gold standard for evaluating patient satisfaction,” said LDI Senior Fellow Karen Lasater, Associate Professor, Center for Health Outcomes and Policy Research, at Penn Nursing.

However, in recent years health experts have expressed concerns about the length and slow returns of the HCAHPS survey in an age when consumers have become used to answering brief surveys via text or email. And at least one health care system—Penn Medicine—has developed its own text-based feedback system to supplement HCAHPS.

A Standard National Survey

Virtually all U.S. acute-care hospitals use HCAHPS. It is different from the frequently encountered consumer satisfaction surveys used by, say, Staples, because rather than asking for a rating of the services obtained, it focuses on patients’ experiences.

The HCAHPS has several components, including asking about the cleanliness and noise level of the environment, and communications with providers. Just two questions address satisfaction specifically: one asking patients to rate the hospital from 1 to 10, and another asking them to indicate whether they would recommend the hospital to friends and family.

HCAHPS is part of a government effort to incentivize hospitals to improve the quality of care, and it works because health systems want to provide good service compared to their competitors and attract more patients, according to Mehta. HCAHPS rankings are publicly available on the CMS website. HCAHPS can also affect Medicare reimbursements. For FY 2024, certain portions of the HCAHPS test make up 25% of CMS’s Hospital Value-Based Purchasing Score. A low score on these portions can trigger up to a 2% loss in Medicare payments to a hospital system. One sign of the effectiveness of HCAHPS: “In the last decade, scores have consistently risen,” said Chatterjee.

Low Response Rates in HCAHPS

Survey response rates may be dampened by the time lag for it to arrive in mailboxes, according to David Asch, Senior Vice President for Strategic Initiatives at Penn.  “HCAHPS tends to be a paper survey that shows up variably, often weeks after a clinical encounter, when patients may not accurately recall their visits,” Asch said. He noted that the survey’s many structured questions also make it burdensome to complete. 

Low response rates—as low as 26% in 2017—are a concern: “You are not getting the full breadth of experiences because you’re only getting responses from certain patients. The problem is that there are too many questions,” Mehta said. He proposes rotating subsets of questions rather than asking respondents to complete all 29. 

To address the response rate issue, CMS has tested various ways of administering the survey. Allowing people to answer HCAHPS questions online can increase response rates, as can using more than one modality (for instance, patients who don’t respond to a mail survey may later respond to a follow-up phone query), according to the authors of a 2024 JAMA Health Forum article. 

“Offering two or more modes sequentially increases the likelihood that patients can respond in their preferred mode,” the authors write. Looking at 36,000 patients from 46 general hospitals, they found that a three-phase survey protocol (web-mail-phone) resulted in the highest response rates and was the most inclusive of underserved patients, who are often people of color. 

Beginning in 2025, hospitals will be able to use web-mail, web-phone, and web-mail-phone protocols to administer HCAHPS.

The survey’s lengthy questions have proven harder to address. The questions “can be anticipated to silence those who cannot read well, and those who cannot read well may have very different health care experiences,” Asch said. About 20 years ago, LDI researchers tried to address this problem by translating HCAHPS questions into pictures. The effort was not a success. “It was quite hard because the questions are so complex,” Asch said. “If you need 137 pictographs to direct someone to put together an IKEA bookcase, chances are the bookcase is difficult to assemble and maybe you ought to design it a different way in the first place.”  

Equity Concerns

Lower response rates for underserved patients are not the only health equity issue raised by the HCAHPS survey. In a 2012 JAMA Internal Medicine article, Chatterjee and colleagues reported that safety-net hospitals—those that serve lower-income patients—receive worse HCAHPS patient experience and satisfaction scores than non-safety net hospitals. 

A follow-up study, published in 2022 in JAMA Health Forum Safety, found that the gap had not improved between 2008 to 2019. 

The implications of the finding are meaningful: “Better HCAHPS scores are correlated with some other measures of hospital quality including processes of care for acute myocardial infarction, pneumonia, and heart failure, though this evidence is somewhat old,” Chatterjee said. In addition, she said, “Safety-net hospitals get financially penalized for their lower performance on quality measures, though the effect of many of these programs on overall hospital finances appears to be small. This is likely because safety-net hospitals derive revenue from a lot of nonpatient care sources, like local/state/other governmental subsidies,” said Chatterjee. 

At a time when equity issues are salient, it is notable that HCAHPS is not written in a way to elicit feedback about racism in medical services. “We need to find ways to measure experiences of racial discrimination,” said LDI Senior Fellow Anish Agarwal, Assistant Professor, Emergency Medicine, at Perelman. 

Agarwal is lead author of a 2022 report in the Journal of the American College of Emergency Physicians Open that used automated text messages to elicit responses from patients discharged from the emergency room at three Penn hospitals. The researchers found that one in 10 African-American people felt that their race has a negative effect on their care, compared to fewer than two in 100 white patients. A second study, reported this year in JAMA Health Forum, used qualitative interviews with a subset of the initial respondents to document their experiences. The most common sentiment was feeling that their concerns were being dismissed, Agarwal said. 

On the other hand, while the surveys “do not ask explicitly about racism or discrimination, they do allow assessment of whether differences in patient experiences occur for members of different groups,”  said LDI Senior Fellow Ingrid Nembhard, Professor of Health Care Management at Wharton and CAHPS Consortium Research Team Member. Responses to any of the questions, including about patient-provider communication, can be compared by race, gender, and primary language. Studies that have conducted these analyses often show differences that confirm inequities. Some also show conditions under which equity is greater.” 

A Question of Timing 

Traditional patient surveys like HCAHPS have been criticized for not providing information quickly enough to act upon the responses. “If someone says that the waiting room is a mess, that’s not helpful to learn 30 days later,” said Christina O’Malley, Director of Network Telemedicine at Penn Medicine. In 2020, a Penn team that included Asch and O’Malley designed and implemented a system that would deliver quick feedback on outpatient experiences via text message, email, or robocall. (The system did not replace HCAHPS, which relates to in-hospital experiences and is mandated by CMS regulations.) “The goal was to make sure that we have an understanding of how patients experience the health care system so that we can improve it,” Asch said. 

PMX Feedback was built on top of an existing tool called Switchboard, which already sent patients texts and emails about appointments and integrated with Penn’s electronic health record system, allowing the program to pull basic demographic information about the patient. “We can say a patient has an appointment scheduled for Friday, May 10 at 2 PM, so at 5 PM when they’re out of that visit, we’ll send them a text message with a couple of short questions, and then when they respond to that text message, we can tie their answers back to them,” O’Malley said.

Since PMX Feedback was implemented in 2021, response rates have more than doubled, from 10% to 15% before to over 30% now. One dramatic improvement was in maternal fetal medicine, a clinical area that serves a high proportion of minority and low-income women. “When we went live with PMX, this section went from receiving five responses a month to 300—an improvement of 60 times,” O’Malley noted. 

The outcomes from using this new system have made clear, Asch said, that “moving to a text-based simple engagement was long overdue.”

A New Direction: Patient Narratives

Currently, the main way patients share and learn about health care quality online is through user-generated content found on forums such as Yelp. But comments on these forums may not represent the complete range of patient experiences. 

To facilitate the systematic collection of representative patient comments, AHRQ funded the development and rigorous testing of a CAHPS Narrative Item Set (NIS), which consists of five open-ended questions. Organizations can add these questions to their existing patient experience surveys such as their CAHPS surveys (surveys similar to HCAHPS exist for different entities and conditions, such as medical groups, cancer care, and nursing homes). Questions include: “Was there anything you wish had gone differently in your experiences with this provider and their staff in the last six months? If so, please explain what happened, how it happened, and how it felt to you.” 

The answers to these questions combine to form patient narratives. LDI Senior Fellow Nembhard described them as “stories about patients’ care experiences in their own words.”. 

“Patient narratives are increasingly being used by organizations to understand why patients gave the scores they did on the closed-ended items, learn about aspects of care experiences that are not captured by closed-ended items, and access patients’ creative ideas for improving care experiences. They’re giving voice to patients’ experiences in a way that complements those survey questions,” Nembhard explained.

Some institutions—including Mass General Brigham—have started posting patients’ ratings and comments on their websites. “They post their comments publicly knowing that patients want insight on what care is really like in various institutions so they can select those that meet their desires,” Nembhard said. Penn displays ratings and comments from its PMX Feedback surveys in provider profiles on its website. 

Nembhard and other researchers have experimented with using an interactive tool to help administrators and staff at several ambulatory practices at New York-Presbyterian interpret and use feedback data from open-ended narrative questions connected to a modified CAHPS survey. Compared to offices where the same information was delivered by email, the intervention sites showed greater behavioral change at the staff and practice level. The experiment resulted in increased patient experience scores related to interactions with office staff and wait times, according to study results. 

In this and other ways, patient feedback surveys increasingly are accommodating a world in which consumers are used to interacting digitally. For many observers, that can’t happen soon enough.  “Quality has been a big focus of health care in this country—and experience is one domain of quality that’s important in itself,” Mehta said. “Surveys are one way to measure patient experience. The results of these surveys should be provided to managers and clinicians, and there need to be mechanisms for practices and hospitals to act on the feedback.”

The Nursing Factor

The HCAHPS has been particularly fruitful for research in the nursing area. “There’s a large literature showing that patients who are in hospitals with better nurse staffing, meaning that nurses care for fewer patients at a time, are likely to have better outcomes, including greater satisfaction with their care,” said Lasater. 

For a 2024 article in Medical Care, a Penn group—including Lasater, LDI senior fellows Matthew McHugh and Eileen Lake, along with then-Penn nursing graduate student Kathleen Fitzpatrick Rosenbaum—     looked at which modifiable nursing resources are most predictive of HCAHPS performance. 

“We found that the single most important thing that hospitals should do to increase patient satisfaction is to improve the nurse work environment,” said Lasater. “Hospitals with more favorable work environments foster nursing authority over decisions affecting patient care, include nurses on hospital committees, invest in staff development and continuing education for nurses, and have nursing leadership that is highly visible and responsive to concerns raised by frontline nurses.” 

The second biggest factor was the hospital’s proportion of nurses with a bachelor’s degree or higher. More favorable staffing to patient ratios was next in impact.

---

Author

---