An answer to “But my hospital’s patient population is sicker…”
A recent article published in JAMA Internal Medicine and highlighted in the AcademyHealth blog documented that hospital leaders are paying attention to publicly-reported quality measures such as Hospital Compare. However, roughly half of them expressed doubt that the measures accurately portrayed the quality of care for specific conditions or could be used to draw inferences about quality at the hospital more generally, and more than one-half reported that the measures were not meaningful for differentiating among hospitals.
In two papers published in Health Services Research, LDI Senior Fellow Jeffrey Silber and colleagues addressed these concerns by developing and testing a new way to audit hospital quality that can more directly account for differences in patient populations. Silber likens existing measures to a classroom teacher trying to grade students by giving each student an entirely different exam. What is needed, he suggests, is a way to standardize that exam to more readily compare how different hospitals perform when treating similar patients.
The first paper, “Template Matching for Auditing Hospital Cost and Quality” details the process and benefits of using template matching to evaluate how hospitals perform when treating the same set of patients created from a random sampling of Medicare data. The auditing process requires three steps: creating the exam, administering the exam and grading the exam.
To create the exam, a template of 300 patients was generated through random sampling of Medicare Part A and Part B claim data and Outpatient Files from 2004-2006 of patients that received similar procedures within three states (New York, Texas and Illinois). For the purposes of this paper, the authors chose 100 patients who had general, gynecologic and urologic surgery and 200 patients who had orthopedic surgery.
Administering the exam involved the selection of 300 patients that matched the template at each of the 217 audited hospitals. The matching was performed using a complex algorithm that individually matched each patient from the template to a patient at each hospital, resulting in the creation of 217 sets of similar patients.
Exam grades are based on outcome measures for each hospital’s set of patients. Each hospital’s performance could be compared against the other 216 hospitals in the set without concerns over differences in patient population. A variety of outcome measures were used including in-hospital and 30-day mortality, readmissions within 30 days of discharge and resource utilization-based costs. Silber explains the significance of the results:
The results of our analysis of general surgery and orthopedics displayed considerable variation in outcomes, despite very uniform patient characteristics in the template for each hospital. The matched sample allows for a fair, directly standardized comparison across hospitals, since the matched sample of 300 patients is closely balanced between each hospital and the template. The resulting variation in outcomes was therefore more believable.
But this set of patients, while representative of the broad population, might be far different from a specific hospital’s average patient population. Silber explains how this method could still lead to unfair comparisons: “think of a physics major writing a genuinely terrible English essay.”
The second paper, “Hospital-Specific Template Matching for Auditing Hospital Cost and Quality” details a solution to this dilemma. The idea here is to provide insight for
(w)hen a hospital’s Chief Medical Officer desires to know precisely how well his or her hospital performs on its own distribution of patients and not on an external template that may not be representative of the type of patients seen at his or her specific hospital.
The procedure of this audit is nearly identical to the one described above but with one major difference: Instead of producing the matching template from a sampling of Medicare data of the broader population, the template is generated from a sampling of the index hospital’s patients. This creates a set of patients representative of that hospital’s population. This template is then used to match with hospitals to determine how well other hospitals would have performed if faced with the same set of patients that the index hospital sees most commonly. One advantage of this approach is that it allows for the evaluation of small hospitals with fewer than 300 patients.
These two new strategies set forth by Silber and colleagues should challenge the current method of auditing hospital quality and may bolster confidence that hospital quality measures reflect true differences in quality among hospitals.
BONUS: Watch Dr. Silber explain this research to Knowledge@Wharton