The October 2016 issue of The Joint Commission Journal on Quality and Patient Safety, features the article "Consumer Rankings and Health Care: Toward Validation and Transparency" by Bala Hota, MD, MPH, and co-authors, Rush University Medical Center (RUMC), Chicago. After RUMC received a lower than expected ranking for patient safety in the 2015-16 U.S. News & World Report's (USNWR) "Best Hospitals" rankings, the authors compared the data that USNWR used for their hospital to their own internal data.
The authors found that the USNWR data showed many more patient safety events, such as pressure ulcers, almost all of which had actually been present at the patient's admission. Suspecting a broader problem, Hota, et al. analyzed data on a sample of hospitals and found that RUMC was not the only organization with discrepancies in data. False-positive event rates were common among high-transfer and high-volume hospitals.
The authors conclude more transparency and validation is needed for consumer-based benchmarking methods. In response to these findings and concerns raised by others, USNWR made changes to its methodology and data sources in 2016.
In an accompanying editorial, "The Quality Measurement Crisis: An Urgent Need for Methodological Standards and Transparency," David M. Shahian, MD, Elizabeth Mort, MD, MPH, and Peter J. Pronovost, MD, PhD, reflect on the Hota, et al. article to conclude, "Just as healthcare providers have ethical and moral responsibilities to the public they serve, rating organizations and journalists that grade providers have similar obligations--in their case, to ensure measure validity and methodological transparency."
RUMC further explored the importance of rating organization's validity and methodological transparency in the following essay, "Hospital Rankings Have Room for Improvement
When consumers make what can be life-or-death decisions about where to seek healthcare, they often look to the numerous online hospital rating services that are available, including the government-sponsored Hospital Compare, Consumer Reports, and Leapfrog.
Rating hospital quality is not as simple as rating the quality of a car, though. One reason is that patients vary in terms of how sick they are when they arrive at the hospital or undergo surgery. Older patients with multiple health problems or advanced diseases are more likely to develop an infection or complication than younger, healthier patients -- even though they receive the same quality of care.
The best hospital rating systems try to account for severity of illness. However, a Rush data expert's recent analysis of U.S. News & World Report's "Best Hospitals" rankings shows that these rating systems often are dependent on faulty data sets and performance metrics.
Rush University Medical Center consistently has scored high on patient safety measures, according to most rankings. In fact, Rush University Medical Center has received nine consecutive "A" grades for safety from the Leapfrog Group, putting it among fewer than 100 hospitals in the country that have received the highest possible grade each time since Leapfrog began issuing the semi-annual ratings in 2012.
But last year U.S. News "Best Hospitals" issue gave Rush the publication's lowest ranking for patient safety (one out of five). To learn what was behind the discrepancy, Rush's quality team - Tom Webb, Bala Hota, MD, MPH, and Omar Lateef, DO - dug into the U.S. News data and methodology.
"When we compared the data that U.S. News used for our hospital to our own internal data, we found big differences," says Hota, Rush's chief research informatics officer. "U.S. News showed many more patient safety events than our actual rates."
For example, U.S. News data showed that 25 patients at Rush had developed pressure ulcers, or bed sores, over a particular time period. In fact, only one patient had developed a pressure ulcer while at Rush.
Suspecting a broader problem, they analyzed data on a sample of hospitals and found that Rush was not the only organization inaccurately unfairly ranked. Hota reported the results of his study in the October issue of the Joint Commission Journal on Quality and Patient Safety.
High demand for hospitals' services skew safety scores down
Two hospital characteristics tended to negatively skew U.S. News patient safety scores: Taking care of large numbers of patients (i.e., high volumes) or admitting a high percentage of patients transferred from other hospitals.
Webb, Hota and Lateef found that the U.S. News data set was missing key information, such as whether patients with pressure ulcers had these sores when they were admitted to Rush. If conditions were present prior to arrival, Rush should not have been penalized.
Large referral hospitals such as Rush, which provide highly specialized care for the sickest patients and most complex conditions, also tend to have high volumes and high transfer rates. "Rush is the biggest transfer center in Illinois," Hota says. "If somebody is sick in a community hospital and not getting better, that hospital knows Rush is a reliable place to transfer the patient to."
Rush notified U.S. News about the Rush findings in 2015 and U.S. News made some revisions to its data methodology. As a result, Rush's patient safety score increased to four out of a possible five in the 2016-2017 U.S. News ratings released in August.
"U.S. News was very responsive to our findings, and the Rush community should take pride in the fact that we take care of very sick patients very well," says Lateef, Rush's chief medical officer. "When you adjust for patient acuity (severity of patients' conditions), our outcomes data are among the best in the country."
The Rush analysis of the U.S. News ratings confirms what many experts have suspected. "People have argued and theorized that if you take care of a lot of very sick people or have a lot of transfers, then the data could be biased," Hota says. "This study proves that. It shows that data is not perfect, and misunderstandings in the data can negatively portray a hospital."
Acknowledging that hospital rankings have room for improvement, U.S. News is convening a panel of experts in November to discuss how to better evaluate hospital quality, safety and performance, according to a press release.
Lateef and Hota, and the quality team believe that right now the rules used by hospital rating systems are too complex for most consumers to navigate. That's why they are urging U.S. News and other hospital rating systems to be more transparent about how they rate hospital quality, so that individual hospitals can validate the accuracy of their scores and work with the rating organization to adjust the scores as appropriate.
"The companies that are producing these ratings have to be more transparent," Hota says. "Their methodology should be reproducible and easily understood."
In the meantime, Rush will continue to measure, track, and share patient safety data rigorously, and use that date to drive improvements to make Rush's care even safer. "We consider quality to be the most important part of patient care," Lateef says. "We believe in being transparent about quality so we can continue marching toward improvement."
Source: Rush University Medical Center
Dear Helpdesk: Working in a Toxic Health Care Environment
March 28th 2024Dear Helpdesk is your steadfast companion, offering life coaching and workplace advice from 2 seasoned IPs for some of your most challenging real-life situations. Let us help you navigate the intersection between work and life, guiding you to navigate the dynamic world of infection prevention with confidence and grace. This article is on handling a toxic health care environment.
Product Locator: Spring and Early Mother's Day Gift Guide for Infection Prevention Personnel
March 27th 2024Whether it's a spring holiday, birthdays, or no reason at all, infection prevention personnel love to give and receive gifts that help at the end of a stressful day. Infection Control Today® offers some gift ideas for infection prevention personnel and their families.
Catching Up With Vangie Dennis, AORN 2022-2023 President at AORN 2024
March 26th 2024Infection Control Today (ICT) had the privilege of catching up with Vangie Dennis, MSN, RN, CNOR, CMLSO, at the Association of periOperative Registered Nurses' (AORN’s) International Surgical Conference & Expo 2024. As the former president of AORN and an esteemed figure in perioperative services, Vangie Dennis shared insights into her recent endeavors and the exciting new chapter she's embarked upon.
How To Optimize Your Time Management Strategies for the Busy Infection Preventionist
March 25th 2024Is your calendar resembling a chaotic masterpiece of overlapping tasks? Join the club of infection preventionists striving to balance responsibilities. Dive into proven strategies from a fellow infection preventionist to reclaim control of your time, streamline tasks, and boost productivity effectively. This is an IP Lifeline article.