In this health care podcast, I talk with Karl Bilimoria, MD. Dr. Bilimoria is a surgical oncologist and a VP of quality over at Northwestern Medicine. Plus, he is also a John B. Murphy professor of surgery. The second I heard that Dr. Bilimoria and his colleagues had worked on an initiative to “rate the raters” of hospital and physician quality, I reached out to get him on the show. I had just had about four conversations with various people about the difficulties of judging quality. And I had also had a confounding personal experience visiting a patient at a hospital judged a top hospital by a well-known national rating scale. And this “top” hospital had some readily apparent issues, and I am no expert. That got me wondering about the validity of some of these quality raters.
Given the importance and the need for health care quality transparency, Dr. Bilimoria and his colleagues set out to fill this gap by undertaking a (as mentioned) Rating the Raters process to evaluate and compare probably the major publicly reported hospital quality rating systems in the United States. These include the CMS (Centers for Medicare and Medicaid) Hospital Compare Overall Star Ratings, Healthgrades Top Hospitals, Leapfrog Safety Grade and Top Hospitals, and the U.S. News & World Report Best Hospitals.
Interestingly, that “top” hospital I was in was scored a top hospital by one of the lowest-rated raters.
You can learn more at the New England Journal of Medicine Web site, thesecondtrial.org, and the NEJM Catalyst Web site.
Karl Bilimoria, MD, is a surgical oncologist and a health services, quality improvement, and health policy researcher at Northwestern University’s Feinberg School of Medicine. He is the vice president for quality for the Northwestern Medicine system. He is also the vice chair for quality in the Department of Surgery and the John B. Murphy professor of surgery. His clinical practice is focused on melanoma and sarcoma. Dr. Bilimoria is the director of the Surgical Outcomes and Quality Improvement Center of Northwestern University (SOQIC), a center of 50 faculty and staff focused on national, regional, and local quality improvement research and practical initiatives. He is also the director of the 56-hospital Illinois Surgical Quality Improvement Collaborative (ISQIC).
Dr. Bilimoria has published more than 350 scientific articles, including numerous publications in JAMA and the New England Journal of Medicine. His research is funded by the National Institutes of Health, the Agency for Healthcare Research and Quality, Health Care Services Corporation, and numerous others. He was recently listed by Becker’s as one of the “Top 50 Experts Leading the Field of Patient Safety” in the United States.
02:06 The impetus for Dr. Bilimoria deciding to rate the raters.
03:38 How high the stakes are when considering these ratings.
05:01 Breadth vs depth when choosing how and what to measure among the rating systems.
05:38 What rating systems the Rate the Raters looked at and why.
06:11 Who got the best scores as a rating system and who got the worst?
06:58 Dr. Bilimoria and the Rate the Raters’ qualifications on rating these rating systems.
07:35 The methodology and criteria that the Rate the Raters came up with to evaluate these rating systems.
08:01 The six criteria that the Rate the Raters system uses to evaluate rating systems.
08:39 “At every step, we included the rating systems.”
09:14 The intent behind Rate the Raters.
09:55 Why having grades is a positive for the health care consumer.
10:41 What conflicts of interest might be of concern among these rating systems?
12:22 “Notable notes” for these rating systems, and what each rating system incorporates or doesn’t incorporate, and how these things affected their rating.
18:22 Creating a gold standard and finding ways to move the field forward.
22:05 Getting to better data in the short term and the long term by doing meaningful audits.
24:09 Individual ratings vs institutional ratings and where these intersect.
26:27 Dr. Bilimoria’s advice to employers and how to pick a rating system.
27:45 “These rating systems should be interpreted very cautiously, even the best of them.”
28:03 “Triangulate the data.”
30:07 What Dr. Bilimoria and Rate the Raters are currently working on.