0
We're unable to sign you in at this time. Please try again in a few minutes.
Retry
We were able to sign you in, but your subscription(s) could not be found. Please try again in a few minutes.
Retry
There may be a problem with your account. Please contact the AMA Service Center to resolve this issue.
Contact the AMA Service Center:
Telephone: 1 (800) 262-2350 or 1 (312) 670-7827  *   Email: subscriptions@jamanetwork.com
Error Message ......
Original Contribution |

Evaluation of a Consumer-Oriented Internet Health Care Report Card:  The Risk of Quality Ratings Based on Mortality Data FREE

Harlan M. Krumholz, MD; Saif S. Rathore, MPH; Jersey Chen, MD, MPH; Yongfei Wang, MS; Martha J. Radford, MD
[+] Author Affiliations

Author Affiliations: Section of Cardiovascular Medicine, Department of Medicine (Drs Krumholz, Chen, and Radford, and Messrs Rathore and Wang), and Section of Health Policy and Administration, Department of Epidemiology and Public Health (Dr Krumholz), Yale University School of Medicine, New Haven, Conn; Yale-New Haven Hospital Center for Outcomes Research and Evaluation, New Haven, Conn (Drs Krumholz and Radford); and Qualidigm, Middletown, Conn (Drs Krumholz and Radford). Dr Chen is currently affiliated with the Department of Radiology, Hospital of the University of Pennsylvania, Philadelphia.


JAMA. 2002;287(10):1277-1287. doi:10.1001/jama.287.10.1277.
Text Size: A A A
Published online

Context Health care "report cards" have attracted significant consumer interest, particularly publicly available Internet health care quality rating systems. However, the ability of these ratings to discriminate between hospitals is not known.

Objective To determine whether hospital ratings for acute myocardial infarction (AMI) mortality from a prominent Internet hospital rating system accurately discriminate between hospitals' performance based on process of care and outcomes.

Design, Setting, and Patients Data from the Cooperative Cardiovascular Project, a retrospective systematic medical record review of 141 914 Medicare fee-for-service beneficiaries 65 years or older hospitalized with AMI at 3363 US acute care hospitals during a 4- to 8-month period between January 1994 and February 1996 were compared with ratings obtained from HealthGrades.com (1-star: worse outcomes than predicted, 5-star: better outcomes than predicted) based on 1994-1997 Medicare data.

Main Outcome Measures Quality indicators of AMI care, including use of acute reperfusion therapy, aspirin, β-blockers, angiotensin-converting enzyme inhibitors; 30-day mortality.

Results Patients treated at higher-rated hospitals were significantly more likely to receive aspirin (admission: 75.4% 5-star vs 66.4% 1-star, P for trend = .001; discharge: 79.7% 5-star vs 68.0% 1-star, P = .001) and β-blockers (admission: 54.8% 5-star vs 35.7% 1-star, P = .001; discharge: 63.3% 5-star vs 52.1% 1-star, P = .001), but not angiotensin-converting enzyme inhibitors (59.6% 5-star vs 57.4% 1-star, P = .40). Acute reperfusion therapy rates were highest for patients treated at 2-star hospitals (60.6%) and lowest for 5-star hospitals (53.6% 5-star, P = .008). Risk-standardized 30-day mortality rates were lower for patients treated at higher-rated than lower-rated hospitals (21.9% 1-star vs 15.9% 5-star, P = .001). However, there was marked heterogeneity within rating groups and substantial overlap of individual hospitals across rating strata for mortality and process of care; only 3.1% of comparisons between 1-star and 5-star hospitals had statistically lower risk-standardized 30-day mortality rates in 5-star hospitals. Similar findings were observed in comparisons of 30-day mortality rates between individual hospitals in all other rating groups and when comparisons were restricted to hospitals with a minimum of 30 cases during the study period.

Conclusion Hospital ratings published by a prominent Internet health care quality rating system identified groups of hospitals that, in the aggregate, differed in their quality of care and outcomes. However, the ratings poorly discriminated between any 2 individual hospitals' process of care or mortality rates during the study period. Limitations in discrimination may undermine the value of health care quality ratings for patients or payers and may lead to misperceptions of hospitals' performance.

Increasing interest in the quality of health care has led to the development of "report cards" to grade and compare the quality of care and outcomes of hospitals,1 physicians,2 and managed care plans.3 The organizations that produce these evaluations span the spectrum of popular periodicals, federal and state agencies, nonprofit accreditation organizations, consulting companies, and for-profit health care information companies.4 In addition, the Centers for Medicare and Medicaid Services (formerly called the Health Care Financing Administration) has recently expressed interest in developing a public performance report for hospitals.5

One of the most prominent organizations involved in providing health care quality ratings is HealthGrades.com, Inc. This company has developed "Hospital Report Cards" as part of an effort to provide comparative information about quality of health care providers via the Internet.68 The company's Web site indicates that as "the healthcare quality experts," it is "creating the standard of healthcare quality."9 Using primarily publicly available Medicare administrative data to calculate risk-adjusted mortality rates for a variety of conditions, HealthGrades.com claims to provide "accurate and objective ratings" for hospitals to enable patients to make "well-informed decisions about where to receive their care." As a free service, public interest in the Web site is substantial, with over 1 million visitors in 2001 and discussion of the company's rating system in publications such as Yahoo! Internet Life10 and in print stories in USA Today and the Los Angeles Times.11,12 HealthGrades.com is publicly traded on NASDAQ and reported over $7 million in revenue in 2000, with a 640% increase in ratings revenue over the fourth quarter of 1999.13 With ratings soon appearing for nursing homes, hospices, home health agencies, fertility clinics, linkages to data concerning individual health plans and providers, and a recently announced partnership with The Leapfrog Group,14 this is one of the most ambitious health ratings resources available online today.

While hospital ratings are widely disseminated to the public, little information is available about their validity. The HealthGrades.com rating system uses publicly available Medicare Part A billing data for many of its ratings, but its statistical methods have not been published in the peer-reviewed literature, nor has any published study, to our knowledge, evaluated its performance. By providing ready access to ratings for all US hospitals via a free, public-access Web site, this rating system offers consumers, who may be unfamiliar with the limitations of rating systems, an option that no other rating system today provides—the opportunity to directly compare 2 individual hospitals' "performance" for a variety of conditions. Use of such ratings may have substantial benefit if it encourages hospitals to compete on quality, but may have significant, unintended, and potentially deleterious consequences if the ratings do not accurately discriminate between individual hospitals' performance. Accordingly, we sought to determine if these ratings could discriminate between hospitals based on their quality of care and outcomes.

For this evaluation we used data from the Cooperative Cardiovascular Project (CCP), a national initiative to improve quality of care for Medicare beneficiaries hospitalized with acute myocardial infarction (AMI). The CCP involved the systematic abstraction of clinically relevant information from more than 200 000 hospitalizations for AMI nationwide. As a highly prevalent condition with significant morbidity and mortality and established quality of care and outcomes measures, AMI is well suited to an assessment of hospital performance. We compared hospitals ratings with process-based measures of the quality of AMI care and risk-standardized 30-day mortality based on medical record review. Since the public is expected to be particularly interested in comparisons between individual hospitals, we determined how often individual higher-rated hospitals performed better than lower-rated hospitals in head to head comparisons.

 America's Best Hospitals: 2001 Hospital Guide.  US News and World Report; 2001. Available at: http://www.usnews.com/usnews/nycu/health/hosptl/tophosp.htm. Accessed February 12, 2002.
Green J, Wintfeld N. Report cards on cardiac surgeons: assessing New York State's approach.  N Engl J Med.1995;332:1229-1232.
National Committee for Quality Assurance.  NCQA's Health Plan Report Card. Washington, DC: National Committee for Quality Assurance; 2000.
 Health Care Report Cards 1998-1999.  4th ed. Washington, DC: Atlantic Information Services Inc; 1998.
Pear R. Medicare shift towards HMOs is planned.  New York Times.June 5, 2001:A19.
Morrissey J. Internet company rates hospitals.  Mod Healthc.1999;29:24-25.
Schifrin M, Wolinsky M. Use with care. Forbes Best of the Web, June 25, 2001. Available at: http://www.forbes.com/bow/. Accessed February 12, 2002.
Prager LO. Criteria to identify "leading physicians" yield a long list.  American Medical News.September 6, 1999. Available at: http://www.ama-assn.org/sci-pubs/amnews/pick_99/prl10906.htm. Accessed February 12, 2002.
 Healthgrades.com: The Healthcare Quality Experts Available at: http://www.healthgrades.com. Accessed June 18, 2001.
Butler R. Fifty most incredibly useful sites.  Yahoo! Internet Life.July 2001. Available at: http://www.yil.com/features/feature.asp?volume=07&issue=07&keyword=usefulsites. Accessed February 12, 2002.
Appleby J, Davis R. Is your doctor bad?  USA Today.October 11, 2000:B1.
Carey B. Say "aah": your health online.  Los Angeles Times.July 2, 2001:S2.
 HealthGrades, Inc announces fourth quarter and year-end results; 2001. Available at: http://www.healthgrades.com. Accessed February 12, 2002.
 HealthGrades announces partnership agreement with the Leapfrog Group; 2002. Available at: http://www.healthgrades.com. Accessed February 12, 2002.
Marciniak TA, Ellerbeck EF, Radford MJ.  et al.  Improving the quality of care for Medicare patients with acute myocardial infarction: results from the Cooperative Cardiovascular Project.  JAMA.1998;279:1351-1357.
Ryan TJ, Anderson JL, Antman EM.  et al.  ACC/AHA guidelines for the management of patients with acute myocardial infarction: a report of the American College of Cardiology/American Heart Association Task Force on Practice Guidelines (Committee on Management of Acute Myocardial Infarction).  J Am Coll Cardiol.1996;28:1328-1428.
Fleming C, Fisher ES, Chang CH, Bubolz TA, Malenka DJ. Studying outcomes and hospital utilization in the elderly: the advantages of a merged data base for Medicare and Veterans Affairs hospitals.  Med Care.1992;30:377-391.
Daley J, Jencks SF, Draper D, Lenhart G, Thomas N, Walker J. Predicting hospital-associated mortality for Medicare patients: a method for patients with stroke, pneumonia, acute myocardial infarction, and congestive heart failure.  JAMA.1988;260:3617-3624.
White HA. A heteroskedasticity-consistent covariance matrix estimator and a direct test for heteroskedasticity.  Econometrica.1980;48:817-838.
Zhang J, Yu KF. What's the relative risk? a method of correcting the odds ratio in cohort studies of common outcomes.  JAMA.1998;280:1690-1691.
Vladeck BC, Goodwin EJ, Myers LP, Sinisi M. Consumers and hospital use: the HCFA "death list."  Health Aff (Millwood).1988;7:122-125.
 How to read the hospital death list.  New York Times.March 17,1986:A18.
Iezzoni LI, Burnside S, Sickles L.  et al.  Coding of acute myocardial infarction: clinical and policy implications.  Ann Intern Med.1988;109:745-751.
Miller MG, Miller LS, Fireman B, Black SB. Variation in practice for discretionary admissions: impact on estimates of quality of hospital care.  JAMA.1994;271:1493-1498.
Mennemeyer ST, Morrisey MA, Howard LZ. Death and reputation: how consumers acted upon HCFA mortality information.  Inquiry.1997;34:117-128.
Green J, Passman LJ, Wintfeld N. Analyzing hospital mortality: the consequences of diversity in patient mix.  JAMA.1991;265:1849-1853.
Green J, Wintfeld N, Sharkey P, Passman LJ. The importance of severity of illness in assessing hospital mortality.  JAMA.1990;263:241-246.
Wilson P, Smoley SR, Wedegar D. Second Report of the California Hospital Outcomes Project: Acute Myocardial Infarction. Sacramento, Calif: Office of Statewide Health Planning and Development; 1996.
Krumholz HM, Chen J, Wang Y.  et al.  Comparing AMI mortality among hospitals in patients 65 years of age and older: evaluating methods of risk adjustment.  Circulation.1999;99:2986-2992.
Dranove D, Kessler D, McClellan M, Satterthwaite M. Is More Information Better? The Effects of "Report Cards" on Health Care Providers. Cambridge, Mass: National Bureau of Economic Research; 2002. NBER Working Paper 8697.
Lohr KN. Outcome measurement: concepts and questions.  Inquiry.1988;25:37-50.
Thomas JW, Hofer TP. Accuracy of risk-adjusted mortality rate as a measure of quality of care.  Med Care.1999;37:83-92.
Braun BI, Koss RG, Loeb JM. Integrating performance measure data into the Joint Commission accreditation process.  Eval Health Prof.1999;22:283-297.
Jencks SF, Cuerdon T, Burwen DR.  et al.  Quality of medical care delivered to Medicare beneficiaries: a profile at state and national levels.  JAMA.2000;284:1670-1676.
Iezzoni LI. Data sources and implications: administrative data bases. In: Iezzoni LI, ed. Risk Adjustment for Measuring Health Care Outcomes. Ann Arbor, Mich: Health Administration Press; 1994.
Mehta RH, Stalhandske EJ, McCargar PA, Ruane TJ, Eagle KA. Elderly patients at highest risk with acute myocardial infarction are more frequently transferred from community hospitals to tertiary centers: reality or myth?  Am Heart J.1999;138:688-695.
Thiemann DR, Coresh J, Powe NR. Quality of care at teaching and nonteaching hospitals.  JAMA.2000;284:2994-2995.
Jencks SF, Williams DK, Kay TL. Assessing hospital-associated deaths from discharge data: the role of length of stay and comorbidities.  JAMA.1988;260:2240-2246.

Figures

Tables

References

 America's Best Hospitals: 2001 Hospital Guide.  US News and World Report; 2001. Available at: http://www.usnews.com/usnews/nycu/health/hosptl/tophosp.htm. Accessed February 12, 2002.
Green J, Wintfeld N. Report cards on cardiac surgeons: assessing New York State's approach.  N Engl J Med.1995;332:1229-1232.
National Committee for Quality Assurance.  NCQA's Health Plan Report Card. Washington, DC: National Committee for Quality Assurance; 2000.
 Health Care Report Cards 1998-1999.  4th ed. Washington, DC: Atlantic Information Services Inc; 1998.
Pear R. Medicare shift towards HMOs is planned.  New York Times.June 5, 2001:A19.
Morrissey J. Internet company rates hospitals.  Mod Healthc.1999;29:24-25.
Schifrin M, Wolinsky M. Use with care. Forbes Best of the Web, June 25, 2001. Available at: http://www.forbes.com/bow/. Accessed February 12, 2002.
Prager LO. Criteria to identify "leading physicians" yield a long list.  American Medical News.September 6, 1999. Available at: http://www.ama-assn.org/sci-pubs/amnews/pick_99/prl10906.htm. Accessed February 12, 2002.
 Healthgrades.com: The Healthcare Quality Experts Available at: http://www.healthgrades.com. Accessed June 18, 2001.
Butler R. Fifty most incredibly useful sites.  Yahoo! Internet Life.July 2001. Available at: http://www.yil.com/features/feature.asp?volume=07&issue=07&keyword=usefulsites. Accessed February 12, 2002.
Appleby J, Davis R. Is your doctor bad?  USA Today.October 11, 2000:B1.
Carey B. Say "aah": your health online.  Los Angeles Times.July 2, 2001:S2.
 HealthGrades, Inc announces fourth quarter and year-end results; 2001. Available at: http://www.healthgrades.com. Accessed February 12, 2002.
 HealthGrades announces partnership agreement with the Leapfrog Group; 2002. Available at: http://www.healthgrades.com. Accessed February 12, 2002.
Marciniak TA, Ellerbeck EF, Radford MJ.  et al.  Improving the quality of care for Medicare patients with acute myocardial infarction: results from the Cooperative Cardiovascular Project.  JAMA.1998;279:1351-1357.
Ryan TJ, Anderson JL, Antman EM.  et al.  ACC/AHA guidelines for the management of patients with acute myocardial infarction: a report of the American College of Cardiology/American Heart Association Task Force on Practice Guidelines (Committee on Management of Acute Myocardial Infarction).  J Am Coll Cardiol.1996;28:1328-1428.
Fleming C, Fisher ES, Chang CH, Bubolz TA, Malenka DJ. Studying outcomes and hospital utilization in the elderly: the advantages of a merged data base for Medicare and Veterans Affairs hospitals.  Med Care.1992;30:377-391.
Daley J, Jencks SF, Draper D, Lenhart G, Thomas N, Walker J. Predicting hospital-associated mortality for Medicare patients: a method for patients with stroke, pneumonia, acute myocardial infarction, and congestive heart failure.  JAMA.1988;260:3617-3624.
White HA. A heteroskedasticity-consistent covariance matrix estimator and a direct test for heteroskedasticity.  Econometrica.1980;48:817-838.
Zhang J, Yu KF. What's the relative risk? a method of correcting the odds ratio in cohort studies of common outcomes.  JAMA.1998;280:1690-1691.
Vladeck BC, Goodwin EJ, Myers LP, Sinisi M. Consumers and hospital use: the HCFA "death list."  Health Aff (Millwood).1988;7:122-125.
 How to read the hospital death list.  New York Times.March 17,1986:A18.
Iezzoni LI, Burnside S, Sickles L.  et al.  Coding of acute myocardial infarction: clinical and policy implications.  Ann Intern Med.1988;109:745-751.
Miller MG, Miller LS, Fireman B, Black SB. Variation in practice for discretionary admissions: impact on estimates of quality of hospital care.  JAMA.1994;271:1493-1498.
Mennemeyer ST, Morrisey MA, Howard LZ. Death and reputation: how consumers acted upon HCFA mortality information.  Inquiry.1997;34:117-128.
Green J, Passman LJ, Wintfeld N. Analyzing hospital mortality: the consequences of diversity in patient mix.  JAMA.1991;265:1849-1853.
Green J, Wintfeld N, Sharkey P, Passman LJ. The importance of severity of illness in assessing hospital mortality.  JAMA.1990;263:241-246.
Wilson P, Smoley SR, Wedegar D. Second Report of the California Hospital Outcomes Project: Acute Myocardial Infarction. Sacramento, Calif: Office of Statewide Health Planning and Development; 1996.
Krumholz HM, Chen J, Wang Y.  et al.  Comparing AMI mortality among hospitals in patients 65 years of age and older: evaluating methods of risk adjustment.  Circulation.1999;99:2986-2992.
Dranove D, Kessler D, McClellan M, Satterthwaite M. Is More Information Better? The Effects of "Report Cards" on Health Care Providers. Cambridge, Mass: National Bureau of Economic Research; 2002. NBER Working Paper 8697.
Lohr KN. Outcome measurement: concepts and questions.  Inquiry.1988;25:37-50.
Thomas JW, Hofer TP. Accuracy of risk-adjusted mortality rate as a measure of quality of care.  Med Care.1999;37:83-92.
Braun BI, Koss RG, Loeb JM. Integrating performance measure data into the Joint Commission accreditation process.  Eval Health Prof.1999;22:283-297.
Jencks SF, Cuerdon T, Burwen DR.  et al.  Quality of medical care delivered to Medicare beneficiaries: a profile at state and national levels.  JAMA.2000;284:1670-1676.
Iezzoni LI. Data sources and implications: administrative data bases. In: Iezzoni LI, ed. Risk Adjustment for Measuring Health Care Outcomes. Ann Arbor, Mich: Health Administration Press; 1994.
Mehta RH, Stalhandske EJ, McCargar PA, Ruane TJ, Eagle KA. Elderly patients at highest risk with acute myocardial infarction are more frequently transferred from community hospitals to tertiary centers: reality or myth?  Am Heart J.1999;138:688-695.
Thiemann DR, Coresh J, Powe NR. Quality of care at teaching and nonteaching hospitals.  JAMA.2000;284:2994-2995.
Jencks SF, Williams DK, Kay TL. Assessing hospital-associated deaths from discharge data: the role of length of stay and comorbidities.  JAMA.1988;260:2240-2246.

Letters

CME
Meets CME requirements for:
Browse CME for all U.S. States
Accreditation Information
The American Medical Association is accredited by the Accreditation Council for Continuing Medical Education to provide continuing medical education for physicians. The AMA designates this journal-based CME activity for a maximum of 1 AMA PRA Category 1 CreditTM per course. Physicians should claim only the credit commensurate with the extent of their participation in the activity. Physicians who complete the CME course and score at least 80% correct on the quiz are eligible for AMA PRA Category 1 CreditTM.
Note: You must get at least of the answers correct to pass this quiz.
You have not filled in all the answers to complete this quiz
The following questions were not answered:
Sorry, you have unsuccessfully completed this CME quiz with a score of
The following questions were not answered correctly:
Commitment to Change (optional):
Indicate what change(s) you will implement in your practice, if any, based on this CME course.
Your quiz results:
The filled radio buttons indicate your responses. The preferred responses are highlighted
For CME Course: A Proposed Model for Initial Assessment and Management of Acute Heart Failure Syndromes
Indicate what changes(s) you will implement in your practice, if any, based on this CME course.

Multimedia

Some tools below are only available to our subscribers or users with an online account.

Web of Science® Times Cited: 70

Related Content

Customize your page view by dragging & repositioning the boxes below.

See Also...
Articles Related By Topic
Related Collections
PubMed Articles