Ratings Game: Lists of “Top” Physicians, Hospitals Has Unclear Impact on Public

Hospital administrators, physicians, and researchers may argue over the validity of quality ratings and rankings, but the fact remains that these reports are readily available to the public. How these measures affect the relationships among clinicians remains an open question.

Individuals can see ratings through a variety of sources, including Thomson Reuters' “100 Top Hospitals” report as well as on such Web sites as US News & World Report's “America's Best Hospitals,” Consumer Reports Health, HealthGrades, the US Centers for Medicare & Medicaid Services' “Hospital Compare,” and the Leapfrog Group's online hospital ratings tool. Also, some hospitals and physicians are rated locally through community media sources.

David Dranove, PhD, professor of health industry management at Northwestern University's Kellogg School of Management in Evanston, Ill, said the cynical reason such public reporting has proliferated is to sell magazines or advertising space. But he also offered a more serious explanation for why individuals may turn to these ratings and rankings. “Consumers are often uncertain about the quality of care they are receiving,” he said. “In the last decade, there has been talk about quality of care being important, but it is not guaranteed.”

Still, for those seeking comparisons among hospitals or physicians to uncover the best possible health care, confusion abounds. These public reports use different measurements to assess performance, including Medicare data, the Dartmouth Atlas of Health Care, self-reports, and personal opinion. Data from different years are used, and different treatments for different conditions are judged. And the elements being evaluated may not truly represent the delivery of quality care, said Thomas R. Russell, MD, executive director of the American College of Surgeons.

“The biggest problem I have heard about is the inaccuracy of a lot of these reports; if you are being evaluated, that is one thing, but let's do it correctly,” Russell said. “The public has a right to know about outcomes, but now there is a lot of misinformation out there, and it drives surgeons crazy when they are given a poor score—you do not have any way to counter this.”

Michael Rothberg, MD, MPH, assistant professor of medicine at Tufts University School of Medicine's Baystate Medical Center in Springfield, Mass, and colleagues published a study looking at 5 Web sites assessing local hospitals for 4 diagnoses. The researchers found that the evaluations involved different measures of health care delivery, a variety of patient definitions, and different reporting periods. They also found that the Web sites failed to agree on hospital rankings within any diagnosis, even when using the same measures, such as mortality (Rothberg MB et al. Health Aff[Millwood]. 2008;27[6]:1680-1687).

Beyond the issue of inconsistent results, Rothberg even questions whether the elements being measured in these reports have any relevance for a patient considering a hospital for a treatment. “Let's say you are going to get your hip replaced. You can see the number of procedures a hospital does, but it does not say how many your doctor has done,” Rothberg said. “You might get the mortality rate from Medicare data, and it will be close to zero. If you went to Leapfrog, you get its safe practice score, but is the safe practice score the most important consideration; is having electronic medical records important?”

Moreover, a high ranking does not necessarily guarantee successful outcomes. Another study looked at risk-standardized 30-day mortality rates for patients admitted with acute myocardial infarction at the top 50 hospitals ranked in US News & World Report's “America's Best Hospitals.” The researchers found that even though admission to these institutions was associated with lower mortality compared with nonranked hospitals (16.0% vs 17.9%), about one-third of the ranked hospitals were outside the best performing quartile based on mortality, with 4 of them in the worst performing quartile (Wang OJ et al. Arch Intern Med. 2007;167[13]:1345-1351).

Hospital administrators and physicians may grumble over the methods used and outcomes from these assessments, but are ratings and rankings making a difference in how patients select where and with whom to receive treatment?

Dranove looked at report cards on hospitals in New York state when they first came out, from 1989 to 1991, and found that higher-ranked hospitals did not consistently increase market share but that hospitals with negative findings may have lost market share (Dranove D and Sfekas A. J Health Econ. 2008;27[5]:1201-1207). “When report cards are new, they do move market share; not by much, but still some,” Dranove said.

Other research suggests that rankings have little influence; rather, the primary care physician is still the leading source for patients seeking specialist physicians, and the opinions of referring physicians remain the leading factor for an individual patient choosing a hospital.

To explore the decision-making process among elderly patients, researchers associated with Dartmouth-Hitchcock Medical Center in White River Junction, Vt, conducted a telephone survey in 2003 of 510 Medicare beneficiaries who had undergone an elective high-risk procedure. The investigators found that 94% of respondents thought their hospital had a good reputation and that 88% considered their surgeon top-notch, with these beliefs mostly determined by what their referring physician recommended (Schwartz LM et al. BMJ. 2005;331 [7520]:821-824).

“The patients consistently said that their desired portal for information was their primary care physician,” said John D. Birkmeyer, MD, an author of the study and now a professor of surgery at the University of Michigan Medical School in Ann Arbor. “Less than 10% sought out information from the Internet.”

Although individuals may still rely on their physicians for hospital referral, Birkmeyer said, that influence is waning as patients appear to be systematically seeking hospitals with the highest volume and lowest mortality. “There is no doubt that over the past decade, for some types of selected procedures, patients are becoming more savvy on the location of where they get their care,” Birkmeyer said. “But we do not know why that is occurring. Is it because of these rating services? Probably not. I personally think the primary care physicians are more aware of the disparities between hospitals and are steering patients accordingly.”

So, are primary care physicians using public reporting sources to shape their opinion on hospital preferences? Rothberg does not think so. In an ongoing study involving referral to the 4 hospitals within 15 minutes' driving time of Springfield for patients with pneumonia (a condition for which quality data are abundant and available), none of the 50 primary care physicians interviewed said they used public reporting to assist in their decision of where to send a patient.

“So why are we spending so much money on it?” Rothberg asked. “The people most concerned about public reporting are the administrators of the hospital and quality officers because they do not want to look bad. If you are ranked last, it probably will not affect your market share or referrals, but who wants to be last? So they will work on metrics.”

Rick Wade, a spokesperson with the American Hospital Association, agrees. Hospitals use public reports for benchmarking, with particular attention to the “Hospital Compare” Web site of the Department of Health and Human Services, he said. “Those results are being tied to reimbursement.”

Another way some hospitals use the public reports is for marketing—for instance, by running advertisements trumpeting that a cardiac unit is now top-ranked in the community. But Dranove noted that the best hospitals in the country tend to ignore their rankings because they do not want the public to remember that, for example, one year the institution was rated the fifth best in the country, but today is only eighth best.

Rothberg added that practical matters may also make the best performers shy away from advertising their success. “Most of the top hospitals are filled to capacity and cannot take on more patients,” Rothberg said. “And for top physicians, there are those for whom you would have to wait 3 months to get an appointment—or their practices are closed to new patients.”

As for physicians specifically, public reporting appears to still have a way to go to convince clinicians that results are valid.

“It is much harder to rate physicians than hospitals because the sample size [of patients being treated by a particular clinician for a specific ailment] is so small, with a lot of variation in the data at the individual level,” Rothberg said. “There is much more patient contribution to quality measures in the outpatient setting than the inpatient setting.”

New, less formal ways to judge physicians may also be emerging. Rothberg speculated that Internet communication may increasingly affect consumer choice in the future as individuals share opinions anecdotally about various physicians. “Today, selecting a physician is done mostly through word of mouth from a family member, friend, or people you work with, but social networks may one day fill that traditional role,” Rothberg said.

Russell speculated that the difficulties hospitals and physicians face when dealing with public reporting may ease over time but will remain a fact of life in the health care delivery system. “We are in a transitional stage,” Russell said. “In the past, physicians were not really questioned, and patients were passive. But patients are becoming more activist, as are insurers—demanding this and that and not paying for things done improperly, and you will see this continue.”