Use of Public Performance Reports:  A Survey of Patients Undergoing Cardiac Surgery FREE

INFORMATION on the quality of care provided by physicians, hospitals, and health plans has traditionally been collected for internal quality assurance and has almost always remained confidential.¹ However, the last decade has seen explosive growth in the publication of reports on the quality of care.² Large-scale purchasers of health care services have driven the process, but state health agencies and traditional accrediting bodies are now demanding that health care providers furnish performance data for public use.^3- 4

Public performance reports are intended to guide patients' selection of providers, aid purchasers in contracting decisions, and stimulate quality improvement among providers. Prior research suggests that providers change their behavior in various ways in response to public reporting,^5- 7 but much less is known about the ways consumers use performance data.^8- 9 In spite of this, national consumer publications such as Consumers Digest,¹⁰U.S. News and World Report,^11- 12 and Newsweek¹³ now publish rankings of health plans and hospitals on patient satisfaction and quality of care. Both the Agency for Health Care Policy and Research¹⁴ and the Health Care Finance Administration¹⁵ have launched major programs to develop, evaluate, and disseminate quality measures to inform consumers selecting health plans and other medical care services.

The Pennsylvania Health Care Cost Containment Council has been at the forefront of this trend in the collection, analysis, and reporting of hospital and provider-specific data on cardiac surgery since 1992. The agency regularly publishes and disseminates risk-adjusted mortality rates on every Pennsylvania hospital, surgeon, and surgical group providing coronary artery bypass graft (CABG) surgery in its Consumer Guide to Coronary Artery Bypass Graft Surgery .^16- 19 The agency distributed 15000 copies of the first and second volumes of the Consumer Guide to hospitals, surgeons, public libraries, business groups, legislators, and the media.²⁰ It is available free to any individual who requests it. Public release of the Consumer Guide has received extensive media coverage.

Cardiac surgery is a dramatic event, frequently elective, with a significant operative mortality rate. Previous studies have shown that mortality rate variations are related to the quality of care.^21- 26 Thus one might expect that patients or their advisors would be particularly motivated to use the reported data. We examined use of the Pennsylvania Consumer Guide by patients who underwent CABG surgery at selected hospitals.

Forty-one Pennsylvania hospitals provide CABG surgery. Volume 4 of the Consumer Guide reported that 3 hospitals had lower and 5 had higher than expected mortality rates. We selected 4 hospitals that performed at least 400 operations within 1 year that are located in different regions of the state and that were willing to participate in our study. At each hospital, we asked individual surgeons or surgical groups to participate. Eighteen of 24 practicing surgeons agreed to participate. Participating surgeons performed 86% of all the CABG procedures in the 4 hospitals.

The Consumer Guide bases its rating on a hospital's in-hospital mortality rate relative to its expected mortality rate.⁷ Expected mortality rates are derived from clinical data describing the patients' preoperative severity of illness. During the year immediately prior to our survey, the 4 study hospitals received 3 distinct Consumer Guide ratings: 2 had a lower than expected mortality, 1 had higher than expected mortality, and 1 was within the expected mortality range. Similar to the range of hospitals statewide, the study hospitals' unadjusted in-hospital mortality rates ranged from 1% to 5%.

Each participating cardiac surgeon or group provided a list of patients who had undergone cardiac bypass surgery between July 1995 and March 1996 — the months following the June 1995 public release of volume 4 of the Consumer Guide. The overall sample included 1140 cardiac surgery patients. After excluding patients known to have died, we randomly selected 200 patients from each institution. To eliminate duplicate entries at1 hospital, we adjusted each hospital's sample to 196 patients. Human research review committees at each hospital granted permission to survey patients.

Using patient focus groups, expert advice, and formal pretesting, we developed a telephone survey to assess patients' perception of their decision making prior to surgery. The survey assessed 4 issues:

Telephone interviews with patients were conducted from June through December 1996 by Datastat (Ann Arbor, Mich). The statistical significance of differences in responses was assessed by a χ²test for binary response items and by a Wilcoxon rank sum test for pairwise comparisons of ordinal scaled responses. To evaluate the significance of associations between sociodemographic characteristics and awareness of the Consumer Guide, we calculated odds ratios (ORs) and 95% confidence intervals (CIs). Two-tailed P values are reported for all comparisons. More than 95% of respondents answered each of the items with the exception of the query about income (80%). Nonrespondents to specific questions were excluded from the analysis of those questions.

Of the 784 patients we attempted to contact, we completed interviews with 474 (60%). Among the original cohort, 111 patients (14.2%) could not complete the survey: 38 had died, 64 were too disabled, 7 had language incompatibilities, and 2 failed to recall having had an operation. Another 137 otherwise eligible patients (20.3%) refused participation, and 62 patients (9.2%) could not be contacted. The response rate among eligible patients was 70.4% (range, 68.7%-74.0% among the participating hospitals).

Characteristics of the respondents appear in Table 1. Comparing the frequency of each characteristic across the 4 hospitals, respondents differed in education levels (P<.01), in number of days between deciding an operation was needed and undergoing the operation (P=.03), and in the proportion reporting the following sources of payment for the operation: private insurance (P=.02), Blue Cross/Blue Shield (P=.03), and health maintenance organization (P<.01). Respondents from different hospitals also varied with respect to the factor most influencing their choice (hospital vs surgical group vs surgeon) (P<.01). Respondents were similar with respect to age, sex, marital status, self-reported health status prior to surgery, income, number of prior catheterizations, and length of time with heart disease. They were also similar in the proportion of those reporting that Medicaid or Medicare paid in part for the operation.

Table 2 summarizes the number and proportion of patients reporting awareness, knowledge, and use of the Consumer Guide. Ninety-three of the patients (20%) were aware of the Consumer Guide, and 56 (12%) of those said they knew of it prior to their operation. Two thirds of these patients (n=37) had only heard of the guide, while one third (n=19) had actually seen a copy. Eighteen (4%) reported knowing the hospital's categorical mortality rating (higher than, lower than, or within the expected number of deaths). Eleven (2%) reported that the information influenced the choice of hospital, but only 4 of these knew the correct categorical rating, which amounted to less than 1% of all respondents. Only 6 (1%) reported discussing the ratings with a physician.

Similarly, very few patients reported knowing the Consumer Guide' s categorical rating of the surgeon or surgical group (n=7). Four patients claimed that the Consumer Guide was a major or moderate influence on the choice of surgeon or knew the correct categorical rating of the surgeon or surgical group. Altogether, only these 4 patients reported that the Consumer Guide was a major or moderate influence on the choice of hospital or surgeon and reported the correct categorical mortality rating of the hospital, surgeon, or surgical group.

Table 3 displays patient characteristics correlated with awareness of the Consumer Guide prior to surgery. Patients were significantly more likely to report awareness of the Consumer Guide prior to the operation if they were younger than 65 years (OR, 2.00; CI, 1.14-3.51), had attended college (OR, 2.10; CI, 1.19-3.70), reported poor or fair preoperative health status (OR, 1.88; CI, 1.06-3.33), or reported having heart disease for more than 1 year (OR, 1.91; CI, 1.05-3.50). Men were somewhat more likely than women to be aware of the Consumer Guide prior to surgery (OR, 2.03; CI, 0.96-4.27), and patients with incomes greater than $30000 were also somewhat more likely to be aware (OR, 1.81; CI, 0.97-3.38). Rates of awareness of the Consumer Guide did not differ significantly among patients operated on in hospitals with categorical ratings higher than, lower than, or within the expected mortality range, nor were they related to whether the patient had previously been admitted to the same hospital or to the number of days between the decision to operate and the date of the operation. In a logistic regression analysis with "being aware of the Consumer Guide prior to surgery" as the dependent variable, younger age (P<.01), higher attained education level (P<.01), and higher health status (P=.02) were statistically significant predictors of "being aware" in the final model.

Table 4 shows findings on 3 measures of patient interest in the Consumer Guide. After the content of the Consumer Guide was described to all patients, 264 (56%) reported being somewhat or very interested in seeing a copy if they required another operation. Younger patients (P=.0002), those having some college education (P=.003), and those who were aware of the Consumer Guide prior to surgery (P<.05) were most likely to be somewhat or very interested in seeing a copy if they needed another operation. There was no significant difference in level of interest between patients who were and were not aware of the Consumer Guide at the time of the survey.

Most patients reported that they probably or definitely would change surgeons if they learned that their surgeon had a higher than expected mortality rate in the previous year. Nearly one third of patients said they would definitely change surgeons under this scenario. Nevertheless, one third of the patients reported that they would not be willing to pay any money to see a copy of the Consumer Guide . Thirty-five percent reported that they would be willing to pay at least $20 to see a copy. Only 8% said they would be willing to pay $50 or more.

Table 5 provides data on selected barriers to consumer choice for cardiac surgery patients. Thirty-eight percent had fewer than 3 days to decide on a hospital or surgeon before their operation. Only 12% of all the patients surveyed perceived that they had less than enough time to learn about the surgeon and hospital. However, 19% of the patients with fewer than 3 days to decide perceived that they had less than enough time, while 7% of the patients who had more than 7 days perceived that they had less than enough time (P<.01). Thirty-three percent of patients reported that there was no alternative hospital within a reasonable distance. Sixty-six percent of all the patients considered distance somewhat or very important in determining their choice of hospital, and these patients were more likely to report that there was no alternative CABG surgery hospital within a reasonable distance of their home (38% vs 23%, P<.01).

Forty-three percent of patients remained in the same hospital from the time it was decided that they would need an operation until the operation was performed. Only 2% reported that cost played any role in the choice of hospital, and only 4% perceived any restriction imposed by managed care insurance.

We are unaware of any previous studies of patient use of outcome data to choose physicians and hospitals.²⁰ Because of the extensive publicity given to the Pennsylvania Consumer Guide to Coronary Artery Bypass Graft Surgery, its 5-year track record, the salience of a major heart operation, and the 5-fold variation in mortality rates among hospitals, we expected that the Consumer Guide would be widely used by patients selecting providers for CABG surgery. We found just the opposite. It is striking that even among those who were aware of the Consumer Guide before surgery, almost no one used it in decision making.

What could account for the lack of awareness and use of the Consumer Guide among cardiac patients? First, referring physicians are a very important source of information about the quality of surgical specialists. Our previous survey of cardiologists⁷ and a similar study conducted in New York State²⁷ showed that very few of these providers discussed the Consumer Guide with patients, citing skepticism about the accuracy of its methods. The present survey confirms that these discussions are indeed rare.

As in New York State, the process for dissemination relies primarily on media, such as television and newspapers. Unlike a hospital quality reporting program in Cleveland, Ohio, the Consumer Guide is free. However, efforts to distribute it to patients appear to have been inadequate. It is possible that budget constraints, criticism of technical aspects of the reports, and political pressure from hospitals and physicians in Pennsylvania have deterred more aggressive dissemination of the Consumer Guide by the Pennsylvania Health Care Cost Containment Council. However, poor distribution alone cannot explain our observation that very few patients who were aware of the Consumer Guide ratings were able to comprehend and make use of them accurately.

A significant number of patients face serious constraints in their ability to seek and use the Consumer Guide. We found that most patients have a limited amount of time for decision making. Many perceived that there were no alternative cardiac surgery hospitals within a reasonable distance despite the fact that the hospitals we studied were relatively near other hospitals that provide CABG surgery. Finally, some patients may be skeptical of the value of such data. A recent survey of Americans' use of quality data on health plans found that the public values anecdotal reports from such trusted sources as relatives and friends more than objective reports from such sources as the government and the news media.²⁸

Our study provides conflicting information about patients' interest in the sorts of quality data that are frequently suggested to be useful to consumers. Although few patients used the Consumer Guide, a much larger number expressed interest in seeing a copy when it was described to them. One third of patients said they would definitely switch surgeons if they found that their surgeon had a higher than expected mortality rate. On the other hand, one third of them were unwilling to pay any amount to see the Consumer Guide, and most were unwilling to pay more than $20. Patients may view such information as a public good that should be inexpensively available.

Of course, public reporting of performance data may help improve quality of care even if patients do not use the data in selecting providers. Both employers and insurers may use such data in contracting decisions. Hospitals may use the reports to select physicians and curtail physician privileges.^29- 30 Health care providers may use the reports to identify specific clinical areas for quality improvement efforts and gauge their success.⁶ Nevertheless, providing data on quality directly to consumers to inform them as they choose providers is a notion with very wide political and popular appeal.¹⁵

Our study has several limitations. We surveyed patients from only 4 hospitals. These hospitals or the patients they serve may differ from other hospitals or patients. However, if willingness to participate in our study signals a more sympathetic attitude toward the Consumer Guide, then estimates of awareness and use might be even lower in other hospitals or patient groups. We surveyed patients after surgery. Some respondents may have forgotten their exposure to the Consumer Guide or may have reported that they were aware before surgery when in fact they only learned of the Consumer Guide afterward. Although we surveyed patients relatively soon after surgery, we cannot exclude the possibility that recall bias may have artificially lowered our estimate of awareness and use of the Consumer Guide among cardiac surgery patients. Another limitation is the inherent challenge of interpreting the responses of consumers regarding their interest in a publication that few have directly seen. We also had limited power to examine differences among hospitals. Our study had a power of 0.80 to detect a 15% absolute difference in rates of awareness (10% vs 25%) among patients at the 4 hospitals. Finally, our design precluded an evaluation of patients who considered but did not have surgery or who went to cardiac surgery centers outside of Pennsylvania after reading the Consumer Guide.

Despite these limitations, we found formidable evidence that public reporting of mortality outcomes in Pennsylvania has had virtually no direct impact on patients' selection of hospitals or surgeons. Nevertheless, a substantial number of patients expressed interest in data on mortality outcomes and claimed that they would use such reports in their decision making. Clearly, measurement and public reporting of physician and hospital performance is only a prelude to serving this interest. Existing quality measurement efforts have been criticized for methodological reasons.^31- 32 Although the methodological barriers to reliable and valid performance measurement are substantial, delivering performance information to patients in an effective and usable format could prove even more formidable. Further efforts to develop quality information for general public use should explore the use of Internet-based and other media for communicating quality information. Providers may also play an important role. Without a tailored and intensive program for dissemination and patient education, efforts to aid patient decision making with performance reports are unlikely to succeed.