The Public Release of Performance Data:  What Do We Expect to Gain? A Review of the Evidence FREE

Information about the performance of hospitals, health professionals, and health care organizations is increasingly being released into the public domain.¹ The data, often produced in the form of "report cards," "provider profiles," or "consumer reports," necessitates the development and dissemination of standardized reports on quality of care and facilitates comparisons of performance over time, among providers, and against defined standards of good practice. Health care performance data have been made public in the United States for more than a decade,² and the production and dissemination of report cards is now a multimillion-dollar industry. However, evaluation of the impact of report cards has not kept pace with the development of reporting systems.^3- 7 In addition, there has been minimal agreement among the various stakeholders about expected gains from the release of comparative performance data. It has therefore proved difficult to judge whether the benefits of public disclosure of performance data outweigh the disadvantages.

The failure to articulate a clear rationale for the public release of performance data is illustrated by the wide variety of ways in which commentators believe the data might be used. The most common expectation is that such data will promote an efficient market economy in health care,^8- 9 usually in the belief that information about performance will encourage consumers to choose to access high-quality providers.^10- 13 Some authorities suggest that the information could be used by providers as a marketing tool.¹⁴ Others suggest that it will help to control costs^15- 16 or at least counter the influence of cost as the principal determinant of purchaser decision making.^17- 19 In addition, public information about performance has been proposed as a tool to regulate the health system,^10,20 a method of ensuring accountability of provider organizations,^14,21 or of making judgments about the performance of individual professionals.⁴ Finally, some perceive public disclosure to be a mechanism to promote quality improvement by informing purchasers¹⁸ or encouraging providers to focus on quality problems.^{10,14,22- 23}

Public disclosure therefore has been advocated as a mechanism to regulate providers of care, ensure accountability, inform, promote quality improvement, and encourage cost control. These diverse aims have made it difficult to identify evaluation criteria that can be used to assess its impact. In this article, we examine the research evidence to determine the extent to which the expectations of public disclosure have been realized and identify a future research agenda.

An extensive review of published articles in peer-reviewed journals was conducted between October 1998 and October 1999 using MEDLINE and EMBASE electronic databases. We limited our review to evaluations of US reporting systems published between January 1986 and October 1999. While organizations may carry out evaluations without publishing the results, we considered peer-review publication an important marker of the rigor of such evaluations and therefore restricted our search to published reports. Searches were conducted independently by 1 of the authors (M.N.M.) and a professional librarian using the following Medical Subject Headings: report cards, public performance reports, provider profiling, public/consumer/patient information, and consumer reports. Only original research articles were reviewed. The citations in all articles were searched. Authors of the principal published studies and other experts in the field were asked to recommend relevant published studies. In addition, documents and Web sites prepared by a range of public and private organizations were searched for references to original evaluative studies relating to public disclosure.

There are many reporting systems in the United States but only 7 of these systems appear to have been subjected to rigorous empirical evaluation. A total of 21 peer-reviewed publications were found relating to these 7 systems (Table 1). Descriptive, observational, and quasi-experimental study designs were most commonly used; there was only 1 randomized controlled trial. For the purpose of this review, the literature has been categorized into use of public reports by the key stakeholders (consumers, purchasers, physicians, and provider organizations), the impact on quality of care outcomes, and the financial costs (Table 2). The term "provider" refers principally to hospitals in the studies. The studies cited in Table 1 that are relevant to the data extraction categories are described in greater detail in Table 2. In addition to these 21 studies, 5 other articles are cited that describe general issues relating to public disclosure but are not specific to any 1 reporting system.

There is a growing body of evidence that consumers want more information about health care provider performance and are able and willing to identify the content and format of the information of greatest use to them.^9,36 However, evidence suggests that the information has only a limited impact on consumer decision making.

Consumer use of the Pennsylvania report card, which publishes hospital- and surgeon-specific data on coronary artery bypass graft (CABG) mortality, was assessed using a telephone survey of patients who had undergone CABG surgery in the previous year in 4 Pennsylvania hospitals.¹² Two of the hospitals had lower and 1 higher than expected mortality rates. Of 474 patients surveyed, 56 were aware of the performance report at the time of their surgery and less than one quarter of these stated that it had any significant impact on their choice of surgeon. Most of these patients were unable to correctly specify the data on which they based their choice of surgeon. When the report was described, more than half of the respondents said that the information would have influenced their choice of surgeon, but there was a low level of willingness to pay for it.

Hibbard and Weeks¹¹ studied the impact of comparative data about physician fees on consumer use of services. The authors randomly allocated Medicare enrollees and state employees in Oregon to receive information about physician fees. No statistically significant difference in physician visit rate, expenditures on ambulatory care, or costs per visit was observed between the intervention and control groups. Other studies have shown that consumers continue to use hospitals with high mortality rates^27- 28 and that willingness to use hospitals is more likely to be influenced by anecdotal press reports of untoward deaths in hospitals than by risk-adjusted mortality data.²⁸

The only study suggesting that public disclosure had a significant effect on consumer decision making evaluated the impact of the New York State CABG mortality report on hospital and physician market share and price changes before and after publication of the data.¹⁹ This quasi-experimental study indicated that hospitals and physicians with better outcomes experienced higher rates of growth in market share and that physicians with better outcomes had higher rates of growth of charges for the procedure. The magnitude of the association declined over time, suggesting that the market responded primarily to new information.

The reasons for consumers' lack of interest in and use of performance data include difficulty in understanding the information,^35,37 disinterest in the nature of the information available,³⁹ lack of trust in the data,³⁶ problems with timely access to the information,¹² and lack of choice.⁴⁰ There is evidence from descriptive studies that consumers rate anecdotal evidence from family and friends more highly than empirical evidence.^28,36

Employers are the main purchasers of health care in the United States, both as individual corporations and in coalitions that have considerable aggregate purchasing power. Policy analysts have expected these coalitions to be more responsive to public performance data than individual consumers, but current evidence suggests that public disclosure has only a small, although possibly increasing, effect on purchasing behavior.

The National Committee for Quality Assurance (NCQA) is responsible for 1 of the highest-profile reporting systems in the United States, the Health Employer Data Information Set (HEDIS) and the NCQA accreditation program. Its impact on employer choice of plan was examined by surveying large US employers.²⁹ The results were compared with a similar survey conducted by the same research team the previous year. The percentage of employers familiar with accreditation increased from 29% to 35%. Of employers, 11% considered NCQA accreditation to be very important, and 5% considered HEDIS data to be very important. Less than 10% required NCQA accreditation for plan selection and 1% provided the HEDIS data to their employees to help them choose a plan. Only marginally greater use of performance data was found in a similar study by Hibbard et al.¹⁸

The acceptability to and use of public performance data by physicians may be an important factor in the successful implementation of a public disclosure policy. The evidence suggests that physicians are interested in but skeptical about such data and that they consider it to be of minimal use.

A survey of cardiologists' attitudes to and use of the New York State CABG mortality data revealed that most cardiologists found the reports easy to read and considered the data to be very or somewhat accurate.³² Twenty-two percent routinely discussed the data with their patients, and 38% considered it to have influenced their referral pattern. The most common objection to the reports was a perception that it discouraged cardiac surgeons from operating on high-risk patients.

A survey of cardiologists' and cardiac surgeons' responses to the Pennsylvania CABG reporting system produced similar results²³—a minority considered the data to be important when assessing the performance of a particular surgeon or discussed the data with their patients. Almost two thirds of cardiologists reported increasing problems finding surgeons to operate on high-risk patients, and the same proportion of cardiac surgeons reported that they were less willing to operate on such patients.

There is some evidence that provider organizations are more responsive to performance data than consumers, purchasers, or individual physicians.

A quasi-experimental design was used to examine the impact of an obstetrics consumer report on hospital behavior in Missouri.¹⁴ Half of the hospitals that did not have an infant car seat program, formal transfer arrangements, or breastfeeding nurse educators prior to publication of the report instituted or planned these services after the report was published. Hospitals in competitive markets were twice as likely to implement changes than those with monopolies. Several clinical outcome indicators improved after publication, including patient satisfaction and cesarean delivery rates.

Studies conducted in organizations exposed to publication of data about their performance have produced mixed assessments of the use of the information. Some organizations, particularly those whose performance is shown in a positive light by the reports, use the information for benchmarking, to promote collaboration across departments, and for internal monitoring of performance.^8,20,33 In contrast, other organizations respond more negatively, particularly in the early stages of release of the data.^{15,22,24- 25} Providers that have been identified as poor performers are more likely to criticize the validity of the data and the resource implications of acting on it.²⁴

The most fundamental question relating to public disclosure is the extent to which it influences outcomes of care. Three studies specifically attempted to address this issue, all of which used quasi-experimental designs.

Hannan et al¹⁰ studied all patients undergoing isolated CABG surgery who were discharged between 1989 and 1992 from New York hospitals where the procedure was performed. A clinical database was used to identify significant independent risk factors. Actual mortality decreased by 21% during the study period. Illness severity of the patients being operated on increased during this time so that risk-adjusted mortality decreased even further, by 41%, which was considerably higher than the national average. The study occasioned considerable debate among academics, and several further studies attempted to support or refute the authors' conclusions that publication made a significant contribution to the observed improvement.^{31,34,41- 44}

Potential explanations for the New York data have been suggested³¹ and relate largely to changes in hospital policy resulting from public disclosure.^30,43 First, an exodus of low-volume and high-mortality surgeons followed the publication of the data, probably as a result of hospitals restricting operating privileges. Second, a marked improvement in the performance of non–low-volume surgeons was observed. Finally, there was an improvement in the performance of surgeons new to the system.

The main criticism of the release of data in New York was that the data could have reduced access to CABG surgery by forcing sicker patients to seek surgery outside the state or by surgeons refusing to operate on high-risk patients. Peterson et al³⁴ used national Medicare data to examine trends in the percentage of New York residents receiving out-of-state surgery before and after initiation of the provider profiling program. They also examined procedure use by elderly patients with myocardial infarctions within the state to determine whether high-risk patients were being refused treatment. Contrary to a previous single-center study,⁴¹ Peterson and colleagues found that the percentage of New York residents receiving out-of-state bypass operations decreased between 1987 and 1992 and the likelihood of bypass surgery following myocardial infarctions increased. They confirmed a reduction in 30-day mortality in New York well above the national average during the period studied. The only other area with similar figures was northern New England, where provider profile reports were also provided, although for internal use rather than for publication.⁴⁵

In an evaluation of the Cleveland Health Quality Choice project, Rosenthal and colleagues²⁶ reported improved outcomes associated with publication of mortality data for 6 common medical conditions and 2 surgical operations. The authors examined 101,060 discharges from 30 northeastern Ohio hospitals and found significant and sustained reductions in risk-adjusted mortality for most conditions following publication.

There has been little evaluation of the costs of publishing performance data, in part because some of the data are collected anyway for internal evaluation and administrative purposes and because of the multiple contributors to data collection, analysis, and dissemination. The only published estimate that could be found described the costs of producing data for the Pennsylvania reporting system.³⁸ Average costs were estimated to be $16 per patient discharge, at 1991 prices, to collect and report the data. Costs depend on the data system used, but given the initial costs of the development of measures, analytical methods, and data management systems and ongoing costs of data collection, analysis, auditing, dissemination, and management of the responses, this figure is likely to be an underestimate of costs at 2000 prices.

Public disclosure of performance data is a major health policy initiative that represents a potential challenge to health professionals' traditional concept of autonomy.⁴⁶ It is therefore surprising that after more than a decade of experience of public disclosure in the United States, so few current reporting systems have been the subject of so little published empirical evaluation.

There are several possible reasons for this lack of examination. One is an inevitable time lag between the introduction and evaluation of a new initiative. In addition, some people may regard public disclosure as so obviously the right thing to do that formal evaluation of its impact is not necessary, or the "political correctness" of challenging a tool of informed consumerism may discourage candid appraisal. Also, an evaluation of current reporting systems may be being delayed until more sophisticated data are available, and vested business interests may feel threatened by formal evaluation of an area in which there is considerable business potential. Moreover, funding agencies may not see evaluation as a priority, and the methodological problems of researching such complex interventions may be a deterrent.⁷

Whether the limited evidence summarized in this article is regarded as supportive of, or counter to, a policy promoting greater public disclosure is open to debate. Those who are in favor will focus on the small but significant gains and will question the sensitivity of conventional research methods to measure subtle improvements in quality of care. They may also point out that the main impact of the information will not be seen for several years, particularly if the quality of the data continues to improve. Antagonists will focus on the negative findings and will emphasize the small gains in relation to the considerable public and private investment of time and resources.

Figure 1 summarizes the issues pertaining to public disclosure in a conceptual framework. The evidence does not provide strong support for those who believe that public disclosure should focus on the role of the consumer or purchaser. Neither individual consumers nor group purchasers appear to search out, understand, or use the currently available information to any significant extent. If consumers are regarded by some as a key audience, then performance data should perhaps be aimed more at consumer advocates and interest groups than at individual consumers. The skepticism of physicians suggests that they too are unlikely to be key players in promoting or encouraging more effective use of publicly available data. The key stakeholders seem to be the provider organizations because the available evidence suggests that publication of performance data results in provider behavior change that contributes to observed improvement in both the processes and outcomes of care. Provider organizations are sensitive to their public image and potential legal risk and often have the authority to act on suboptimal levels of performance and promote good standards of practice. They therefore respond to performance data as a competitive opportunity or risk management imperative even though the purchasing side of the market is less sensitive to it. An intrinsic professional competitiveness on the part of the provider organizations, based on a desire to be seen to be performing well, might therefore be just as important a motivator as economic gain. If this is the case, the reasons for organizational providers being more sensitive to the data than individual physicians warrants further investigation.

This review of the published research reveals many unanswered questions. First, while few would question the importance of performance data as one component of quality improvement, there is considerable uncertainty about the relative merits and risks of using such data internally or making it publicly available. Second, the unintended and potentially adverse consequences of public disclosure have received minimal attention. There is anecdotal evidence that making performance data public can result in gaming and an inappropriate focus on what is being measured.^47- 48 The negative impact of publication of poor performance on public trust and professional morale⁶ has also been highlighted, but these issues require empirical investigation. Third, there is considerable doubt about the most effective and appropriate level for the reporting of performance data and the degree of risk adjustment of health outcomes required to achieve a balance between cost, effectiveness, and fairness to providers.^49- 50

The philosophical stance of those who evaluate the impact of public disclosure should be explicit. The narrowest perspective, and the one that has been most commonly adopted, would view public disclosure as a specific intervention and consider only those benefits and costs that relate directly to release of the data. A broader perspective, analogous to other examples relating to freedom of information, would consider implications for society, such as benefits arising from public discussion, damage to professional reputations, and resource implications resulting from demands to see the "best" provider. The broadest perspective would consider disclosure as a right in a democratic society, irrespective of the costs, and focus on release of data as an intended outcome, rather than a controllable intervention.

An increasing level of public disclosure of health care performance data is inevitable and reflects greater demands for information and accountability in all walks of life. We believe that the use of public performance data by consumers and purchasers or for regulation purposes, will remain relatively less important for the foreseeable future than use of the data as a catalyst to stimulate and promote internal quality improvement mechanisms at the level of the organizational provider. Articulation of such a clear purpose for public disclosure and the development of an evidence base to guide its implementation will ensure that disclosure is conducted in a way that will maximize the potential benefits and reduce the associated risks.