Pay-for-Performance Research: Title and subTitle BreakHow to Learn What Clinicians and Policy Makers Need to Know

Pay-for-performance has become increasingly common in health care. According to recent surveys, almost 100 pay-for-performance initiatives nationwide are now sponsored by a variety of health plans, employer coalitions, and government purchasers that are intended to improve the quality of care provided.¹

However, the rationale for pay-for-performance comes almost entirely from experience with incentives in other industries. Only 9 randomized controlled trials (RCTs) of pay-for-performance have been reported in the literature.² Even these studies are not clearly applicable to current pay-for-performance because they all focus on performance for a single indicator or a single aspect of care (eg, preventive care), whereas most current pay-for-performance initiatives use multiple indicators for multiple conditions and types of care.¹ Moreover, most studies have important flaws in reporting, such as the failure to note the market share of the organization offering the incentive in the practices of the providers being studied. Because pay-for-performance programs have targeted both individual clinicians and clinical organizations such as hospitals, medical groups, and nursing homes, the collective term “provider” is used herein to refer to all potential parties.

In this issue of JAMA, Rosenthal and collagues³ offer an excellent example of how this research should proceed. The investigators evaluated a natural experiment using administrative reports from 2 groups, one of which received pay-for-performance incentives for cervical cancer screening, mammography, and hemoglobin A_1C testing in diabetic patients. They found that paying to reach a common, fixed performance target produced little gain and largely rewarded those with high performance at baseline.

While the study by Rosenthal et al is well designed, much more is needed. Pay-for-performance involves a common problem in health services research: despite little evidence, clinicians and policy makers are responding to a major policy trend, while researchers determine how to inform those decision makers. In this context, investigators and research funders need to develop strategies that address 4 fundamental aspects of research: study design, selecting theory-driven hypotheses, reporting research findings in a complete and informative manner, and setting research priorities. Until these issues are clearly addressed, clinicians should be skeptical of any research that purports to describe the impact of pay-for-performance.

Some have likened the state of pay-for-performance research to phase 1 clinical trials, in which basic issues of safety and dosing are considered. In the sense that pay-for-performance research is nascent this is appropriate, but in important ways the analogy is not apt. In particular, the expectation with clinical research is that there will eventually be large RCTs in which the effects of the intervention are isolated from other potential confounders. In health services research, however, RCTs are rare, and pay-for-performance is no exception.

A recent review of research in progress supported by major health services research funding agencies and foundations showed no ongoing RCTs of pay-for-performance.² This reflects in part the difficulty of enrolling health care organizations or professionals in RCTs that could affect their practice patterns or income. Furthermore, RCTs reduce the impact of confounding variables only when there are many subjects in each group of the trial. Since many pay-for-performance programs pay hospitals or medical groups, these organizations (not individual patients or clinicians) are the unit of observation. For most pay-for-performance projects, it would be difficult for researchers to enroll sufficient numbers of organizations to ensure that randomization really balances confounding factors.

In the absence of RCTs, researchers must carefully consider study design. The primary alternative to RCTs involves a payer deciding to implement pay-for-performance in one component of its network but not in others. Rosenthal et al report on one such natural experiment.³ In such studies, decisions to introduce pay-for-performance in one part of a network reflect business judgments that probably limit the generalizability of the results. At this early stage in its development, payers may choose to implement pay-for-performance where they judge its chances of success to be highest. Thus, if pay-for-performance works in the portion of the network the payer selected, it cannot be assumed that it will work in the rest of the network (in which the payer initially believed pay-for-performance would be more difficult to implement and/or less successful). Alternatively, as was the case in the study by Rosenthal et al, the payer may implement pay-for-performance for an entire region.³ In this situation, negative results do not imply that pay-for-performance would not work if the strategy were applied to a more targeted group of providers or with a different set of incentives.

Some study design approaches reduce the impact of the confounders that drive payer choices about where to use pay-for-performance. For instance, a payer might start pay-for-performance in a location because several large health care organizations there had better clinical registries of patients with chronic diseases, while all other areas in the network had few organizations with registries. To assess the impact of the pay-for-performance program, instead of comparing the test location with the rest of the network, the researchers might compare the change in performance among the test location organizations with good registries with a select group of organizations from various places in the network that were of similar size and had similar registries.⁴ This approach requires understanding the payer rationale for choosing sites for pay-for-performance and considering that rationale in study design, but could significantly reduce the risk of biased results, even without RCTs.

Given that pay-for-performance research will be primarily observational, careful reference to theories about incentives will be necessary to identify potentially confounding variables that might differ between the intervention and control groups. Theories about the use of incentives are extensive and come from the economics, psychology, and organizational behavior literature.^{2 ,5} However, it is clear that characteristics of the incentive offered—such as its magnitude and whether it is structured as a reward or a penalty—can be crucial in the determining response. Furthermore, many factors beyond the pay-for-performance incentives themselves can enhance or inhibit their effects (Table), such as the other incentives providers face aside from the new pay-for-performance program—especially the overall approach to paying for services. A pay-for-performance financial incentive to increase mammography rates in asymptomatic women aged 50 through 70 years might be more likely to succeed in a fee-for-service environment (in which positive results yield subsequent services and additional fees) than in a capitated environment (in which even false-positive results must be evaluated within the preset budget of capitation).

Nonfinancial factors also matter. Improvement in quality is strongly related to the activity of accrediting organizations such as the Joint Commission on Accreditation of Health Care Organizations (JCAHO).⁶ Therefore, a pay-for-performance program that used JCAHO measures might show improvement, but the gains might not reflect the impact of pay-for-performance but rather secular trends in performance indicators on which clinicians were intensely focused for other reasons.

Similarly, characteristics of the organizations targeted for pay-for-performance could be significant. For instance, response to a pay-for-performance program designed to improve several aspects of diabetes care might be greater among medical groups with good information technology and a real-time reminder system than in organizations without such capacities. Patient variables also could be important. Health plans with a high concentration of patients of lower socioeconomic status or lower educational levels may face greater challenges than others in patients’ access to appropriate medications and treatment adherence.⁷

Thus, in selecting intervention and control groups, it is not sufficient to have novel study designs. Rather it is crucial that subject selection take into account theories of how incentives might work.

Performing rigorous research will be useful only if the findings are subsequently described in such a way that both clinicians and policy makers can draw justified conclusions. This requires complete reporting of all variables that could be important determinants of the effect observed, since each could be an important confounder if not explicitly considered. While this might seem obvious, it is not reflected in the extant literature. The article by Rosenthal et al³ is the first report of a pay-for-performance study in which the authors describe the percentage of clinicians’ patients for whom the incentive is relevant.³ Similarly, no prior pay-for-performance research has reported the cost of improving quality or how that compares with the incentives offered in the pay-for-performance program. That is, if incentives to achieve smoking cessation are less than the mean cost of actually getting patients to quit, the health care organization must still expect to lose money on average by aggressively pursuing smoking cessation. It should not be surprising if such an “incentive” does not have much effect.⁸

Other key variables that have been ignored in most reports about pay-for-performance programs include organizational factors such as the adoption of guidelines and most clinician characteristics (eg, the specialties studied).²

The difficulty of performing good research in this area suggests that research funders also need a clear strategy to push the field forward. In particular, funders should consider soliciting research that explicitly enables sequential hypothesis testing of fundamental characteristics of incentives offered in pay-for-performance and parallel hypothesis testing of other key enabling or inhibiting factors.

With a sequential hypothesis-testing strategy, funders would ensure that research moves steadily from situations in which incentives are most likely to have some positive effect (because the magnitude of the incentive is large relative to the costs of improving quality) to situations in which success is less preordained. For example, initial evaluations could target instances in which the pay-for-performance program targeted tasks that reduce resource utilization in a capitated environment (eg, pay-for-performance for influenza vaccination, which reduces the subsequent risk of expensive illnesses for which the health care organization might bear financial responsibility).

Other initial projects could address pay-for-performance targeting quality improvement that increases subsequent resource utilization in a fee-for-service environment (eg, pay-for-performance for smoking counseling, assuming that follow-up visits would also generate new fees). In these examples, the finding that pay-for-performance improved quality cannot be generalized to less favorable situations, but a negative result would signal that pay-for-performance is unlikely to be effective under any circumstances.

After these initial studies, researchers could evaluate situations in which the incentive is weaker, either because the pay-for-performance payment is smaller (analogous to a dose titration in clinical research) or because the costs of improving quality are greater, such that the net incentive is weaker. This might mean a second study assessing the impact of pay-for-performance either with smaller payments or for tasks with greater difficulty (eg, improving blood pressure control among hypertensive patients) would be necessary. As subsequent studies are carried out, policy makers, clinicians, and researchers will learn more about how to structure incentives to improve quality.

As that research proceeds, funding agencies should concurrently support studies for parallel hypothesis testing, ie, studying how factors external to the incentive—market, regulatory, organizational, or patient variables—mitigate or augment the impact of pay-for-performance. For instance, since information technology is so frequently cited as a key aspect of the infrastructure needed to improve quality, funders should support research about how organizations with different information technology capacities perform under pay-for-performance initiatives.

If funders support sequential and parallel hypothesis testing, they can ensure that clinicians and policy makers will come to understand the nuances of when and how incentives work. Since almost every physician and hospital in the United States is now a potential participant in some form of pay-for-performance program, such research is essential.