Does Comparative Effectiveness Have a Comparative Edge?

Researchers, policy makers, insurers, and other stakeholders have voiced enthusiasm about the value of comparative effectiveness research that rigorously evaluates 2 or more drugs or devices. The most recent boost for these efforts has been the US congressional financial stimulus package that contains provisions for $1.1 billion to be devoted to this effort.¹ The appeal of comparative effectiveness research is undeniable. If there is one issue that stakeholders agree about, it is that increasing health care costs are ultimately unsustainable and society needs more value for its money. However, it is not clear that comparative effectiveness research, as it is commonly framed, has a comparative advantage when it comes to improving the US health care system.

If comparative effectiveness research is to succeed, future initiatives will need to (1) generate data prior to the widespread adoption of a drug or treatment; (2) link evidence directly to strategies proven to modify clinical practice; (3) expand the research agenda far beyond drugs and devices; (4) incorporate the principles of comparative effectiveness research throughout the process of US Food and Drug Administration (FDA) approval and regulation; and (5) examine the costs, as well as the effectiveness, of treatment alternatives (Table).

The historical method of generating comparative effectiveness information in the United States has largely relied on publicly and corporately funded researchers to produce it. This effort, however, has often not only lacked coordination, rigor, and objectivity, but also timeliness. As a result, this approach has failed to curtail the widespread adoption of pharmaceuticals and devices beyond the clinical populations that gain the most from them.² In many cases, therapies have been prematurely adopted, outpacing the generation of evidence necessary to define the boundaries in which a drug or device offers clinical benefit. Atypical antipsychotics are the latest example, with rapid adoption and expanding use at least a decade before the relatively recent consensus about the similar efficacy of typical and atypical agents and the full recognition of the previously underappreciated metabolic and cardiovascular adverse effects of the atypical agents.^{3 - 4} Enthusiastic adoption of innovations, only later found wanting, has been a recurrent problem, with examples far beyond short-acting calcium channel blockers for hypertension, troglitazone for diabetes, tegaserod for irritable bowel syndrome, and rofecoxib for mild to moderate pain.⁵ Of course, during the past decades, many innovations have been brought to market that have withstood the test of time. But is this a chance that society needs to take?

While timely, coordinated, and rigorous comparative effectiveness data are needed, these data are not sufficient to improve the clinical application of drugs and devices. The substandard clinical use of pharmaceutical agents and devices results often as much from a collective inability to translate evidence into practice as it does from insufficient knowledge. Despite rigorous evidence, outdated clinical strategies often persist while many effective therapies fail to be broadly adopted.⁶ Such inadequate diffusion is commonplace, ranging from the use of home blood pressure monitors, inhaled corticosteroids to treat persistent asthma, warfarin for atrial fibrillation, angiotensin-converting enzyme inhibitors for congestive heart failure, and β-blockers and aspirin for secondary prevention of coronary artery disease. Just as there are compelling examples of underuse, so too overuse and misuse are not a function of knowledge deficits alone. About half of patients with viral upper respiratory infections receive antibiotics and overuse occurs in numerous other contexts, including the use of atypical antipsychotics in dementia and cardiac pacemaker implantation for marginal indications.

These patterns that defy the clinical evidence are due to dozens of nonclinical factors that influence treatment choice. More than $20 billion is spent annually on marketing and promotion of pharmaceuticals, representing nearly 63-fold more than the entire budget of the Agency for Healthcare Research and Quality ($319 million in 2007), the lead federal agency charged with improving the quality, safety, efficiency, and effectiveness of health care delivery in the United States. Although such promotion and marketing can facilitate the uptake of important new therapies, they can also lead to substantial shifts in the use of technologies unsupported by clinical evidence.

Other important barriers prevent translation of knowledge into practice. Physician lack of awareness of the evidence and countermanding local standards of care pose formidable hurdles. Social and psychological obstacles^{7 - 8} also play an important role, including widely held beliefs that clinical value is automatically associated with more costly products,⁹ newly introduced products,¹⁰ or the very act of recommending a clinical intervention. As a whole, these barriers contribute to systems of care that prevent delivering the right therapy at the right time¹¹ and suggest the failure of rational choice models in which information is seen as the prime motivator of clinician behavior. Why should data from comparative effectiveness research have a comparative edge in overcoming these barriers?

Initiatives are needed to ensure that comparative effectiveness data can be successfully used to increase the value of health care services. Such efforts must begin with the recognition that knowledge of the comparative benefits and risks of 2 drugs or devices alone is seldom sufficient to change behavior. Because of this, broadening the comparative effectiveness agenda, modifying the regulatory environment that shapes the production of data, and developing cost-effectiveness research to guide coverage decisions are all crucial if comparative effectiveness research is to have a chance of substantially improving health care delivery.

Most discussions of comparative effectiveness research focus on comparing 2 or more pharmacotherapies or devices, yet comparative effectiveness research should more broadly encompass many higher levels of health care delivery and organization. For example, comparisons of procedures with each other (eg, percutaneous coronary intervention vs coronary artery bypass grafting) or with medications (eg, medical management vs percutaneous coronary intervention) may be particularly important because of the high cost of these procedures. Equally important is the inclusion of strategies that involve patient behavior (eg, diet or physical activity) or that are often initiated without physician input (eg, complementary and alternative therapies). Comparative effectiveness research also must be used to identify the best methods to improve health care quality, such as the value added by health information technology (eg, manual vs computer order entry), and even modified systems of care such as use of nurse disease managers.

Despite recent enthusiasm for comparative effectiveness, little of the underlying premise of this research has been incorporated into the approach for drug and device approval within the FDA. The process of technology evaluation in the United States stands in stark contrast with other industrialized countries because the primary method of drug and device regulation used by the FDA has a framework that is mismatched with the needs of funders, payers, and other policy makers.¹² The FDA's historical focus on common harms and on evaluating efficacy against placebo has led to testing in small, highly selected populations with limited comorbidities. In turn, these studies have failed to provide information most relevant to the clinical contexts in which FDA-approved drugs or devices are ultimately used.

Insofar as comparative effectiveness is important, reliance on placebo-controlled trials as the sole mechanism of approval is insufficient. A drug with no objective advantage over other available drugs can enter the market easily when the threshold is performance relative to placebo. This process may favor look-alike drugs and allow for market differentiation based on characteristics (eg, capsule color) that obscure potential clinical comparison based on effectiveness. It also may be beneficial to require drug labeling that indicates nonsuperiority. As an example, a package insert might state: “This drug has not been found to be superior to the other calcium channel blockers in the treatment of hypertension.” Such statements might better inform prescribers and patients, while providing industry with much-needed motivation to perform trials using active comparators (personal communication, Todd Wagner, PhD, August 10, 2007).

The most contentious difference between the perspective of the FDA and that of other stakeholders has been the role that treatment costs should play in government-sponsored research and in guiding coverage decisions. For instance, the Sentinel Initiative, which has the goal of developing a national, integrated, electronic medical record system for monitoring medical product safety, omits discussion of costs and evidence from its aims.¹³ Even in the private sector, there has been a general reluctance to develop and use cost-effectiveness research to guide coverage decisions. There are good reasons for this, including concerns that access to care will be impeded, fears that current analytic methods lack adequate rigor, apprehension that innovative research and development will be stifled, and the potential for litigation regarding limitations imposed by cost-effectiveness analysis.¹⁴ Nevertheless, what good is comparative effectiveness research if it cannot be used to discern anything about value to clinicians, insurers, patients, and society?

Despite the allure, no amount of comparative effectiveness data alone, regardless of how rigorously assembled, will suffice to fundamentally transform clinical practice. Serious review of the failure of past efforts to improve practice suggests that comparative effectiveness research is at high risk of a similar fate. Without attention to timeliness, transforming evidence into practice, inclusion of strategies beyond drugs and devices, minimizing regulatory mixed messages, and the comparative costs of therapies, current investments in comparative effectiveness will fall far short of their ultimate potential for improving the health and health care of all. The primary problem is not the absence of knowledge regarding comparative effectiveness, but the absence of the necessary mechanisms to put this knowledge to work.