Users' Guides to the Medical Literature: Title and subTitle BreakXIX. Applying Clinical Trial Results
B. Guidelines for Determining Whether a Drug Is Exerting (More Than) a Class Effect

Most classes of drugs include multiple compounds. The opinions of clinicians, manufacturers, and purchasers may differ as to whether a particular drug is more efficacious, safer, or more cost-effective than others in its class.¹ In this article, we review the types of evidence commonly cited to support the prescribing of a particular drug rather than another of the same class and provide a hierarchy for grading studies that compare a drug with another of the same class, expanding on our discussion in part A of this Users' Guide.²

As a clinician, you care for many patients with elevated serum cholesterol levels. A speaker at a recent continuing medical education event reviewed the benefits of cholesterol-lowering therapy, particularly with 3-hydroxy-3-methylglutaryl coenzyme A reductase inhibitors (statins), in the primary and secondary prevention of ischemic heart disease but did not recommend a particular statin. You decide to consider statin therapy for all your patients with elevated cholesterol levels, but are uncertain which of the statins on the market is best. You ask a general internist, cardiologist, and endocrinologist for their opinions, and each suggests a different statin, citing different reasons. You contact pharmaceutical representatives to provide you with evidence that their statins are better than those of their competitors. Although you use the JAMA series on Users' Guides to the Medical Literature to assess the validity of published studies, faced with a variety of competing claims, you realize that you need a framework for grading the strength of these studies.

Your colleague, a purchaser for a large health maintenance organization (HMO), is faced with a similar dilemma when she is asked to consider replacing the statin on her HMO's formulary with a newer one. She wonders whether there is enough evidence to support the contention that the new statin is as good as, or better than, the one currently on formulary. While the new statin is cheaper, it has been evaluated only in short-term trials, with cholesterol lowering as the solitary end point.

Although there is no uniformly accepted definition of a drug class—and some argue that it cannot be defined at all—drugs are generally said to belong to the same class for 1 of 3 reasons (Table 1). Herein, we define a drug class as those drugs that share a similar structure and mechanism of action. Most classes of drugs include multiple compounds, and because of their similar mechanisms of action, they are generally thought to confer similar pharmacologic effects and clinical outcomes (class effects). This assumption is a key medical heuristic³ and underlies clinical practice guidelines in which evidence from studies involving 1 or more drugs within a class is extrapolated to other drugs of the same class. For example, it is recommended that β-blockers be prescribed for survivors of myocardial infarction or angiotensin-converting enzyme inhibitors to patients with heart failure. In this circumstance, clinicians are likely to be interested in the drug within each class with the most attractive efficacy-to-safety ratio; purchasers, in the most cost-effective drug from a class; and manufacturers, in the most frequent prescribing of their drugs.

The absolute treatment effects seen with a drug (defined by the absolute risk reduction or number needed to treat) are influenced by the baseline risk or control event rate of those patients in whom it is used. Thus, the absolute risk reduction varies considerably among different groups of patients. On the other hand, the relative treatment effect of a drug (defined by the relative risk reduction [RRR]) is often (but not always⁴ ) similar, irrespective of the baseline risk of trial participants.^{5 - 6} If 2 drugs are tested in separate placebo-controlled trials, only proportional effects such as the RRR resulting from each drug can be compared (and then only under the assumption of constant RRR over different control event rates). Although the point estimates of effect size vary, a class effect is considered to be present when drugs with similar mechanisms of action generate RRRs (or odds ratios [ORs]) that are similar in direction and magnitude. For example, the Collaborative Group on ACE Inhibitor Trials⁷ suggested that there is a class effect for angiotensin-converting enzyme inhibitors in patients with symptomatic heart failure, despite the fact that the OR point estimates for effects on total mortality ranged from 0.14 (95% confidence interval [CI], 0-7.6) for perindopril (1 trial, 125 patients) to 0.78 (95% CI, 0.67-0.91) for enalapril (7 trials, 3381 patients). We are confident in this class effect, because the overall OR in 32 trials involving 7105 patients was 0.77 (95% CI, 0.67-0.88), the CIs for each of the angiotensin-converting enzyme inhibitors overlapped, and there was no statistical heterogeneity between trials of different agents.

Although drugs of the same class typically exhibit similar pharmacological effects and clinical outcomes, this may not always be the case. Note the current controversy regarding the safety of sotalol hydrochloride in myocardial infarction survivors with congestive heart failure after the publication of the SWORD Trial,⁸ which suggested an increase in mortality with sotalol, compared with the decrease in mortality with other β-blockers. It is useful to recall a previous controversy regarding the efficacy of β-blockers with intrinsic sympathetic activity (ISA) in patients with myocardial infarction. Although a meta-analysis⁹ suggested that the treatment effect was greater with non-ISA β-blockers, subsequent trials¹⁰ failed to confirm this, and the evidence¹¹ suggests there is little difference between β-blocker subgroups. It would seem reasonable to accept a priori that drugs within the same class exert similar effects, unless there is clear evidence of important differences.

However, this assumption can lead to 2 important errors of extrapolation with major clinical consequences. First, when agents in a class of drugs (such as the thiazide diuretics) all produce similar pharmacological effects (blood pressure lowering) and similar clinical effects (stroke reduction), a second class of drugs (for example, the calcium channel blockers) that produce the same pharmacological effects might be assumed to produce the same clinical benefits. In the absence of randomized trials verifying that final assumption, this type of extrapolation may be erroneous. For example, consider the issue raised in part A of this Guide—some calcium channel blockers have unfavorable effects on total mortality.¹² Second, even within the same class, individual drugs may have physiologic effects other than the mechanism of action that defined them as being from the same class. It therefore may be inaccurate to extrapolate the clinical outcomes shown in randomized trials of 1 drug in a class to another member of that class that has not been subjected to similar outcome-centered trials. For example, some authors have argued that, although all of the statins act on the 3-hydroxy-3-methylglutaryl coenzyme A reductase enzyme, they may have different nonlipid effects on the atherothrombotic process that may influence their clinical efficacy.¹³

To reduce the risk of faulty extrapolation and to maximize the optimal selection of treatments within a class of drugs, it may be useful to develop and apply a hierarchy of evidence when making decisions about the comparative clinical efficacy and safety of drugs within a class. As pointed out in part A of this Users' Guide, no matter how strong the pathophysiologic rationale or indirect evidence, the efficacy and safety of a new drug must be established in clinical outcome studies that test more than just biological plausibility.

Levels of evidence are increasingly used by groups that make recommendations about patient care,^{14 - 16} and we have used some of them to develop guidelines for comparing 1 drug with other drugs in the same class (Table 2). This comparison should occur as part of a systematic review of all the relevant evidence on the effects of a treatment, identified and assessed by thorough and clear methods such as those used in the Cochrane Collaboration [Update Software, Oxford, England; 1998]. We will describe each level in turn, using the choice of statin drugs as an example to illustrate their use (Table 3).

Level 1 includes randomized clinical trials providing head-to-head comparisons of the drug of interest with other drugs of the same class for their effects on clinically important outcomes. This would generate the strongest evidence for the decision maker; however, there are potential threats to validity (Table 2) and several methodologic issues unique to these trials. First, at least 1 of the drugs should have been shown to have a clinically important impact vs placebo in previous trials carried out in a population similar to that of the current trial. Second, the choice of appropriate dosage for each drug is a complicated issue, as this will affect the outcomes and safety profiles for both drugs. Finally, one must carefully consider the trial size and methods before concluding equivalence of 2 drugs—equivalence trials require much larger sample sizes than standard trials,¹⁷ and any laxity in trial conduct or patient compliance will tend to mask any real differences between drugs.

The choice of clinically important outcomes for level 1 studies depends on the target intervention. In the case of therapies designed to prevent or arrest atherosclerosis (such as statins), this implies long-term efficacy data on events such as myocardial infarction, stroke, and all-cause mortality. On the other hand, for interventions designed to treat symptomatic diseases (such as gastroesophageal reflux disease), clinically important outcomes could include symptom scores and other quality-of-life measures.

Although there are examples of level 1 evidence in other branches of medicine,^{18 - 19} they are rare in the cardiovascular literature. Our literature search failed to find any level 1 evidence for statins.

Level 2 includes randomized clinical trials providing head-to-head comparisons of the drug of interest with other drugs of the same class for their effects on validated surrogate outcomes or comparisons across 2 or more placebo-controlled trials for effects on clinically important outcomes or validated surrogate outcomes. Part A of this Users' Guide discussed criteria for deciding whether to accept results of trials based on surrogate outcomes. Ecologic studies, cohort studies, and randomized clinical trials with prestatin lipid-lowering agents were supportive of the lipid-lowering hypothesis²⁰ (that lowering low-density lipoprotein [LDL] cholesterol levels lowers the risk of atherosclerotic heart disease); however, it was not until the publication of the large-scale statin trials^{21 - 25} (Table 3) consistently linking reductions in LDL cholesterol to reductions in morbidity and mortality that we agreed to accept the surrogate end point of LDL cholesterol lowering as a proxy for clinically important outcomes. Thus, to accept head-to-head comparisons for surrogate outcomes as level 2 evidence, at least 1 of the comparators must have demonstrated efficacy in long-term trials with clinically important outcomes.

Whereas a randomized trial²⁶ comparing 4 statins for their effects on LDL cholesterol, high-density lipoprotein cholesterol, and triglycerides during an 8-week period would be an example of level 2 evidence, it also is important to incorporate considerations of the size and duration of trials in the decision-making process.

On the other hand, a number of level 2 comparisons can be made between various statins—for example, one can compare the treatment effects seen with simvastatin vs pravastatin in secondary prevention trials (such as the 4S²¹ and LIPID²⁵ studies [Table 3]). Although consistency of effects in such comparisons would be strong evidence for the presence of a class effect, these comparisons are less useful in determining whether a drug is more efficacious than another, because the advantages of randomization are lost, and the comparison is essentially that between 2 or more cohorts. In addition to the potential biases outlined in Table 2, there is also the possibility of confounding a subject's risk or responsiveness with exposure to a particular treatment in those situations in which subjects from different trials have different risk statuses. For example, if one were to compare the statin used in a primary prevention trial (such as lovastatin in AFCAPS/TexCAPS²⁴ ) with another statin tested in a secondary prevention trial (such as simvastatin in 4S²¹ ), such a comparison would only be valid if the drug efficacy is known to be independent of baseline risk, an assumption that appears valid to make in some situations (such as antiplatelet⁶ or antihypertensive⁵ therapy) but has been questioned for the statins.^{27 - 32}

It is theoretically possible to compare the efficacy of 2 drugs tested in separate placebo-controlled trials. As outlined by Bucher et al,³³ an indirect estimate of the association between drugs A and B can be obtained by comparing the OR (or relative risk) from studies of drug A vs placebo (p) and the OR from studies comparing drug B vs placebo: OR_{A vs B}=OR_{A vs p} / OR_{B vs p}. However, this assumes that none of the potential biases outlined in Table 2 are operative and that an intervention's treatment effect is consistent across different patient subgroups. Furthermore, these indirect estimates may provide substantially different effect-size estimates than direct comparisons of drug A against drug B. For example, a systematic overview of strategies to prevent Pneumocystis carinii pneumonia in human immunodeficiency virus–positive patients documented that the indirect comparison of trimethoprim-sulfamethoxazole vs a combination of dapsone and pyrimethamine suggested a much larger effect size from trimethoprim-sulfamethoxazole (OR, 0.37; 95% CI, 0.21-0.65) than was seen in the direct comparisons (overall OR, 0.64 in the 9 trials of trimethoprim-sulfamethoxazole vs dapsone and pyrimethamine; 95% CI, 0.45-0.90).³³ Thus, the strength of inference from indirect comparisons is limited.

Level 3 includes comparisons across subgroups from different placebo-controlled trials or comparisons across placebo-controlled trials in which outcomes are restricted to unvalidated surrogate markers. In addition to the biases that affect higher-level studies, comparisons based on subgroup analysis are potentially flawed (Table 2). Both simple statistics and experience have taught us that many initial subgroup conclusions (especially those that result from data-dredging) are subsequently disproven.^{34 - 35} An example of such a comparison would be looking at the efficacy of simvastatin in the 4S subgroup with the lowest lipid levels (241 patients with total cholesterol levels of 5.5-6.24 mmol/L [213-241 mg/dL])²⁸ vs the efficacy of pravastatin in the CARE subgroup with comparable lipid profiles (2087 patients with total cholesterol levels of 5.4-6.21 mmol/L [209-240 mg/dL]).²³

Level 3 evidence may also include the use of surrogate markers that, although they may lie along a recognized pathogenetic pathway from mechanisms of action to important clinical outcomes, have not been validated in long-term randomized clinical trials. To return to an example cited in part A of this Users' Guide, this would involve making inferences about reductions in fractures from the effects on bone density of 2 different bisphosphonates in 2 independent randomized trials.

Level 4 includes comparisons involving or confined to nonrandomized evidence. This type of evidence is only possible for conditions in which there are a large number of potential treatments commonly used by practitioners. Nonrandomized evidence can include cohort or case-control studies, modeling studies (using risk-prediction equations such as those derived from the Framingham data³⁶ ), and/or outcomes research using administrative databases. Although these types of analyses can provide useful insights (particularly with respect to dose-response relationships),³⁷ they are best viewed as exercises in hypothesis-generation. In particular, outcomes research studies, originally developed to determine whether the efficacy of interventions proven in randomized trials have their anticipated impacts at a population level, have sometimes been used to pursue the primary determination of efficacy—a purpose for which they were not intended. When used to establish efficacy, they present, in addition to other limitations (Table 2), unique problems in interpretation that restrict the validity of inferences drawn from them about the relative efficacy of medications from the same class.³⁸

An example of level 4 evidence is a recent reanalysis of the WOSCOPS database, designed to infer whether pravastatin's efficacy exceeds that expected of other statins.²⁹ Using the constellation of risk factors and mean on-treatment cholesterol levels seen in the trial, the observed coronary event rates in pravastatin-treated patients were compared with those predicted from the Framingham coronary risk equation to determine whether the treatment benefit with pravastatin exceeded that expected from the degree of cholesterol lowering achieved.

Level 3 and 4 studies have numerous flaws as outlined above and are best viewed as exercises in hypothesis generation.

While we have thus far focused on the validity of the evidence, the number, size, and duration of studies are essential factors to be considered in the decision-making process. Certainly, the superiority of 1 drug within a class can only be definitively established with level 1 evidence. However, while level 1 evidence would be ideal for establishing that a group of drugs exert a class effect (by showing narrow confidence limits around the difference between drugs), we recognize that it is rarely available and is unlikely to ever be available for many classes of drugs because of difficulties in funding and conducting trials so large that they are unlikely to appeal to researchers, manufacturers, or funders. In this situation, the amount of level 2 evidence becomes important. For instance, one would feel more comfortable in concluding that a drug produced a class effect if there were a number of placebo-controlled trials demonstrating that various drugs from the same class had similar treatment effects. However, our goal is not to set a level that must be achieved before a drug can be claimed to be superior to others in its class or before a class effect can be established. Those are decisions that individual clinicians or policymakers must make, taking into account their local circumstances and individual comfort levels.

In the past decade, there have been numerous examples of drugs within the same class that have been shown to have different safety profiles. Although not our primary focus, considerations of drug safety are part of any treatment or purchasing decision, so we offer a set of levels of evidence for determining drug safety in Table 4. Phase 1 drug studies in humans are designed to determine the maximally tolerated dose, and clinical trials are generally designed to determine the efficacy of the drug. As such, the sample sizes of neither are adequate to detect uncommon adverse effects. The inverse rule of 3 states that to be 95% sure of seeing at least 1 adverse drug reaction that occurs once in every given number of patients, you need to follow up 3 times that many patients.³⁹ Given the size and duration of most clinical trials, adverse effects that occur in fewer than 1 in 1000 participants or that take more than 6 months to appear will generally remain undetected.³ However, randomized clinical trials are still the strongest design for detecting real differences in adverse effects (such as the different rates of intracranial bleeding with different thrombolytic agents^{40 - 41} ), and meta-analyses of such trials can give unbiased estimates of excess hazards. In the absence of clinical trials, premarketing safety data must be considered preliminary, and large, phase 4 studies or systematic postmarketing surveillance data are necessary to confirm the safety of new drugs.

While once-a-day medications are more convenient and usually have higher compliance rates, evidence about drug compliance derived from trials may translate poorly in clinical practice. For instance, while compliance with the various statins described in Table 3 ranged from 90% to 94% during the course of the trials, analyses of administrative databases in Canada and the United States⁴² revealed that only half of statin-treated patients were still taking their medication 1 year after it was prescribed.

Faced with a decision as to whether a new drug from a class should be offered to eligible patients within the population, clinicians and policymakers have different perspectives. For clinicians, this decision usually hinges on the efficacy, safety, convenience or compliance, cost of the new drug vs the old, and the applicability of the trial evidence to their patients.⁴³ However, for policymakers, these issues form only 1 piece of the puzzle. They also must evaluate the efficiency, affordability, and opportunity costs of any new drugs. The efficiency of any intervention is determined by formal economic analyses, and the Users' Guides series offers criteria for evaluating methodological quality.⁴⁴ Although cost-minimization analysis is the simplest and least controversial of the economic analysis techniques, it requires proof that the outcomes resulting from both alternatives are the same. As this rarely exists, the policymaker must rely on other types of analyses (cost-effectiveness, cost-benefit, or cost-utility analyses) that involve varying degrees of assumption and guesswork. As pointed out by Naylor and colleagues,⁴⁵ economic analyses should be viewed as "promising, clearly helpful, still in need of refinement and open, like any new technology, to both wise use and well-intentioned abuse."

The decision as to whether a new drug is efficient enough to warrant its adoption depends critically on the social, political, and economic realities of the particular health care setting, complicating the policymaker's task. Thus, attempts to establish universal cut-points (using cost or quality-adjusted life-year ratios) have been largely unsuccessful.⁴⁶ Although there are occasions for which there is compelling evidence for a new drug's adoption (the new drug is as effective or more effective than others of its class and is less costly) or rejection (the new drug is less effective than others of its class and is more costly), the policymaker operates most often in a cost-utility gray zone between these 2 extremes.⁴⁵

Given the qualitative consistency of the RRR for acute myocardial infarction in patients treated with 3 of the statins in large trials with clinically important outcomes (Table 3) and the convincing nature of LDL cholesterol lowering as a surrogate outcome,^{20 ,30 ,47 - 49} our clinician concludes that there is a class effect of statin drugs on the occurrence of ischemic heart disease. In the apparent absence of differences in safety or compliance profile between the various statins, he decides to pursue a cost-minimization strategy. While the newer statin has been evaluated only for cholesterol-lowering efficacy in a short-term trial (<6 months), he decides to prescribe it because it is the cheapest statin in his local setting.

The policymaker agrees with the clinician that the statins appear to exert a class effect in terms of efficacy. However, she is concerned that the efficacy of the newer statin has not been evaluated in long-term trials with clinically important outcomes or validated surrogate outcomes. Thus, she decides to keep the older (and more expensive) statin on her formulary until level 1 or long-term level 2 evidence is available that proves that the newer statin is as good as or better than the currently provided statin.

While it would be preferable that every drug in each class (and indeed every dose and every formulation) be evaluated in randomized clinical trials with active comparators from the same class for its effects on clinically important outcomes, this has not been accomplished for several important classes of drugs. We believe that advocates of newer drugs within a class must provide evidence of equivalence (or superiority) to the older agents and "randomized comparative trials . . . remain the preferred evidentiary standard."⁵⁰ Recognizing that this criterion standard is not always attainable (in the case of the statins, such randomized clinical trials would require very large sample sizes and long follow-up to detect significant differences in myocardial infarction or death between 2 different statins), we suggest that discussions about class effects will benefit from citing the levels of evidence behind the arguments and recognizing the strengths and weaknesses inherent in each study design.