Estimating Treatment Effects From Observational Data: Title and subTitle BreakDissonant and Resonant Notes From the SYMPHONY Trials

Can observational data substitute for randomized controlled trials for developing treatment recommendations and initiating public health interventions? This question arises from the article by Newby and colleagues¹ on the early initiation of statins in patients with acute coronary syndromes (ACS), reported in this issue of THE JOURNAL. The authors present results from an observational cohort study that indicate no difference in mortality and cardiovascular outcomes among patients with ACS who initiated statin treatment prior to hospital discharge and patients who did not receive statins.

Almost 2 million patients with ACS are discharged from US hospitals each year.² Since these patients are at high risk for recurrent events, secondary prevention is an important medical challenge. Randomized clinical trials (RCTs) have shown the benefits of aspirin, β-blockers, and clopidogrel in these patients.² Furthermore, there is evidence that administration of these agents prior to hospital discharge reduces recurrent event rates. Large RCTs have indicated that initiation of 3-hydroxy-3-methylglutaryl coenzyme A reductase inhibitors (statins) months to years following an ACS event is effective for secondary prevention.³ Recently, attention has focused on the optimal time to commence statin therapy.

Several observational studies, among them 2 large studies comprising some 40 000 patients,^{4 - 5} found that early statin therapy was associated with lower rates of death. In addition, the results from 2 RCTs, although not definitive, yielded consistent results.^{6 - 7} Consequently, a strong momentum is building to recommend routine, early initiation of statin therapy in post-ACS patients. Indeed, such therapy is recommended in contemporary practice guidelines.^{2 - 3}

The retrospective cohort study using data from the Sibrafiban vs Aspirin to Yield Maximum Protection From Ischemic Heart Events Post-acute Coronary Syndromes (SYMPHONY) ACS trials strikes a dissonant note.¹ In contrast to the previous studies, the report by Newby et al indicates no benefit from early initiation of statin therapy for any clinical end point, except stroke, for which statins appeared protective. How should the results from the SYMPHONY populations be interpreted and how can the differences between this and the previously reported studies be explained? One possible reason for the surprising findings is that the fractions of patients in the 2 SYMPHONY trials for whom statins were started in the hospital as well as those for whom revascularization was performed were much larger than in the other 2 observational studies.^{8 - 9} Perhaps the analysis reported by Newby and colleagues signals that statins simply do not offer a significant additional benefit in the presence of revascularization therapy in relatively low-risk patients.

Other possible explanations for the different results obtained by the various studies are measurement error and confounding. Any results, whether from observational studies or RCTs can be distorted by measurement error.¹⁰ Measurement error may have affected the ascertainment of statin use, the diagnosis of ACS, and all clinical covariates. Furthermore, observational studies are especially vulnerable to confounding. Although analytic techniques can address the control of measured confounders, unmeasured confounders remain an omnipresent threat to the validity of nonrandomized research. Mismeasurement of a confounder, such as low-density lipoprotein cholesterol (LDL-C) level, might not create a problem if the physician decided treatment based on the mismeasured variable. Nevertheless, any discrepancy between the value of a confounder used by the physician and that used in the analysis could lead to residual confounding.

To overcome the lack of randomization in their cohort, Newby et al¹ used propensity scores to account for imbalances in baseline characteristics. In the 2 other large observational studies on this issue available to date, propensity scores were also used.^{4 - 5} Propensity scores have become a popular tool in pharmacoepidemiology. Of particular concern in the evaluation of treatment effects in observational cohorts is the problem of confounding by indication, the unequal distribution of pretreatment morbidity among the control and treatment groups. Rather than accounting for each of the confounders individually, the propensity score reduces them to a single composite variable. The reduction from many pretreatment variables to 1 composite score can make it easier to assess how different the treated patients and controls are with respect to background characteristics.

The propensity score is the probability of receiving treatment given the values of the predictors; it is estimated using, for example, a logistic regression model to predict treatment status from the observed predictors.^{11 - 13} Stratifying the data by the propensity score will group individuals with similar propensity score values; within these strata, the treated and untreated patients tend to have similar distributions of the pretreatment variables used to model the propensity score. Two patients who have the same value of the propensity score (a function of age, sex, smoking, disease history, and whatever other characteristics may seem relevant for determining treatment) have the same probability of receiving treatment. Having 2 such patients, 1 treated and 1 untreated, resembles the consequences of randomization. However there is 1 all-important difference: in an RCT, the propensity scores are known because treatment is randomly assigned. In contrast, in an observational study, the scores must be estimated from the data. The accuracy of these estimates will depend critically on the accuracy and completeness of the model and measurements used to create them. The propensity score is only as good as the variables it includes. Furthermore, if the model fit to estimate the probability of treatment given background characteristics is not specified correctly, the propensity score will not perform well in balancing background characteristics.

To maximize efficiency (and thus obtain tighter confidence intervals) when estimating treatment effects, baseline variables accounted for should be predictors of the outcome; including variables only predicting treatment in the regression model or the propensity score will result in a loss of efficiency.

Newby et al included 12 365 patients in their analysis and created a propensity score based on 30 variables that independently predicted the likelihood of early statin use. Stenestrand and colleagues⁴ using Swedish Registry data and Aronow et al⁵ using observational data from the Global Use of Strategies to Open Occluded Coronary Arteries in Acute Coronary Syndromes (GUSTO IIb) and Platelet Glycoprotein IIb/IIIa in Unstable Angina: Receptor Suppression Using Integrilin Therapy (PURSUIT) trials derived similar propensity scores. It would have been helpful, in evaluating the 3 analyses, if the authors had also presented evidence for the adequacy of the fit and performance of their propensity scores, eg, by presenting the baseline variables by propensity score strata separately for treatment and control group. This would be informative about the balance in the strata. It is possible that the fractiles derived for propensity score adjustment did not achieve sufficient balance of the confounders and a finer stratification would have been necessary to remove residual intrastratum confounding.

A strength of the analyses of all 3 observational data sets is the combination of a propensity score with adjustment for covariates predicting the disease outcome. This approach is superior to either individual method alone because it makes the regression model more robust to misspecifications.^{14 - 15} Such double use of covariates helps protect against errors in modeling either treatment propensity or survival.¹⁴ Comparison of the effect estimates obtained from all 3 approaches—propensity score–adjusted model, covariate-adjusted model, and the model including both—provides a useful informal goodness-of-fit test.¹⁴ It is reassuring that all 3 analytic models used by each group produced similar results.^{1 ,4 - 5} Nevertheless, unmeasured confounders still could have biased the results.

It is interesting to note that the unadjusted analyses in all 3 cohorts produced relative risk estimates indicating a significant reduction in mortality for early initiation of statin treatment, ranging from 0.43⁴ to 0.58.¹ After covariate- and propensity-score adjustment, the hazard ratios were markedly attenuated in all 3 studies, indicating the presence of confounding. In the analyses by Stenestrand et al⁴ and Aronow et al⁵ the adjusted hazard ratio remained of borderline statistical significance, whereas Newby et al arrived at adjusted hazard ratios that were close to unity. One possible explanation for these differences is more complete control of confounding by indication in the SYMPHONY cohorts.

Among the 2 RCTs that have been conducted to date, 1 randomized 126 patients and thus no separate data on mortality end points are available.⁶ In the MIRACL trial, in which 3086 patients were randomly assigned to receive atrovastatin or placebo, the relative risk estimate for mortality was 0.94 (95% confidence interval, 0.67-1.31) among early statin users compared with patients receiving placebo.⁷

When ethically justifiable and practically feasible, well-conducted RCTs are preferable to observational studies, because the effect estimates and SEs from RCTs with full compliance and follow-up account for measured and unmeasured confounding.¹⁶ But, often there is no choice but to rely, at least temporarily, on observational data. Ethical principles prohibit clinical investigators from randomly assigning humans to many of the exposures they are interested in learning about, such as the effect of smoking or asbestos. Even when there are no ethical concerns, the time necessary to complete RCTs often renders them unavailable for immediate treatment decisions. Thus, it is essential to ensure the optimal use of observational data.

The observations made within the SYMPHONY cohorts are interesting and noteworthy. The analyses by Newby et al indicate the presence of confounding by indication in the observational data and underscore the need for well-conducted large RCTs on the benefit of early statin initiation.

In a subgroup analysis of the SYMPHONY population including patients for whom core laboratory analyses of lipids were available, a significant interaction of cholesterol level with early statin therapy was observed, with a suggestion of benefit in patients with higher LDL-C levels. These observations, from SYMPHONY, resonate with those of a pooled analysis of 13 173 patients with long-term coronary artery disease from the Cholesterol and Recurrent Events (CARE) and Long-Term Intervention with Pravastatin in Ischaemic Disease (LIPID) trials.¹⁷ In those populations, benefit from statin therapy was confined to the patients whose LDL-C levels were in the upper 4 quintiles, and no salutary effects of statin therapy were observed in the 2067 participants with baseline LDL-C levels less than 125 mg/dL.

Given the totality of information currently available, including the data from the SYMPHONY cohorts, how should patients with ACS be managed with respect to the early initiation of statin therapy? The current guidelines,³ which call for the goal of reducing LDL-C levels to below 100 mg/dL in patients with established coronary artery disease are reasonable and should be followed. Since patient compliance with long-term medication is greatly enhanced by commencing therapy before hospital discharge, this approach can be recommended. Ultimately, the results of ongoing large RCTs of early statin treatment will provide further insight into the role of early statin treatment for the prevention of cardiovascular disease.^{18 - 19}