Translating New Medical Therapies Into Societal Benefit: Title and subTitle BreakThe Role of Population-Based Outcome Studies

Phase 4 studies have traditionally monitored safety of new drugs and occasionally served as marketing tools for pharmaceutical companies. In the 1990s these “seeding studies” came under criticism for the substantial influence industry had on study design and interpretation.^{1 - 2} Recently, innovative study designs and access to large electronic databases have allowed investigators to evaluate outcomes in the general population and provide information regarding uptake, safety, and outcomes in the real world.³ These population-based outcome studies may have important potential roles in clinical research. In this Commentary, we describe several examples that demonstrate the emerging roles of these studies and propose that they could be considered as adjuncts to and occasionally as substitutes for randomized controlled trials.

In 1999, Pitt et al⁴ evaluated the addition of spironolactone to standard medical therapy for congestive heart failure. The authors reported that spironolactone was associated with an 11% absolute (30% relative) reduction in mortality and significant reductions in heart failure hospitalizations and symptoms.

Five years later, Juurlink et al⁵ reported on a population-based outcome study that examined trends in treatment and outcome at the population level. Using comprehensive administrative data on prescriptions and hospitalizations, the investigators compared the management and outcome of heart failure before and after publication of the trial by Pitt et al. All Ontario residents older than 65 years treated with angiotensin-converting enzyme inhibitors who had recently been hospitalized for heart failure (1994-2001) were included (n = 20 820 in 1994; n = 32 283 in 2001). The spironolactone prescription rate increased 5-fold after publication of the trial results, including a substantial proportion of patients who would not have been eligible for the randomized controlled trial. Uptake of spironolactone at the population level was not associated with any significant decrease in the rates of readmission for heart failure or death from all causes. Moreover, the authors found a substantial increase in the rate of hospitalization for hyperkalemia (from 2.4/1000 patients in 1994 to 11.0/1000 patients in 2001, P < .001) and associated mortality (0.3/1000 patients in 1994 to 2.0/1000 patients in 2001, P < .001).

In 1999, the National Cancer Institute released a clinical announcement recommending that “strong consideration should be given to adding chemotherapy to radiation therapy in the treatment of invasive cervical cancer.”⁶

To evaluate the outcome of chemoradiotherapy in the general patient population, Pearcey et al⁷ undertook a population-based outcome study of the management and outcome of cervical cancer in Ontario. Using linked administrative and cancer registry databases, the authors identified all cases of cervical cancer diagnosed in Ontario between 1992 and 2001 (n = 4069). Prior to the National Cancer Institute announcement, fewer than 10% of patients receiving curative radiation therapy received concurrent chemotherapy. Early in 1999, the use of chemoradiotherapy increased rapidly, and from 1999 through 2001 more than 60% of patients receiving curative radiation therapy received chemoradiotherapy. The authors found that the 3-year overall survival for the entire cervical cancer population of Ontario remained stable in 1992-1998 but had increased significantly among the radiation therapy cohort from 56% in 1995-1998 to 69.8% in 1999-2001 (P < .01). This survival benefit was virtually identical to what would have been expected based on the results of the clinical trials. Furthermore, the rate of hospitalizations remained stable, suggesting that there was no increase in acute toxicity.

Because the risk of recurrent stroke is up to 10% in the week after a transient ischemic attack or minor stroke, Rothwell et al⁸ sought to determine the association between more rapid treatment and outcomes in this patient population. The investigators concluded that a randomized controlled trial of urgent vs standard therapy for transient ischemic attack was unethical and therefore designed a prospective before (phase 1, April 1, 2002, to September 30, 2004) vs after (phase 2, October 1, 2004, to March 31, 2007) study of urgent assessment and immediate treatment of all patients with transient ischemic attack or minor stroke who were not admitted directly to a hospital (n = 620). The study was nested within a rigorous population-based incidence study of all patients with transient ischemic attack and stroke, in which case ascertainment, investigation, and follow-up were complete and identical in both periods.⁹

Rothwell et al found that median delay to clinical assessment decreased from 3 days in phase 1 to less than a day in phase 2 (P < .001) and that median delay to first prescription of treatment decreased from 20 days to 1 day (P < .001). The 90-day risk of recurrent stroke in patients referred to the study clinic was 10.3% in phase 1 and 2.1% in phase 2 (P < .001). The authors concluded that early initiation of existing treatments after transient ischemic attack or minor stroke was associated with an 80% relative reduction in the risk of early recurrent stroke.

These contemporary examples demonstrate several roles of population-based outcome studies (Box). The study of spironolactone for heart failure demonstrated rapid uptake of novel therapy and failed to confirm that the novel therapy translated into benefit at the population level. This study also demonstrated important treatment-related toxicity not manifested during the randomized controlled trial. Conversely, the study of chemoradiotherapy for cervical cancer demonstrated that the survival benefit and toxicity in the general population were consistent with outcomes that might be expected based on the randomized controlled trial results. The study of early assessment and treatment for transient ischemic attack or minor stroke provides a powerful example of how a population-based outcome study can answer critical questions that cannot (or most likely will not) be addressed by a randomized controlled trial. These studies represent a fundamental component of the proposed 3 T's of translational research to improve the quality of health care by evaluating who benefits from emerging medical therapies.¹⁰

Although a detailed methodological review of population-based outcome studies is beyond the scope of this Commentary, it is worth highlighting a few fundamental principles. Each of the 3 studies discussed herein included the entire population of interest and for that reason can minimize the referral bias that plagues traditional institution-based observational studies. The large sample size of each study provided ample statistical power to identify even small absolute differences in outcome and toxicity. Furthermore, a well-defined temporal (or geographic) difference in practice allowed investigators to use a before-and-after approach to minimize error by confounding.

This quasi-experimental design relies on the assumption that the underlying population and other treatments did not change across time period (or region). Within most clinical contexts, when there is an abrupt change in treatment practice, it is likely that these assumptions hold true and any difference in outcome usually can be attributed to the uptake of the novel therapy in question. The technique of grouping patients with a variable that is a marker for different practice policies (usually temporal period or geographic region), but is not directly associated with treatment selection or outcome, originated in econometrics and is now recognized within outcomes research as the “instrumental variables approach.”^{11 - 12} In addition, population-based outcome studies require comprehensive administrative databases to provide information on the complete population; this is facilitated in jurisdictions with universal health insurance and in conditions for which a registry exists.

Population-based outcome studies also may contribute to a better understanding of research methodology in providing an empirical process to determine which aspects of randomized controlled trial design (ie, “the large simple trial”) are associated with a minimal efficacy-effectiveness gap.¹³ Furthermore, there is increasing interest among research funding organizations to identify the return on their investment. Population-based outcome studies may have a novel role in shifting from output-based measures of performance (ie, publications, citations) to outcome-based measures (ie, improvement in patient well-being).¹⁴

In addition to evaluating the effectiveness of new therapies in the general population, population-based outcome studies provide a mechanism for studying whether physicians are delivering appropriate care to patients and whether rare (but serious) toxicities emerge outside the clinical trial. Population-level data also can provide insight into the relative benefits of therapies when they are administered to individuals who would not have met eligibility criteria for clinical trials. The ability to evaluate care in this “penumbra of knowledge” (ie, the area just beyond the scope of the relevant randomized controlled trial) is unique to this study design. Traditional epidemiological teaching acknowledges the superior internal validity of randomized controlled trials and superior external validity of phase 4 studies. However, methods of contemporary population-based outcome studies have evolved considerably from the original industry-sponsored “seeding studies” and now can provide robust external and internal validity.

Well-designed population-based outcome studies should be considered a natural step in the evolution of evidence and should be conducted in follow-up of all major randomized controlled trials. Ongoing areas of investigation include refining appropriate statistical methods¹⁵ and exploring the feasibility of population-based economic analyses. It also remains to be established who should conduct and fund such studies. Although the randomized controlled trial will remain the gold standard for establishing the efficacy of new medical therapies, population-based outcome studies have the ability to provide unique insight into the societal benefits of treatments in the real world.