Registries for Robust Evidence

Although randomized clinical trials (RCTs) are the bedrock for establishing which interventions are efficacious, there is increasing recognition that they cannot address all needs, especially the need to determine, in a timely manner, the safety and effectiveness of different interventions used in the diverse array of patients and settings that make up a health care system. Interest has increased in the role of observational studies, and more specifically in registries and other electronic data sets, as a way to fill these critical gaps in evidence and as useful guides for helping to determine formulary placement. The recently released “highest priority challenge topics” from the National Institutes of Health (NIH) pointedly reference registries more than 20 times.¹ Yet little guidance is available to help patients, physicians, payers, researchers, and policy makers evaluate the quality of information derived from these nonexperimental sources.

The Agency for Healthcare Research and Quality defines a patient registry for evaluating outcomes as “an organized system that uses observational study methods to collect uniform data (clinical and other) to evaluate specified outcomes for a population defined by a particular disease, condition, or exposure, and that serves a predetermined scientific, clinical, or policy purpose(s).”² Registries and other noninterventional studies are often referred to as real-world data to distinguish them from most clinical trials. These real-world studies are a heterogeneous mix ranging from prospective observational studies like registries to studies of large administrative and electronic medical record data sets collected for other purposes. As more attention is focused on using real-world data to build an evidence base, it is important to know how to evaluate the quality and usability of studies derived from these types of observations.

Registries could be organized around conditions or exposures, such as a particular disease; a health care service (eg, procedure); or a product (drug or device) and can address questions ranging from treatment effectiveness and safety to the quality of care delivered. Registries vary in complexity from simply recording product use as a requirement for reimbursement to more systematic efforts to collect prospective data on many types of treatment, risk factors, and clinical events in a defined population. Follow-up could be retrospective, prospective, or a combination of both. The mode and duration of follow-up could range from days (eg, hospital admission registry) to decades (eg, orthopedic implant registry).

How the denominator population is defined and enumerated depends on the research question. For example, to estimate the proportion of a population that has been vaccinated, the underlying population would need to be systematically sampled along with those who receive vaccines. In contrast, the safety and effectiveness of a new treatment could be studied by following typical patients who receive the new treatment to evaluate how their condition resolves and whether any untoward events occur that appear to be related to the treatment.

Registries are being used to fill important gaps in evidence and contribute to understanding how trial results can be applied in practice. For example, observational data were used to compare coronary artery bypass graft (CABG) surgery with percutaneous coronary intervention (PCI).³ The available trial evidence was derived mainly from patients with single- or 2-vessel coronary disease—and did not reflect other therapies in use at that time, such as minimally invasive surgery, off-pump surgery, and drug-eluting stents. Physicians and health insurers sought evidence to help guide therapeutic decisions, but there were no trial data for the clinically diverse range of patients commonly treated. Although it appeared that myocardial infarction and mortality were comparable for PCI and CABG among patients with similar levels of coronary disease, registry data showed a strong gradient of benefit of CABG by severity of disease.³

Data from registries are also used to support timely decisions by regulatory agencies about safety and about coverage (payment). Safety data from the acyclovir pregnancy registry were used to change the Food and Drug Administration (FDA) pregnancy labeling category from category C (risk cannot be ruled out) to category B (no evidence of risk in humans).⁴ The FDA has also used observational data to expand labeled indications, such as broadening the age groups for intra-ocular lenses through review of registry data,⁵ and as supporting information for a new indication (suicide prevention) for an antipsychotic agent.⁶ The Centers for Medicare & Medicaid Services recently expanded its coverage for positron emission tomography scans in diagnosing certain cancers because of information obtained through a registry.⁷ The UK National Institute for Health and Clinical Excellence also uses registries for decision support; the British Society for Rheumatology Biologics Register has provided information for evaluation of anti–tumor necrosis factor α drugs,⁸ and registry data are being evaluated for use to assess novel interventional procedures.⁹

Although registries can provide realistic estimates of how well interventions may work in practice, these studies require complex analyses and careful interpretation. The choice of treatments observed in everyday practice is influenced by characteristics of the patient (eg, disease severity, past treatments, comorbidities), the clinician (specialty, practice setting, training), and the terms of health insurance coverage or national guidelines. All these may influence the outcomes of interest. For example, selection biases, also referred to as selective prescribing and confounding by indication, can lead to sicker patients being treated with newer treatments, and when these patients have worse outcomes due to their underlying disease, it can be misinterpreted as a treatment effect.

Like all good clinical research, it is important to understand how patients come to be treated, whether the treatments and outcomes of interest are likely to be recorded accurately and reliably for all patients, and if there is systematic loss to follow-up. Understanding the potential sources of biases is important, and the value of a registry can be further enhanced by providing a quantitative assessment of the extent to which bias may have affected the results (eg, through sensitivity analyses). Although careful planning, execution, and analysis can sometimes offer a number of solutions to control confounding, the role of registries in decision making remains controversial. The contention is largely due to generalized concerns about data and evidence quality and rarely takes into account recent advances in methodological and process applications.¹⁰

Policy makers and regulatory bodies, like the European Medicines Evaluation Agency and the US FDA, are stipulating postapproval long-term safety evaluations as a condition of approval for certain drugs and biologics. Commercial, national, and other insurers are calling for observational research, including long-term follow-up data, to support formulary decisions and pricing. Some argue that these data may be more valid for the inferences needed in clinical decision making because few exclusion criteria are used and inferences are drawn from measurements customarily used by clinicians. The demand for timely real-world data to support decision making will continue to drive the development of increasing numbers of registries. At the same time, incentives for adopting electronic health record systems under the American Recovery and Reinvestment Act will make more clinical data available in digital format. If these systems are truly interoperable with research systems, these data also will be much more accessible for registries and other research purposes. However, the speed at which these data become available is not always matched by the rigorous attention to quality necessary for research purposes.

To enhance the evidence base with timely and representative real-world studies such as registries, 2 efforts are needed: methodological research to increase understanding of what constitutes quality in these studies and in the data sources and a more directed effort to meaningfully evaluate the strengths and limitations of different types of evidence for particular questions. As the diversity of the evidence base increases, the focus should turn to what constitutes high-quality research and evidence for a particular purpose and how quickly and reliably the information can be obtained, and less on the label of the particular study design.