The Clinical Research Enterprise: Title and subTitle BreakTime to Change Course?

To keep up with the latest information, a physician needs to read 75 primary studies and 11 meta-analyses daily.¹ Yet despite this avalanche of published research, solid evidence for everyday interventions remains scarce,² and even available evidence has been called into question because of various ubiquitous threats to validity.³ As a result, despite the growth of available technologies and information, knowledge and understanding of the appropriate use of these tools is far from optimal. One possible manifestation of this lack of understanding is the death toll associated with medical interventions, particularly in hospitals. Iatrogenic events are the third leading cause of death in the United States,⁴ and unfortunately, despite a decade of focused interventions aimed at quality improvement in hospitals, these numbers have not changed.⁵ This is partly due to high barriers in the physician community to adoption of evidence-based practice guidelines,⁶ but other factors are likely at play. One such factor may be the paucity of individualizable evidence.

In the United States, multiple large stakeholders use a substantial proportion of clinical research output for making decisions and setting policy. Three of the largest are the Food and Drug Administration (FDA), Agency for Healthcare Research and Quality (AHRQ), and Centers for Medicare & Medicaid Services (CMS). Each has a unique mandate, informing the respective study designs. Because the FDA is charged with evaluation of new technologies' safety and efficacy prior to market release, there must be a rigorous understanding of whether the new technology works. The sine qua non of answering this question is internal validity, or whether the technology is truly doing what it is intended to do. For this reason, the large, double-blind, randomized controlled trial (RCT) is the gold standard for gathering this type of information. However, a well-appreciated limitation of such rigorous investigation is the lack of generalizability. Furthermore, even in a large and potentially more representative trial of the relevant populations, the stringent statistical considerations by the FDA preclude investigation of appropriate subgroups of patients, in whom responses to the intervention may vary substantially from the mean or median calculated for the entire trial group. Such measures of central tendency, while reducing the noise in the data, at the same time may conceal important signals on either side of the center, both positive and negative.

Two other agencies in the Department of Health and Human Services are creators and consumers of clinical research. The AHRQ and CMS focus on effectiveness and efficiency of interventions in the real-world setting. Here, the data generated in an RCT are frequently irrelevant precisely because of their limited generalizability. Thus, research geared at quality improvement, translation, and coverage decisions requires a different set of data and methodologies. For the most part, these consist of before-and-after clustered studies, as well as other observational designs. While well describing what goes on in the real world, and thus providing better generalizability than an RCT, these data are limited by their ability to provide valid inferences of causality. Furthermore, they are rarely analyzed in a way to provide a granular enough picture for use at the bedside.⁷

While there is a trade-off between internal and external validity in both interventional and observational study designs, neither type appears to provide the easily individualizable data every physician requires at the bedside. This represents a potentially critical, yet not well-articulated objection to evidence-based guideline implementation. This objection can be defined as the phenomenon of heterogeneous treatment effect.⁸ This is illustrated well by the fact that the majority of medicines on the market, having passed the efficacy test in a large and varied group of study participants, in reality works in a minority of the patients who qualify for them.⁷ It appears that in pursuit of valid and generalizable population knowledge, the most important constituency for the application of such knowledge, the physician-patient dyad, has been underrepresented and with dire consequences.⁴

Therefore, what is the relevance at the bedside of the information contained in the daily dose of 75 primary studies and 11 meta-analyses,¹ and is there a need to overhaul how most clinical research is designed, performed, and reported? There is a need for deep introspection and adjustment to the course. Although the approach of the last 40 years has brought much knowledge to the field of medicine, it has been unable to overcome its continued underperformance in terms of safety, effectiveness, and efficiency.

The way forward is far from clear. As computing capabilities continue to increase, they will facilitate a transition from the current approach to clinical characteristics that drives analyses today and a move to the more nuanced and realistic representation of the complexity of individual human physiology. As it becomes more feasible in the computing environment of the 21st century to manage and analyze massive naturalistic data sets without rapidly losing statistical power, knowledge gaps will be closed on how various combinations of multiple comorbidities may affect health interventions and outcomes of interest, making the evidence more relevant at the bedside.⁹ Another component in this progress will be better understanding of genetics. Because of the cognitive load this knowledge will bring, more robust decision-support systems will be critical as an adjunct to a physician's hands-on evaluation.

Although much of this is still far in the future, some of the more relevant methods already in existence need to be used more consistently, and new methods need to be developed and vigorously debated by all of the constituencies consuming the data. Perhaps instead of performing a 10 000-patient RCT as usual, more thought needs to be given to relevant effect modifiers a priori to base enrollment stratification on these factors. This will allow exploration of important subgroup effects without compromising statistical validity. This approach will retain the usefulness of the data to regulators yet extend relevance to bedside encounters. Similarly, solid methods already exist for performing n-of-1 trials, considered to generate high-quality evidence for treating individual patients in appropriate circumstances.¹⁰ Clinicians and payers need to agree on the importance of such studies in order to build appropriate and sustainable infrastructures to make them feasible. Given the low-level use of heterogeneity testing in the current literature,⁷ much room exists to augment its use by adding a few simple steps to analyses already performed, again extending their usefulness and validity to the critical end-user constituency.⁷ As for implementation of evidence, the explosion of information over the last decade in behavioral sciences has provided valuable insights on how to influence human behavior, underscoring the need for collaborations not only within the medical profession, but also across disciplines that can accelerate the diffusion of evidence into practice.

It is likely that the current research enterprise, having provided much insight into the science of human diseases, diagnostics, and therapies, may have exhausted its usefulness. The harm in the health care system today requires rapid, drastic, and creative ways of stemming the problem. In the context of resource constraints, the usability of generated data must be streamlined and optimized. This agenda must be moved forward if the primary obligation of medicine is to be fulfilled: primum non nocere.