Heterogeneity Is Not Always Noise: Title and subTitle BreakLessons From Improvement

As physicians, we are ambivalent about heterogeneity in medicine. We actively suppress it—ignore it, tune it out—because doing so is crucial for establishing the efficacy of tests, drugs, and procedures. But heterogeneity is ubiquitous in complex systems, including all of biology and human society; at some level, therefore, we recognize that suppressing it exacts a heavy price and struggle to take it into account.^{1 - 4} This article explores the roots of our mixed responses to heterogeneity; it then examines the ways in which evaluating health care improvement has helped to understand the influence of heterogeneity on study methods. Finally, it considers some of the ways that are being explored to capture the rich information embedded in heterogeneity for use in both clinical and improvement science.

Establishing the efficacy of clinical interventions by studying single participants (patients or care units)—that is, using n-of-1 studies—is risky because by coincidence any single person or care unit might not respond to an effective intervention or might appear to respond to an ineffective one. Clinical research circumvents the heterogeneity problem primarily by measuring the effects of interventions in groups. Deliberately “los[ing] sight of the man taken in isolation in order to consider him as only a fraction of the species”⁵ makes it possible to detect signals that would otherwise be hidden in background noise and to calculate point estimates of effect size.

Although useful, point estimates are fiction, not reality, because most individual participants in group studies experience responses that differ from the point estimate for the entire group. Ignoring the fictional nature of point estimates leads directly to the frequent misperception known as the ecological fallacy: “a relationship derived or observed using aggregate data assembled from various populations erroneously assumed to imply that the relationship holds at the elemental or individual level.”⁶

At the same time, the frequent concern that a point estimate might misrepresent the efficacy of interventions in segments of a study population encourages the use of subgroup analyses.⁷ But subgroup analyses (“fishing expeditions”) are understandably suspect because they can easily lead researchers to perceive discordant subgroup effects that do not exist. The risks of subgroup analyses further discourage interest in exploring the “noise” created by the heterogeneity of study populations.

Since heterogeneity is ubiquitous, it is neither possible nor desirable to ignore its influence in any aspect of medicine. For example, a core element of all clinical practice is back-translation from summary results of clinical studies to decisions with individual patients. Researchers frequently estimate the effect of multiple independent input variables on outcomes by applying multivariable and meta-regression analyses within⁸ and across⁹ clinical studies.

Building on earlier work,^{10 - 11} the technique of risk stratification has recently been proposed as a statistically sound method for incorporating respondents' baseline heterogeneity into measurements of efficacy.¹² In this approach, the point estimate of the effect of an intervention is first established in 1 or more well-defined but broad study populations using traditional experimental methods.¹³ Adjusted outcome measures are then calculated for chosen participant subgroups by combining that point estimate with a risk index or score for each subgroup—an independently validated estimate of outcome probabilities in the absence of treatment, based on the presence or absence of disease-appropriate risk factors; the result is a fine-grained picture of the effects of the intervention across subpopulations. Measured in this way, the absolute clinical benefit for some interventions is as much as 70 times greater in the highest compared with the lowest preintervention risk quartile.¹²

The emerging discipline of health care improvement is designed to change patient outcomes by changing the way clinical systems deliver care. Because the care process is the sum of actions of many human agents, changes in human performance provide the crucial link between improvement interventions and better outcomes. Health care improvement is therefore a social discipline—in effect, a form of experiential learning¹⁴ in which effectiveness depends on the ability to change human behavior—as much as it is a clinical discipline.

There are compelling reasons to evaluate the efficacy of health care interventions as rigorously as all other clinical interventions,^{15 - 16} but the complex social nature of improvement complicates the evaluation process in several ways. First, because most social programs consist of multiple components, it is usually “unclear on the basis of any simple experiment what worked, where there is an apparent effect, and hence what needs to be replicated reasonably to expect similar outcomes.”¹⁷ Second, “[M]ost social programs change over time. Social programs involve intentional interaction. Differing subgroups interact with program components in different ways. Stakeholders, including research participants, adapt over time, meaning not only that the intervention but also responses to it change over time. There is ineluctable complexity as programs set off chains of action, interaction, feedback, and adaptation.”¹⁷

Third, behavior change interventions are literally created by their interaction with social contexts; that is, in an important and real sense, these interventions do not exist until users instill meaning into them, accept them as valid, and modify them to fit their local situations, a process that has been referred to as “adaptive” work.¹⁸ The malleability of behavior change interventions contrasts with the fixed properties of biological interventions, whose inherent structure and function do not change when they interact with their targets.

Fourth, within the domain of social inquiry, contexts are construed as tightly knit systems made up of the characteristics of the individuals involved, the structure of the outer and inner settings, and the process of measuring or observing the system's properties.^{4 ,19 - 20} These elements are all parts of social change programs, not simply the background in which a social intervention operates. Understood in this way, context cannot be experimentally “controlled out” of social programs because it is a major determinant of any given program's effectiveness.

As a consequence, when a social change intervention is evaluated in an individual setting, the entire local group becomes the unit of analysis, just as an individual patient becomes the unit of analysis in the trial of a biologically active intervention; arguably, therefore, single-site evaluations of social interventions are n-of-1 studies. But although they are not easy to perform, n-of-1 studies can produce meaningful evidence of efficacy. Moreover, because “the problems of internal validity are soluble within the limits of probability statistics,”²¹ single-site studies of social programs can be internally valid, provided the study methods (for example, stepped-wedge designs²² or time-series analyses²³ ) take into account the special properties of social programs.

The extreme context dependence of social programs means it is no more logical to judge them as ineffective when they work in only a limited proportion of single-site studies than it is to judge drugs as ineffective when they benefit only a limited proportion of patients.^{24 - 25} (For drugs of proven efficacy, the “number needed to treat” [NNT] to benefit 1 patient is generally at least 5 to 20, and sometimes as high as 100.) Extreme context dependence thus seriously limits the utility of systematic reviews for assessing the efficacy of social programs from single-site data^{26 - 27} ; at the same time, it highlights the underappreciated utility of systematic reviews for exploring and explaining divergent findings among single-site studies,^{10 - 11 ,27 - 28} much as genetic studies help to explain the variation in individual patients' responsiveness to drugs. The options for going beyond single-site studies to strengthen the evidence for efficacy of a social program are limited but include multisite quasi-experimental studies and cluster randomized designs, both of which are labor-intensive, expensive, and difficult to execute well.

Demonstrating the generalizability of social programs can be equally problematic, mainly because doing so requires first obtaining evidence that they work, which clearly can be difficult. Moreover, testing for generalizability requires checking the reproducibility of a program's effect in “other equally specific but different conditions.”²¹ Choosing the appropriate conditions can be difficult, however, because the inherent heterogeneity of social contexts means there is no certainty that study populations selected from spatiotemporally specific populations are “representative of all or even any given set of populations.”¹⁷

Taken together, the special features of social programs create the following methodological dilemma: the complexity and context dependence of social initiatives make it hard to tell whether those initiatives have changed performance; in turn, the difficulty of knowing whether social initiatives have changed performance makes it hard to tell which features of interventions and contexts contribute most importantly to the success or failure of a behavior change program.

Recent studies of 2 major ongoing improvement initiatives provide examples of the challenges posed by this dilemma. The first program focuses narrowly on reducing central line infections in intensive care unit (ICU) patients.²⁹ The aim of the second initiative is more diffuse: to reduce a multiplicity of adverse clinical events—primarily unnecessary deaths, cardiac arrest, unplanned ICU admissions, and postsurgical complications—by establishing early response team systems that expedite bedside intervention throughout the hospital in clinically deteriorating patients.³⁰

As is true of most improvement programs, both initiatives involve complex, multicomponent interventions. Their technical components—that is, bedside tools and maneuvers—were not innovations (Table 1). In the central line infection control studies, for example, the efficacy of the components of the central line infection control bundle had been well established in prior independent studies,²⁹ and confidence in the efficacy of these components therefore was far above equipoise. Inability of the program to affect staff's use of the bundle would therefore have been a more likely explanation than inherent lack of the bundle's efficacy if infection rates did not change. In studies of early response team systems, in contrast, confidence was not high in the ability of standard bedside tools and maneuvers to reduce adverse outcomes because few studies had independently assessed the efficacy of those approaches in that regard, except perhaps for some measures used to treat shock. Confidence in the ability of these tools and maneuvers to correct important physiological disturbances, however, did seem to have been well above equipoise.³⁰

Measurement of clinical outcomes in both of these programs could potentially serve a dual purpose. From the broad perspective of clinical practice, particularly the patient's point of view, the most important question was whether fewer adverse outcomes occurred. When focusing on causal pathways, however, the question closest to equipoise appears to be whether the interventions led staff to use well-established clinical tools in more consistent, timely, and appropriate ways. Reframed in this fashion, the changes in clinical outcomes can be seen as surrogate (indirect) measures of changes in staff performance as well as of overall clinical effectiveness.

The Central Line Infection Control Project. The median infection rate in the 103 ICUs that participated in this study²⁹ decreased from 2.7 infections per 1000 catheter days to 0 during the initial 3-month postimplementation period. Moreover, this statistically significant result was sustained for 16 to 18 months, providing unequivocal evidence of efficacy, despite the study's uncontrolled, quasi-experimental design. However, the mean infection rate for the entire group of ICUs, which at baseline was 7.7 infections per 1000 catheter days, was still 2.3 infections per 1000 catheter days in the period 0 to 3 months after implementation and was 1.4 infections during the entire 18 months of follow-up.

Taken separately, each of these numerical expressions of outcome indicates that the intervention was effective in controlling infection across the entire group of ICUs. When taken together, however, it is clear that in half the ICUs the intervention failed to control infections about 60% of the time during the first 3 months and 36% of the time over the entire 18 months of the study. (All postimplementation infections occurred in about half of the total patient days, and the mean infection rates in the nonzero infection subgroup were about 4.6 and 2.8 per 1000 patient days in the 2 study periods, respectively.) Although the complexity of ICU care makes it impossible to identify with certainty which factor or factors interfered with infection control in the units that experience nonzero infection rates, the most plausible hypothesis appears to be that staff in those units used the bundle incompletely or incorrectly at least part of the time.

Since “extraneous variation in settings and respondents is a threat to statistical conclusion validity,”^{13 (p85)} the finding that the program was completely effective in some ICUs but not others can be interpreted as reducing, however slightly, the strength of the evidence that the bundle works. Within an efficacy framework, the variation in observed infection rates can therefore be considered noise.

But as a surrogate measure of performance change, the finding that infection rates were higher in some units than others provided an opportunity to discover exactly which staff were most and least affected by the program, the circumstances under which the program did or did not gain traction, and the mechanisms for the differences. Within a behavior change framework, the variation in observed infection rates can therefore be considered signal.

Although the researchers paid meticulous attention to the sociocultural aspects of the program during its implementation,^{18 ,31} the study was not designed to evaluate the mechanisms by which the program worked. The researchers subsequently provided some “educated hunches” about which components of the intervention might have contributed most importantly to its success; namely, changing the safety culture, using strong evidence, and carrying out rigorous measurement.^{18 ,32} However, aside from statistical assessment of the relation between infection rate and unit bed size, hospital teaching status, and geographic location²⁹ the study did not formally explore the contribution of local variations in context factors (for example, staff experience and personalities,³³ unit staffing structure, patient-staff ratios, power relationships, physical resources, staff understanding and acceptance of the intervention, and the execution of measurement procedures) to variations in behavior or clinical outcomes.

Early Response Team Systems. Observers in the 1990s noted that in most hospitalized patients who experienced cardiopulmonary arrest, physiological deterioration was clearly detectable but not acted on in the immediate prearrest period. These findings suggested that early detection and intervention might reduce not only arrest but also death, ICU transfer, and other serious adverse outcomes. In response, many hospitals introduced rapid response team (RRT) or medical emergency team (MET) systems that respond to patients showing signs of early clinical deterioration. Others have implemented critical care outreach team systems (CCOTs) that carry out both prospective surveillance and management of early deterioration in selected populations of high-risk patients; for example, following surgery or discharge from intensive care units. The aggressive development of these systems has been criticized as premature, primarily because published evidence on their efficacy and efficiency has been seen as unconvincing or too limited.^{34 - 35}

Reports of more than 25 studies of the clinical efficacy of RRTs have been published.^{35 - 46} Most studies used relatively weak observational methods, primarily before-after designs; 2 undertook more rigorous approaches.^{47 - 48} The most striking finding across the entire group of reports is the wide variation in clinical outcomes: absolute risk reductions in mortality were statistically significant in 9 of the 20 studies that assessed this outcome and ranged from 0.3% to 18%; statistically significant absolute risk reductions in cardiopulmonary arrest, which ranged between 0.1% and 27%, were reported in 5 of 15 studies.

The 2 most rigorous studies^{47 - 48} yielded conflicting results. Introduction of a critical care outreach team system in 1 hospital using a stepped-wedge controlled design was followed by a statistically significant reduction in mortality of almost 50% (odds ratio for mortality, 0.52; 95% confidence interval, 0.32-0.85 in the intervention vs control group) in the units covered by the team.⁴⁷ In contrast, the multihospital MERIT study that used a randomized cluster design was unable to demonstrate statistically significant decreases in cardiac arrest, unplanned ICU admission, unexpected death, or the combination thereof in the group of hospitals with MET systems.⁴⁸ Because of its rigorous methods, this study is widely cited as providing strong evidence against the efficacy of MET systems. Despite its strong design, however, the MERIT study proved to be inconclusive, partly because it lacked statistical power, but also because of the likelihood of contamination between the 2 study groups; wide interhospital variation in baseline event rates; a demonstrable association of baseline event rates with changes in clinical outcomes, which was consistent with ceiling effects⁴⁹ ; low MET call rates; and confidence intervals for the primary result that were probably wide enough to include the differences in outcomes the authors had prespecified as clinically important.

From a broader perspective, the heterogeneity of sites and settings across the 25 studies was substantial.³⁰ Participating hospitals were located in multiple countries, in both urban and rural settings, and served a wide variety of patient populations. The mix of pathophysiological problems that teams encountered varied considerably, as did the mix of bedside interventions used in response. The organization, training, and experience of hospital staff varied widely, as did the range of hospital services, their teaching status, type and extent of physical facilities (including number and structure of ICUs), financial resources, administrative structure, and organizational culture.

Although the published data on early response team systems are for the most part based on quasi-experimental data, findings from these studies do not exclude the possibility that early response team systems meaningfully reduce adverse events, at least in some sites and settings.³⁷ Equally important, a number of observations made in these studies generate testable hypotheses about aspects of the intervention and local contexts that could affect the clinical effects of these systems⁵⁰ (Table 2). Those hypotheses include the influence of team structure and function (CCOTs may be more effective than RRTs/METs), the target population (benefit may be greater in narrow, higher-risk populations), baseline event rates (ceiling effects may limit outcome effects), team activity (efficacy may increase with call rates), and staff response to the team system (greater understanding and acceptance of the system may correlate with greater effectiveness).

The suppression of participant heterogeneity in rigorous clinical trials helps to explain why the published clinical literature is overwhelmingly explanatory rather than pragmatic; that is, focused on what works rather than on informing real-world decisions among alternative clinical interventions.⁵⁶ The dearth of reliable, nuanced information on outcome variation across risk subgroups appears, in turn, to contribute importantly to the well-documented difficulty of translating hard clinical evidence into practice.⁵⁷ At the other extreme, the social science literature is a rich source of detail on the features of interventions and contexts that shape behavior change programs to meet specific local needs but has had limited success in obtaining evidence for causality in “particular context-mechanism-outcome alignments.”⁵⁸

The combined clinical-plus-social nature of health care improvement has prompted efforts to develop a hybrid science for evaluating improvement programs that draws on the methodological strengths of both clinical and social disciplines while minimizing their limitations. One such effort proposes a series of steps that resemble the phased process of drug development.⁵⁹ The initial steps focus primarily on optimizing the intervention, tailoring it to the nature of the clinical problem, and adapting it to local staff and patient needs before formally evaluating its ability to improve clinical outcomes in a controlled trial. Other proposals for methods that could provide a deeper understanding of complex medical interventions include comparative effectiveness trials,⁶⁰ mixed methods and Bayesian analysis,⁶¹ theory-driven approaches,^{62 - 63} ethnographic techniques,^{64 - 65} and pragmatic trials.^{66 - 67}

The techniques being developed to measure how the effects of clinical interventions vary by level of baseline risk^{10 - 12} suggest an alternative hybrid approach to the evaluation of improvement programs. This technique, which might be called “system-change likelihood stratification,” would first establish the effect of an improvement program on clinical outcomes across a diverse group of trial sites, either in 1 multisite study or multiple separate single-site studies, all of which would ideally use the same or similar rigorous study designs. The resulting aggregate measure of effect would then be combined with a system-change likelihood index or score, analogous to clinical risk stratification indexes or scores, for each study site. These indexes would be independently established measures that reflected organizations' predicted responsiveness to social change interventions, based on relevant context features. They would probably need to include generic elements (for example, the nature and quality of leadership, personnel, structure, and measurement) analogous to age and sex in clinical risk scores, as well as other elements specific to the type of intervention (for example, previous experience with similar interventions, or competing priorities) analogous to disease-specific factors in clinical risk scores. The resulting combined measure could serve as prospective estimate of the likelihood of benefit from a particular improvement intervention at a particular site, in relation to that site's system-change likelihood index.

Traditional experimental methods are powerful instruments for knowing whether medical interventions work. However, those methods are not well designed to assess the influence of variation among study participants on outcomes, particularly heterogeneity of the social systems that drive health care delivery. Experimental and quasi-experimental studies of what works are most appropriately seen as starting points in evaluating the effects of many clinical interventions, including improvement programs, rather than as the only meaningful measure of their effectiveness. However, study methods that might help judge the effectiveness of clinical interventions in particular patients and of improvement interventions in particular sites and settings have remained elusive. We have needed such methods for a long time; the need is clearly greater now.