Improving the Science and Politics of Quality Improvement

Quality and safety are critical issues for health care systems around the world and have appropriately been highlighted in ongoing efforts to reform and improve health care. Care of critically ill patients is a domain in which issues of quality and safety take on monumental importance because of the severity of illness and the complexity of providing high-quality critical care.

In this issue of JAMA, Scales and colleagues¹ report the results of an important study designed to evaluate the effectiveness of quality improvement efforts in community-based critical care units in Ontario. The authors use a multifaceted “knowledge transfer” intervention including education, dissemination of algorithms, and audit and feedback, transferred through an interactive telecommunication strategy. The goal of the intervention was to increase adherence to 6 quality measures that have been documented to improve patient outcomes: prevention of ventilator-associated pneumonia (VAP), prophylaxis for deep venous thrombosis (DVT), daily spontaneous breathing trials, prevention of catheter-related bloodstream infections, early enteral feeding, and prevention of decubitus ulcers. Although debate remains about some of the evidence supporting these measures, there is general consensus that appropriate implementation of each measure can decrease risk of harm for critically ill patients.

Numerous studies have investigated the value and implementation of these quality measures, but the study by Scales and colleagues has several notable features. First, the design and implementation reflect state-of-the art methods for a study evaluating quality improvement and advancing the state of science. Second, the study design, knowledge transfer intervention, and analyses are extremely complex but appropriate to the complex phenomenon under study. Third, the results were positive, documenting significant improvement in quality, although benefits were modest. Fourth, this study—with its state-of-the-art, complex methods—was funded by the Ontario health care delivery system rather than a research funding agency, which has important policy implications for improving health care in North America and beyond. Each of these 4 points warrants further discussion.

Many studies have been designed to improve health care quality, but relatively few are randomized trials. Although much can be learned from nonrandomized and nonexperimental studies, the authors' use of a randomized design with an appropriate control group improves their ability to control for bias and greatly enhances the ability to draw causal inferences. An observational study with historical controls would be unable to exclude temporal change as an explanation for the findings. These authors also addressed the problems that result from having the control group receive no intervention by using the same knowledge transfer intervention targeting different quality measures in pairs. Each intensive care unit was randomly assigned to simultaneously be the experimental group receiving the knowledge transfer intervention for one quality measure (eg, VAP prevention) and the control group receiving no knowledge transfer intervention for the other measure of the pair (eg, DVT prophylaxis). Additional innovative methods included a “decay-monitoring period” demonstrating that the improvements persist and qualitative interviews with participants to explore mechanisms of action and facilitators of success.

Another feature of this study is its complexity. Rather than using the standard scientific method of changing one variable and examining its effect, the authors designed a multifaceted knowledge transfer intervention. Yet this complexity is not only appropriate; it is essential. Years of research suggest that multifaceted interventions are needed to change clinician behavior and transfer knowledge of best practices into the complicated health care system.^{2 - 4} The cluster randomized design is appropriate for an intervention targeting the hospital, yet requires complex statistical approaches to account for clustering by center, can significantly reduce effective sample sizes (if intracluster correlations are high), and substantially reduces effectiveness of randomization, thereby increasing risk of randomization imbalances. Cluster randomized trials are complex to design, conduct, analyze, and interpret. The primary outcome in this study—a summary ratio of odds ratios—is also complicated yet appropriate for the primary research question: does this multifaceted knowledge transfer intervention improve quality across a series of measures?

The answer to the primary research question is yes, the intervention improved quality—yet the improvement was relatively modest. There was significant improvement in adherence to quality measures in the intervention sites compared with the control sites across all quality measures, but improvement in individual measures was less clear. Improvements were seen in only 2 of 6 measures, semirecumbent positioning to prevent VAP and precautions to prevent catheter-related bloodstream infections. Only the latter measure improved significantly compared with the control group. The findings were limited by relatively high adherence to some quality measures at baseline—a good problem to have, but one that makes it difficult to demonstrate an intervention effect. Furthermore, this study only examined changes in process measures assessing delivery of health care and did not demonstrate or examine differences in patient outcomes. Nonetheless, the focus on improvement in overall quality and on process measures was reasonable. Despite having more than 9000 intensive care unit admissions, this trial did not have the necessary sample size to find differences in patient outcomes. Such a trial would have been prohibitively large and expensive; requiring patient-level outcomes in every such study would slow progress.

Perhaps most interesting about this study is that this high-quality cluster randomized trial was funded not by a research funding agency but by an organization that funds delivery of health care. To make significant steps toward improving the quality of health care and controlling the rate of increase in health care costs, this is an important model for the future. In the United States, the Affordable Health Care for America Act calls for demonstration projects that document effective methods to improve quality and control costs. Using these demonstration projects to make significant advances will require the same type of rigorous, high-quality research used by Scales and colleagues. The use of health care reimbursement to encourage and enforce quality is a reality of the US health care system today and in the future, but these quality measures must be selected and implemented based on rigorous science, and the implementation must be demonstrated to be effective without unintended consequences that lower quality in other ways or other areas of health care.⁵

Despite efforts to develop rigorous quality measures and demonstration projects, there are a number of examples of significant problems and unintended consequences.^{5 - 8} High-quality science in quality improvement is needed to minimize such problems. One way to promote high-quality science in this area is for research funding agencies to continue to fund projects that advance the science of quality improvement and knowledge transfer. Another is to ensure that agencies responsible for developing and implementing quality measures and demonstration projects are working in collaboration with the scientists and clinicians in the relevant fields. One example of this collaboration is the Critical Care Societies Collaborative (CCSC), which includes 4 of the largest scientific and professional critical care societies: the American Association of Critical-Care Nurses, the American College of Chest Physicians, the American Thoracic Society, and the Society of Critical Care Medicine.⁹ Working in collaboration, the CCSC joined forces with the Department of Health and Human Services (DHHS) to design and implement quality improvement initiatives. The DHHS cannot work with each individual professional society, but large collaboratives like the CCSC offer an opportunity to bring together scientists, clinicians, and policy makers to develop and implement quality measures and reimbursement programs that improve quality and control costs.

Debate about the future of health care continues to rage and yet all parties agree about the importance of finding ways to maintain and improve quality while also controlling costs. The study by Scales and colleagues provides a good example to help in this quest.