0
We're unable to sign you in at this time. Please try again in a few minutes.
Retry
We were able to sign you in, but your subscription(s) could not be found. Please try again in a few minutes.
Retry
There may be a problem with your account. Please contact the AMA Service Center to resolve this issue.
Contact the AMA Service Center:
Telephone: 1 (800) 262-2350 or 1 (312) 670-7827  *   Email: subscriptions@jamanetwork.com
Error Message ......
Review |

Effectiveness of Teaching Quality Improvement to Clinicians:  A Systematic Review FREE

Romsai T. Boonyasai, MD, MPH; Donna M. Windish, MD, MPH; Chayan Chakraborti, MD; Leonard S. Feldman, MD; Haya R. Rubin, MD, PhD; Eric B. Bass, MD, MPH
[+] Author Affiliations

Author Affiliations: Department of Internal Medicine, Johns Hopkins School of Medicine, Baltimore, Maryland (Drs Boonyasai, Feldman, and Bass); Department of Internal Medicine, Yale University School of Medicine, New Haven, Connecticut (Dr Windish); Department of Internal Medicine, George Washington University School of Medicine, Washington, DC (Dr Chakraborti); and Palo Alto Medical Foundation Research Institute, Palo Alto, California (Dr Rubin).

More Author Information
JAMA. 2007;298(9):1023-1037. doi:10.1001/jama.298.9.1023.
Text Size: A A A
Published online

Context Accreditation requirements mandate teaching quality improvement (QI) concepts to medical trainees, yet little is known about the effectiveness of teaching QI.

Objectives To perform a systematic review of the effectiveness of published QI curricula for clinicians and to determine whether teaching methods influence the effectiveness of such curricula.

Data Sources The electronic literature databases of MEDLINE, EMBASE, CINAHL, and ERIC were searched for English-language articles published between January 1, 1980, and April 30, 2007. Experts in the field of QI were queried about relevant studies.

Study Selection Two independent reviewers selected studies for inclusion if the curriculum taught QI principles to clinicians and the evaluation used a comparative study design.

Data Extraction Information about the features of each curriculum, its use of 9 principles of adult learning, and the type of educational and clinical outcomes were extracted. The relationship between the outcomes and the number of educational principles used was assessed.

Results Of 39 studies that met eligibility criteria, 31 described team-based projects; 37 combined didactic instruction with experiential learning. The median number of adult learning principles used was 7 (range, 2-8). Evaluations included 22 controlled trials (8 randomized and 14 nonrandomized) and 17 pre/post or time series studies. Fourteen studies described educational outcomes (attitudes, knowledge, or skills or behaviors) and 28 studies described clinical process or patient outcomes. Nine of the 10 studies that evaluated knowledge reported only positive effects but only 2 of these described a validated assessment tool. The 6 assessments of attitudes found mixed results. Four of the 6 studies on skill or behavior outcomes reported only positive effects. Eight of the 28 studies of clinical outcomes reported only beneficial effects. Controlled studies were more likely than other studies to report mixed or null effects. Only 4 studies evaluated both educational and clinical outcomes, providing limited evidence that educational outcomes influence the clinical effectiveness of the interventions.

Conclusions Most published QI curricula apply sound adult learning principles and demonstrate improvement in learners' knowledge or confidence to perform QI. Additional studies are needed to determine whether educational methods have meaningful clinical benefits.

Figures in this Article

Thought leaders and professional organizations in medicine and nursing have endorsed teaching quality improvement (QI) to clinicians.15 The Accreditation Council on Graduate Medical Education has linked the accreditation of postgraduate training programs with demonstration that their physicians-in-training achieve competency in systems-based practice and practice-based learning and improvement.6 However, the potential effect of teaching QI methods to clinicians is uncertain. Most reviews of QI interventions have described educational interventions meant to improve clinicians' knowledge of or adherence to guidelines instead of providing them with skills to implement systems changes themselves.79 One of 2 reviews that examined teaching QI to clinicians focused on establishing a framework for organizing QI content for learners10 and the other was not a systematic review.5

We therefore systematically reviewed the literature to determine the effect of teaching QI to clinicians, and to determine whether the use of adult learning principles or the presence of specific curricular features is associated with the effectiveness of these educational interventions.

Literature Search

We searched for relevant studies published through April 2007 using electronic literature databases, including MEDLINE, Excerpta Medica (EMBASE), the Cumulative Index of Nursing and Allied Health Literature (CINAHL), and the Education Resource Information Center (ERIC).

The search strategy was limited to English-language articles published between January 1, 1980, and April 30, 2007, using the Medical Subject Headings (MeSH) health care economics and organizations; health care quality, access, and evaluation; health care facilities; manpower and services; health services administration; information management; informatics; medical informatics; and systems analysis. These were combined with the quality-related terms of quality improvement, quality management, continuous improvement, performance improvement, (improve, improving, manage, or managing and quality), QI, CQI, TQM, quality assurance, quality assessment, patient safety, practice based, system based, systems based, plan do study act, plan do check act, pdsa, pdca, six sigma, and lean management and the education-related terms of curriculum, curricul$, educat$, teach$, train$, learn$, and the MeSH term curriculum. The MeSH terms were exploded to include all associated subheadings, and the core search was limited using the terms NOT comment or editorial or letter.

We also queried experts in the field of QI and hand-searched the references of all included studies, relevant review articles, and the table of contents of key journals from April 2006 through May 2007. ProCite version 5.0.3 (Thompson ISI ResearchSoft, Berkeley, California) was used to store and track the search results.

Eligibility Criteria

Quality improvement theory was defined as a set of principles that involve knowledge, skills, and methods used to evaluate and implement change in a health care system using a systems-based approach.5Curriculum was defined as a formal supervised program for changing knowledge, skills, or behaviors.11Clinician was defined as any practitioner who provides direct medical care to patients, excluding dentists and clinical laboratory workers. Studies were included if they described both a curriculum that teaches QI theory to clinicians and a comparative evaluation (ie, pre/post evaluations, time series, or randomized or nonrandomized controlled evaluation). Studies were excluded if they did not (1) occur in North America, Western Europe, Australia, New Zealand, or Japan; (2) teach QI theory; (3) describe a curricular intervention; (4) target clinicians; (5) include original data; (6) have a full article available for review; or (7) publish the results of a comparative evaluation. Studies also were excluded if they were published before 1980 or were not written in English.

These criteria were chosen because industrial process improvement methods first became popular in the 1980s (no eligible studies were published before 1995) and because we were primarily interested in the effects of QI curricula in the US health care system and other similarly developed medical systems. We included curricula for practicing physicians as well as for medical trainees because QI concepts should be relevant to both groups.

Title and Abstract Review

Eligible studies were identified through 2 phases. In the first phase, 2 authors (D.M.W. and L.S.F. or R.T.B. and C.C.) independently reviewed the titles of citations to assess eligibility for review. Citations were returned for adjudication if disagreement occurred. If agreement could not be reached, the citation was retained for further review.

In the second phase, the abstracts of remaining citations were independently reviewed by 2 pairs of reviewers using forms for determining eligibility criteria. If reviewers disagreed, the study article was returned for adjudication until agreement was reached. If eligibility could not be determined because an abstract had insufficient information, the full article was retrieved. Reviewers were not masked to author or journal name because previous work has shown that such masking is unlikely to make a significant difference in the results of data abstraction.12

Study Review

All remaining studies were retrieved for data abstraction. Two reviewer pairs (R.T.B. and C.C. or D.M.W. and L.S.F.) used a standardized form to confirm eligibility of the article. Reviewers resolved disagreements by consensus. Eligible studies were abstracted using a serial review process. Three reviewers divided the primary reviews and one reviewer checked all initial abstractions (R.T.B., L.S.F., C.C., and D.M.W.). Extracted information assessed the curricular context, characteristics of targeted learners, educational strategies, evaluation methods, and results. Quality improvement curricular content was based on the Institute for Healthcare Improvement (IHI) domains of knowledge for improving health care.13 The 8 IHI domains are a well-recognized framework for classifying QI knowledge.10 They include customer knowledge; health care as a system; variation and measurement; leading, following, and making change; collaboration; developing new, locally useful knowledge; social context and accountability; and professional knowledge. Educational content that overlapped with more than 1 domain was classified into all relevant domains. Studies also were classified according to whether they involved other types of noneducational QI interventions such as audit, feedback, or patient registries by adapting the Cochrane Effective Practice and Organization of Care taxonomy.14

We hypothesized that an association exists between using sound adult learning principles and curriculum effectiveness. To evaluate the use of adult learning principles, a list of fundamental principles was developed based on a review of 5 major educational theories: andragogy, social cognitive theory, self-directed learning, constructivism, and reflective practice.15 Using this list, curricula were assessed for whether they fully or partially exhibited 9 characteristics: (1) enabling learners to be active participants, (2) providing content relating to learners' current experiences, (3) assessing learners' needs and tailoring teaching to their past experiences, (4) allowing learners to identify and pursue their own learning goals, (5) allowing learners to practice their learning, (6) supporting learners during self-directed learning, (7) providing feedback to learners, (8) facilitating learner self-reflection, and (9) role-modeling behaviors. The reviewers relied on the authors' descriptions to determine if a principle was used. For example, a curriculum was classified as fully exhibiting facilitation of reflective learning if the article specifically described using learner self-reflection, or partially exhibiting that principle if it described features likely to encourage self-reflection, such as asking learners to discuss lessons learned.

The reliability of all categorical questions was assessed for the first 27 articles abstracted. Overall agreement was 91.5% and interrater reliability was good (κ = 0.77) for data abstraction. Interrater reliability for assessing adult learning principles was moderate (κ = 0.51).

Data Synthesis and Statistical Analysis

Due to the heterogeneity of results, outcomes were classified as beneficial effects, no effects, or detrimental effects based on whether differences from either baseline (in uncontrolled studies) or controls were statistically significant at the P value level of .05 or less. If the authors did not calculate statistical significance, an effect was counted as beneficial or detrimental if the absolute change or difference in an outcome measure was greater than 10%. We used 2-tailed χ2 statistics to assess differences in proportions. All statistical analyses were performed using Intercooled Stata version 9.2 (StataCorp, College Station, Texas).

Characteristics of Eligible Studies

Of 16 278 citations identified, 39 studies met eligibility criteria (Figure). Four study articles described evaluations of 2 separate curricula.1619Table 1 summarizes the characteristics of the studies. Eighty percent of studies were published in the last 5 years. All targeted physicians, medical trainees, nurses, or nursing students. Many curricula also taught other clinicians. Ten studies targeted trainees2029 and 29 targeted nontrainees.1619,3054 Two of the IHI knowledge domains were covered by all of the curricula: leading, following, and making change; and developing new, locally useful knowledge. Nearly all of the studies described the use of didactic instruction in combination with experiential learning.1629,3133,3554 Only 8 of the studies used a randomized controlled group design.16,17,31,32,35,36,40,41Table 2 and Table 3 describe the characteristics of individual curricula, which were grouped into 3 categories to facilitate analysis: curricula for trainees, curricula for nontrainees that have an educational focus, and curricula for nontrainees that are included in a multifaceted QI intervention.

Figure. Literature Search and Study Selection Process
Graphic Jump Location
Table Graphic Jump LocationTable 1. Characteristics of Quality Improvement (QI) Curriculum Studies (N = 39)a
Table Graphic Jump LocationTable 2. Characteristics of Quality Improvement (QI) Curricula With Controlled Evaluations
Table Graphic Jump LocationTable 3. Characteristics of Quality Improvement (QI) Curricula With Uncontrolled Evaluations
Features of Curricula for Trainees

Of the 10 curricula targeting trainees, 3 involved either medical or nursing students.26,27,29 Program details varied greatly: one involved preclinical nursing students observing patients' perspectives of hospital wards for 10 weeks before learning and discussing how to apply QI methods,27 while another was a 4-year longitudinal program that integrated didactic instruction and QI activities into an existing medical school curriculum.26

Seven studies involving residents had more similarities.2025,28 All took place during ambulatory care assignments or electives, all combined didactic instruction with participation in QI activities, and none included noneducational QI interventions. Five curricula were integrated into a 4-week rotation,2022,2426 while 2 held weekly or biweekly meetings for a year.23,28

Features of Curricula for Nontrainees That Have an Educational Focus

Five of the 29 studies (17%) involving nontrainees described their primary focus as conveying QI concepts to learners.3034 Peters et al33 described a 4-month medical school faculty development program teaching QI concepts. Irvine Doran et al31 taught QI concepts to hospital-based clinicians and encouraged application of these skills in their hospitals. The remaining 3 studies described brief (1- to 7-day) continuing education–style seminars.30,32,34

Features of Curricula for Nontrainees That Are Included in a Multifaceted QI Intervention

Twenty-four of the 29 studies (83%) involving nontrainees were QI interventions with educational components.1619,3554 All obtained administrative support for the intervention and provided faculty with QI expertise. We have grouped these studies into 4 subcategories according to characteristics of the overall QI intervention.

QI Training Session Supplementing Noneducational QI Interventions. Five studies described QI curricula integrated into larger QI interventions primarily managed by project organizers.17,3538 All taught QI concepts and methods during an initial training session associated with audit and feedback. Learners then implemented QI interventions with personalized faculty support. Most of these interventions also provided learners with a package of predeveloped QI instruments, such as patient educational materials or customized flow sheets.17,3537

QI Curricula in IMPROVE and IDEAL Collaboratives. Nineteen of the eligible studies (49%) described teaching learners through QI collaboratives, whereby learners attended facilitated workshops in which they shared ideas and taught each other QI concepts related to a clinical process or disease focus. The Improving Prevention Through Organization, Vision and Empowerment (IMPROVE) and Improving Diabetes Care Through Empowerment, Active Collaboration and Leadership (IDEAL) collaboratives taught QI concepts to primary care teams focused on preventive care and diabetes management.40,41 These curricula offered didactic instruction and interactive discussions during 7 half-day workshops over an 18- to 22-month period, interspersed with QI activities that were supported by faculty telephone calls and site visits. Instead of providing learners with a preexisting package of QI interventions, they focused on facilitating networking and sharing of ideas by learners, who implemented the ideas they believed to be most appropriate for their sites.

QI Curricula in Breakthrough Series Collaboratives. Twelve studies described curricula based on the IHI-sponsored Breakthrough Series.16,18,19,4249,54 Breakthrough Series collaboratives teach a QI approach using the chronic care model (a multidisciplinary approach to providing decision support and supporting patient self-management) and the model for improvement (a conceptual model for implementing and testing process change).55 They require use of an electronic patient registry and have been used with more than 20 different conditions throughout the world. Similar to the IMPROVE and IDEAL programs, learners in Breakthrough Series curricula also attend multiple QI workshops over a 6- to 18-month period, interspersed with QI activities. However, Breakthrough Series collaboratives emphasize incremental small tests of change. For example, learners in one Breakthrough Series program implemented and tested an average of 42 small process changes over 12 months,42 in contrast to learners in an IDEAL collaborative who implemented and tested 1 comprehensive process change in 18 months.40

QI Curricula in Interventions Using Other Collaborative Models. Five studies described QI curricula in the context of other collaboratives. Three were described as modifications of the Breakthrough Series model.5052 One used a unique conceptual model for teaching QI in the neonatal intensive care unit.53 One supplemented QI training for coronary bypass graft surgery teams with observational site visits among learners' hospitals.39 All 5 studies presented learners with audit and feedback at an initial learning session followed by teleconferences or site visits to facilitate collaboration during QI projects.

Use of Adult Learning Principles

The extent to which QI curricula used adult learning principles is presented in eTable 1. Most studies described more than 6 of the 9 adult learning principles. The studies involving trainees often provided richer descriptions of the educational intervention, and as a group they described the presence of more adult learning principles than those involving nontrainees (median number of principles 8 vs 6). Among the 10 studies involving trainees,2029 all curricula allowed learners to actively contribute to their education, engaged learners with topics that were relevant to their current experiences, helped learners direct their own learning, facilitated practice through QI activities, and described feedback strategies. Only 3 studies (30%) described role modeling as a teaching method.23,26,29

The 29 studies of nontrainees described using adult learning principles less often than studies of trainees but more often than has been described for traditional continuing education.56 All of these studies facilitated learners' active contributions and addressed topics relevant to their current experiences.1619,3054 However, only 1 of these studies (3%) used role modeling,33 only 6 studies (21%) described a needs assessment,17,18,33,35,42,46 and only 18 studies (62%) reported giving feedback to learners.1618,31,3539,43,4650,5254

Results of Evaluations

Because standard measures for assessing QI curricular effectiveness do not exist,57 the studies in this review present heterogeneous outcomes (details are provided in eTable 2). Table 4 and Table 5 summarize the evaluation methods and direction of outcomes, organized by type of learner, type of intervention, and study design. Only 3 of the studies (8%) described both clinical and educational outcomes.22,29,48 Additionally, one study19 described behavioral changes related to clinical outcomes reported in another study.18

Table Graphic Jump LocationTable 4. Evaluation Methods and Direction of Educational Outcomes
Table Graphic Jump LocationTable 5. Evaluation Methods and Direction of Clinical Outcomes

With the exception of a single detrimental effect in learner attitudes,29 all outcomes had either positive or null effects. Twelve studies (31%) reported only beneficial effects,17,22,23,25,27,28,30,33,34,39,43,46 24 studies (62%) reported mixed effects (positive, null, and detrimental),1921,24,26,29,31,32,3538,41,42,44,45,4754 and 3 studies found only null effects16,18,40 (Table 6). Studies most frequently reported improved knowledge and rarely found benefits in patient outcomes. Fewer controlled studies reported only positive outcomes compared with those using uncontrolled designs (4 of 22 controlled studies vs 8 of 17 uncontrolled studies; P = .05).

Table Graphic Jump LocationTable 6. Summary of Evaluation Results of Quality Improvement Curricula
Educational Outcomes

Of 14 studies describing educational outcomes,19,2124,26,27,2934,48 4 described curricula for residents2124; 3 for medical or nursing students26,27,29; 1 for medical school faculty33; and 6 for nontrainee clinicians.19,3032,34,48 The direction of outcomes did not appear to differ by the training level of the learners.

Of the 10 studies (71%) that reported the effect of teaching QI on knowledge,21,22,24,2934,48 all 5 involving trainees reported improvement in either perceived knowledge24,27 or scores on a QI knowledge test.21,22,24,29 Four of 5 studies involving nontrainees found improvement in either perceived30,33 or tested32,34 knowledge. One study of hospital-based clinicians found no change in scores on a QI knowledge examination.31 Only 2 studies used the same knowledge test (Quality Improvement Knowledge Assessment Test). 21,22

Five of 6 studies describing attitudinal changes involved trainees.21,23,24,26,29 One found residents' confidence in their ability to provide good diabetes care improved23; another reported no change in residents' interest in QI.24 One curriculum reported improved medical students' attitudes toward managed care,26 while 2 studies found mixed attitudes toward QI interventions.21,29 In these studies, learners reported improved confidence in their own or in their organization's ability to engage in QI projects but no change was seen in their confidence to sustain systemic change. Only 1 study described attitudinal changes in nontrainees. That study found that learners' confidence in their organization's system for diabetes management improved but their confidence in their organization's ability to support staff or community linkages did not change.48

Six curricula evaluated changes in learners' skills or behaviors.19,24,3033 The only program involving trainees found that residents' perceived ability to implement QI projects improved.24 Of the 5 studies involving nontrainees, one curriculum for hospital-based teams reported improvement in functional group interactions using a validated scale,31 another curriculum for medical school faculty described improvement in their perceived ability to teach QI,33 and 3 studies described improvement in self-reported ability to implement QI projects and greater involvement in QI activities.24,30,32 A final study found no changes in organizational behaviors for 4 of 6 chronic care model domains.19

Clinical Outcomes

The 5 curricula involving trainees that evaluated clinical outcomes generally reported positive results.20,22,25,28,29 Most used uncontrolled evaluation methods with small sample sizes. Varkey et al22 reported 18% to 37% improvement in documentation processes before and after the QI interventions but did not assess statistical significance. Coleman et al25 reported 24% to 34% improvement in documentation processes (P = .001) and 26% improvement in diabetic microalbuminuria screening (P = .02). Mohr et al28 found significantly increased on-time chart availability (40%; P < .01), chart completeness (61%; P < .01), and immunization rates of infants (26%; P = .04). In a nonrandomized controlled study, Holmboe et al20 found that diabetic patients of residents in the intervention group had 12% more annual monofilament examinations (P = .02), 10% more baseline electrocardiograms (P = .01), and greater decreases in levels of hemoglobin A1c (1.1%; P = .001) and low-density lipoprotein cholesterol (20 mg/dL [0.52 mmol/L]; P = .01) than those followed up by control residents. No improvement, however, was seen in obtaining screening laboratory test results, providing immunizations, or patients' blood pressure.20 Gould et al29 described the only trainee-centered curriculum that integrated noneducational QI interventions, including chart audit tools, and patient education materials. The documentation of diabetic eye and foot examinations and patients' hemoglobin A1c levels improved significantly, although rates of documenting levels of hemoglobin A1c (which was high at baseline) did not change.29

All curricula that involved nontrainees and reported clinical outcomes were associated with other noneducational QI interventions. They reported a diverse set of outcomes, individually reporting a mean of 9.7 (range, 1-25) different clinical outcomes. The results of these studies were mixed and are detailed in eTable 2. The key outcomes of controlled studies are described herein.

Most studies that supplemented traditional audit and feedback with individual coaching in QI methods reported positive process outcomes and highlighted the importance of in-person faculty coaching.17,3538 One curriculum—described in 2 separate studies17,35—randomized pediatric practices to receive audit and feedback with either cursory instruction in process improvement methods or extensive coaching in QI methods. Margolis et al17 found higher adjusted odds of providing 4 preventive services in interventional practices than in control practices (odds ratio, 4.6; 95% confidence interval [CI], 1.6-13.2). Rosenthal et al35 reported that rates of all anticipatory counseling measures increased 18.7% to 22.6% compared with controls (P = .002 to P = .001) but found no differences in parents' health knowledge or behaviors. Similar benefits were seen in 2 uncontrolled nursing home studies that used similar teaching methods.37,38 In contrast, McClellan et al36 found that adding didactic QI training and optional telephone-based faculty support to traditional audit and feedback was associated with improvement in only 4 of 10 measures of hemodialysis quality.

Evaluations of the IMPROVE and IDEAL collaborative models reported mostly null findings.40,41 Although Solberg et al41 found increased rates of pneumococcal immunizations and offers for cholesterol screening compared with controls, the authors found no significant differences in 10 other preventive care measures. Similarly, O’Connor et al40 reported no significant improvement compared with controls in 12 diabetes process measures.

By comparison, Breakthrough Series collaboratives16,18,4249,54 reported more positive outcomes. Among the controlled studies, 1 reported only beneficial results for 5 measures of adherence to cardiac care guidelines,43 and 3 found significant improvements in 26 of 51 process outcomes and 4 of 12 patient outcomes related to asthma, cardiovascular care, and diabetes care.42,44,45 However, not all Breakthrough Series curricula reported beneficial effects. Landon et al18 found no improvement in 6 process measures related to the human immunodeficiency virus (HIV) or in the proportion of controlled viral loads. Homer et al16 reported no change in use of asthma action plans or in the patients' symptoms and suggested that incomplete learner attendance may have influenced their findings. McInnes et al19 surveyed clinicians and administrators from the HIV collaborative described by Landon et al18 and theorized that the absence of organizational changes in 4 of 6 domains of the chronic care model may explain their null findings.

In contrast to QI collaboratives held in outpatient settings, those in highly controlled inpatient settings reported beneficial effects for a focused set of outcomes.39,5053 One curriculum involving neonatal intensive care units reported significantly higher adjusted odds of on-time surfactant delivery in the intervention group compared with controls who received only audit and feedback (odds ratio, 5.38; 95% CI, 2.84-10.20).53 Similarly, O’Connor et al39 reported a significantly lower standardized mortality ratio for cardiothoracic surgery at interventional hospitals compared with historical controls (standardized mortality ratio, 0.76; 95% CI, 0.67-0.90).

Association Between Use of Adult Learning Principles and Curriculum Outcomes

An exploratory analysis was performed to assess the relationship between use of adult learning principles and curriculum effectiveness by comparing the proportion of curricula reporting only beneficial effects with the number of adult learning principles described. Curricula with only beneficial effects, mixed effects, and only null effects all described a median of 7 principles (range 2-8). No relationships with this measure were seen using only controlled studies or those with only educational outcomes.

Analyses also were performed to assess if relationships exist between clinical outcomes and learner attendance, as well as between clinical and educational outcomes. Eight studies described incomplete attendance.16,18,25,37,41,47,48,51 Among these, all 4 controlled evaluations found either no benefit or benefit in less than 25% of described outcomes.16,18,41,51 Results for both educational and clinical evaluations were only available for 4 curricula. One reported improvements in both residents' QI knowledge and documentation processes,22 and another found improvements in practitioners' attitudes and in their patients' hemoglobin A1c levels.48 Neither study evaluated the statistical significance of their clinical outcomes. Gould et al29 reported increased knowledge and confidence in QI methods for 26 of 40 survey items and statistically significant improvement in diabetes-related processes and outcomes at medical students' preceptor sites. In contrast, the null clinical findings of an HIV collaborative were linked to an absence of change in organizational behaviors.18,19

Resources Used by Curricula

Only 6 studies described QI curricular costs.20,21,41,45,46,48 In one program, residents spent an average of 119 hours in 4 weeks on their projects, faculty spent 6 hours per month teaching the curriculum, and project sponsors devoted 1 to 2 hours per week supporting residents' activities.21 These findings were comparable with other studies that reported time costs. Holmboe et al20 found that residents committed 4 half days and faculty committed 3 hours over 4 weeks, while Solberg et al41 reported that practicing clinicians spent 5 to 11.7 hours per month on their projects.

Monetary costs varied greatly and likely reflected resources needed to integrate QI curricula with other noneducational QI interventions. Ogrinc et al21 described the direct costs of their residency-based curriculum as negligible, whereas health care organizations each paid $12 500 plus travel expenses to enroll a 3-person team in a Breakthrough Series curriculum.45,46,48

To our knowledge, this is the first systematic review to evaluate the effect of teaching QI to clinicians. The 39 studies in our sample varied in their coverage of topics within the IHI domains of QI knowledge. All described teaching content required for implementing and evaluating local systems changes but few taught learners about the larger health care system or how to view the system from a patient's perspective.

Fourteen studies evaluating educational outcomes collectively suggest that using a variety of QI teaching strategies improves QI-related knowledge, as well as attitudes toward health systems and participation in QI activities for learners of many training levels and professions. However, relatively weak study designs and evaluation instruments hamper these assessments. Most educational evaluations use uncontrolled designs, which are vulnerable to secular trends and other biases. Furthermore, those that report changes in knowledge often rely on learners' self-report of knowledge, and the validity of this assessment method has been questioned.58 Of those using knowledge tests, only 2 studies use a validated test.21,22 The validity of learner attitudinal and behavioral assessments is similarly uncertain.

Curricula with clinical outcomes provide mixed evidence for improvement in processes and little evidence for beneficial patient outcomes. Those involving trainees report mostly beneficial outcomes in documentation and disease management processes but use relatively weak study designs.

All nontrainee curricula with clinical outcomes were components of multifaceted QI interventions that provided expert faculty and received administrative support. Of these, the IMPROVE and IDEAL programs placed greatest emphasis on theoretical QI content and included the fewest external interventions. Although their teaching strategies applied many adult learning principles, they nevertheless found little evidence for clinical benefit compared with nonparticipant controls. Their null findings stand in contrast to the more positive findings of programs that combined QI training with noncollaborative QI interventions. From these studies, it appears that providing learners with a package of QI tools and individualized coaching in QI methods can improve clinical outcomes compared with audit and feedback and limited QI training.

The positive outcomes associated with these programs and with the Breakthrough Series curricula also suggest that providing learners with access to preexisting performance data or process improvement tools can improve outcomes. For example, the IMPROVE and IDEAL curricula encouraged learners to collect their own performance data, whereas curricula in the Breakthrough Series collaboratives and those supplementing noncollaborative QI interventions provided learners with data from an electronic registry or external audits. The positive outcomes seen in curricula set in data-rich inpatient and intensive care environments also support the importance of access to performance data. Similarly, many curricula with positive outcomes also provided learners with preexisting decision support or patient education tools, which may have allowed them to focus on implementing interventions instead of developing their own tools. The relatively greater benefits seen in Breakthrough Series curricula also suggest that encouraging learners to implement and test interventions through several small cycles of trial and error may result in better outcomes than undertaking a single comprehensive intervention.

Nevertheless, the null findings in studies of HIV and pediatric Breakthrough Series collaboratives indicate that performing incremental tests of change and having ready access to performance data are insufficient to ensure beneficial clinical outcomes. The attendance issues discussed by Homer et al16 and the absence of organizational changes following the HIV collaborative18,19 may indicate that how well clinicians learned QI concepts influences clinical outcomes. The findings of 4 curricula for which educational and clinical outcomes are available18,19,22,29,48 suggest that clinical benefits occur when educational outcomes improve and are absent when learners’ behaviors do not change. Although our analysis did not find convincing evidence for an association between the use of adult learning principles and curriculum effectiveness, we were limited to the details that the authors published and we could not reliably assess important details such as attendance, learners' prior experiences, faculty QI expertise, or faculty teaching skill. Furthermore, counting curricula with only beneficial, mixed, or only null effects provides a crude measure of curriculum effectiveness.

This review has several limitations. First, it is subject to publication bias. Although we could not formally test for this, we did examine the sample size distribution for studies that had positive, mixed, and null effects, and found a relatively similar sample size distribution for each group, arguing against the presence of major publication bias. Second, our findings may not apply to medical education in all countries because we limited our review to studies that were published in English and occurred in countries with health care systems similar to the United States. In doing so, we inadvertently excluded countries such as Korea, Singapore, and Taiwan, which also have highly developed health care systems. However, in a retrospective review of the citations we excluded, we found no studies from those countries that would have met our eligibility criteria. Third, we focused on curricula that described results with comparisons to either baseline results or with controls, and thus excluded several descriptions of innovative QI curricula.5963 We did so because we wished to examine the effects of teaching QI concepts instead of trying to describe the variety of ways in which they have been taught. Similarly, we excluded studies about QI interventions that did not provide at least an adequate description of the curriculum because we hoped to determine the effect of educational methods on curriculum effectiveness. Also, our analysis of the relationship between use of adult learning principles and curriculum effectiveness should be considered exploratory because no validated instrument exists for assessing the use of adult learning principles. Finally, our ability to combine outcomes was limited by their heterogeneity, which is an ongoing issue in systematic reviews of educational and organizational interventions in health care.64,65

Despite these limitations, we offer some recommendations for developing and evaluating future QI curricula. Published QI curricula teach methods for implementing systems change but infrequently address the IHI knowledge domains of health care as a system and customer knowledge. Because there was little evidence for improvement in patient knowledge or behaviors despite improvements in several measures of counseling processes, we recommend that future QI curricula also cover these IHI-defined domains. To advance the understanding of how teaching QI influences educational and clinical outcomes, educators should describe key features of their curricula, including learner and faculty characteristics, attendance, and learners' interventions. They also should compare learners with controls and present both educational and clinical outcomes. Part of the attractiveness of teaching QI to clinicians is that learners will theoretically expand process improvement efforts independently.3,66 Thus, future curricula may be evaluated based on whether learners' attitudes, knowledge, and skills improve, particularly if improvements are associated with intermediate clinical gains. Although past efforts to assess educational outcomes in QI have been hampered by the lack of assessment tools, we did find examples of potentially useful instruments for assessing QI-related knowledge and skills.21,67,68

We also recommend that future curricula teach collaborative skills, facilitate experiential learning with incremental change from trial and error, and provide learners with opportunities to work closely with colleagues from other disciplines. These are accepted principles of adult learning, and collaborative small tests of change appear to be important for achieving clinical benefit. Although evidence suggests that intensive coaching from expert faculty improves clinical outcomes, few educators currently have QI expertise. Furthermore, it is important to balance the desire for improved processes with the need for clinicians to learn through trial and error. Quality improvement curricula should provide learners with access to resources that facilitate their QI activities, such as performance data and predeveloped process improvement tools, because evidence suggests that these resources may enhance the clinical effectiveness of teaching QI. Although such curricula can be complex, time-consuming, and expensive, we found several examples of curricula that exhibited these features and were successfully incorporated into busy training programs.20,21,24,29

Corresponding Author: Romsai T. Boonyasai, MD, MPH, Collaborative Internal Medicine Service, Johns Hopkins Bayview Medical Center, 5200 Eastern Ave, West Tower, Sixth Floor, Baltimore, MD 21224 (romsai@jhmi.edu).

Author Contributions: Dr Boonyasai had full access to all of the data in the study and takes responsibility for the integrity of the data and the accuracy of the data analysis.

Study concept and design: Boonyasai, Windish, Chakraborti, Rubin, Bass.

Acquisition of data: Boonyasai, Windish, Chakraborti, Feldman.

Analysis and interpretation of data: Boonyasai, Feldman, Rubin, Bass.

Drafting of the manuscript: Boonyasai.

Critical revision of the manuscript for important intellectual content: Boonyasai, Windish, Chakraborti, Feldman, Rubin, Bass.

Statistical analysis: Boonyasai, Chakraborti.

Obtained funding: Rubin.

Study supervision: Rubin, Bass.

Financial Disclosures: None reported.

Funding/Support: The authors are members of the Quality Improvement Curriculum Committee appointed by the Society of General Internal Medicine, which provided small honoraria to Drs Boonyasai, Windish, Chakraborti, and Feldman, as well as support for a portion of the time spent by research staff. Dr Boonyasai was supported by a National Heart, Lung, and Blood Institute T32 training grant (HL07180). Dr Chakraborti was supported by National Research Service Award-Health Resources and Services Administration fellowship training grant (5-T-32-HP10025). Drs Bass and Feldman receive support from the Osler Center for Clinical Excellence at Johns Hopkins University.

Role of the Sponsor: The funding organizations and professional societies had no input in the design and conduct of this study; in the collection, analysis, and interpretation of the data; or in the preparation, review or approval of the manuscript.

Additional Contributions: Renee Wilson, MS (Johns Hopkins University Evidence-Based Practice Center, Baltimore, Maryland), and Cindy Sheffield, MBA, MLS (Welch Medical Library, and Johns Hopkins University, Baltimore), provided assistance in performing the literature search and using the data management software. Haeseong Park, MD, MPH (Johns Hopkins University Evidence-Based Practice Center and Bloomberg School of Public Health), provided assistance with hand searching. Ira Wilson, MD, MSc (Tufts-New England Medical Center, Boston, Massachusetts), provided intellectual support for this review. None of the persons listed received compensation for their assistance with this project.

American Association of Colleges of Nursing.  The Essentials of Master's Education for Advanced Practice Nursing. Washington, DC: American Association of Colleges of Nursing; 1996:6-9
Institute of Medicine.  Crossing the Quality Chasm. Washington, DC: National Academy Press; 2001:207-214
Berwick DM. Continuous improvement as an ideal in health care.  N Engl J Med. 1989;320(1):53-56
PubMed   |  Link to Article
Headrick LA, Neuhauser D, Schwab P, Stevens DP. Continuous quality improvement and the education of the generalist physician.  Acad Med. 1995;70(1):(suppl)  S104-S109
PubMed   |  Link to Article
Institute of Medicine; Committee on Health Professions Education Summit.  Health Professions Education: A Bridge to Quality. Washington, DC: National Academy Press; 2003:45-96
Accreditation Council on Graduate Medical Education Outcome Project Web site. http://www.acgme.org/outcome/comp/compFull.asp. Accessed May 9, 2006
Oxman AD, Thomson MA, Davis DA, Haynes RB. No magic bullets: a systematic review of 102 trials of interventions to improve professional practice.  CMAJ. 1995;153(10):1423-1431
PubMed
Gilbody S, Whitty P, Grimshaw J, Thomas R. Educational and organizational interventions to improve the management of depression in primary care: a systematic review.  JAMA. 2003;289(23):3145-3151
PubMed   |  Link to Article
Evans D, Sheares BJ, Vazquez TL. Educating health professionals to improve quality of care for asthma.  Paediatr Respir Rev. 2004;5(4):304-310
PubMed   |  Link to Article
Ogrinc G, Headrick LA, Mutha S, Coleman MT, O'Donnell J, Miles PV. A framework for teaching medical students and residents about practice-based learning and improvement, synthesized from a literature review.  Acad Med. 2003;78(7):748-756
PubMed   |  Link to Article
Kern DE, Thomas PA, Howard DM, Bass EB. Curriculum Development for Medical Education: A Six-Step Approach. Baltimore, MD: Johns Hopkins University Press; 1998:1
Berlin JA.University of Pennsylvania Meta-analysis Blinding Study Group.  Does blinding of readers affect the results of meta-analyses?  Lancet. 1997;350(9072):185-186
PubMed   |  Link to Article
Batalden P, Berwick DM, Bisognano M, Splaine M, Baker G, Headrick L. Knowledge Domains for Health Professional Students Seeking Competency in the Continual Improvement and Innovation of Health Care. Boston, MA: Institute for Healthcare Improvement; 1998:1-2
Shojania KG, McDonald KM, Wachter RM, Owens DK. Closing the Quality Gap: A Critical Analysis of Quality Improvement Strategies. Vol 1. Rockville, MD: Agency for Healthcare Research and Quality; 2006:14-18
Kaufman DM. Applying educational theory in practice.  BMJ. 2003;326(7382):213-216
PubMed   |  Link to Article
Homer CJ, Forbes P, Horvitz L, Peterson LE, Wypij D, Heinrich P. Impact of a quality improvement program on care and outcomes for children with asthma.  Arch Pediatr Adolesc Med. 2005;159(5):464-469
PubMed   |  Link to Article
Margolis PA, Lannon CM, Stuart JM, Fried BJ, Keyes-Elstein L, Moore DEJ. Practice based education to improve delivery systems for prevention in primary care: randomised trial.  BMJ. 2004;328(7436):388
PubMed   |  Link to Article
Landon BE, Wilson IB, McInnes K.  et al.  Effects of a quality improvement collaborative on the outcome of care of patients with HIV infection: the EQHIV study.  Ann Intern Med. 2004;140(11):887-896
PubMed   |  Link to Article
McInnes DK, Landon BE, Wilson IB.  et al.  The impact of a quality improvement program on systems, processes, and structures in medical clinics.  Med Care. 2007;45(5):463-471
PubMed   |  Link to Article
Holmboe ES, Prince L, Green M. Teaching and improving quality of care in a primary care internal medicine residency clinic.  Acad Med. 2005;80(6):571-577
PubMed   |  Link to Article
Ogrinc G, Headrick LA, Morrison LJ, Foster T. Teaching and assessing resident competence in practice-based learning and improvement.  J Gen Intern Med. 2004;19(5 pt 2):496-500
PubMed   |  Link to Article
Varkey P, Reller MK, Smith A, Ponto J, Osborn M. An experiential interdisciplinary quality improvement education initiative.  Am J Med Qual. 2006;21(5):317-322
PubMed   |  Link to Article
Nuovo J, Balsbaugh T, Barton S.  et al.  Development of a diabetes care management curriculum in a family practice residency program.  Dis Manage. 2004;7(4):314-324
Link to Article
Djuricich AM, Ciccarelli M, Swigonski NL. A continuous quality improvement curriculum for residents.  Acad Med. 2004;79(10):(suppl)  S65-S67
PubMed   |  Link to Article
Coleman MT, Nasraty S, Ostapchuk M, Wheeler S, Looney S, Rhodes S. Introducing practice-based learning and improvement ACGME core competencies into a family medicine residency curriculum.  Jt Comm J Qual Saf. 2003;29(5):238-247
PubMed
O'Connell MT, Rivo ML, Mechaber AJ, Weiss BA. A curriculum in systems-based care.  Fam Med. 2004;36:(suppl)  S99-S104
PubMed
Kyrkjebø JM, Hanssen TA, Haugland BO. Introducing quality improvement to pre-qualification nursing students.  Qual Health Care. 2001;10(4):204-210
PubMed   |  Link to Article
Mohr JJ, Randolph GD, Laughon MM, Schaff E. Integrating improvement competencies into residency education.  Ambul Pediatr. 2003;3(3):131-136
PubMed   |  Link to Article
Gould BE, Grey MR, Huntington CG.  et al.  Improving patient care outcomes by teaching quality improvement to medical students in community-based practices.  Acad Med. 2002;77(10):1011-1018
PubMed   |  Link to Article
François P, Vinck D, Labarere J, Reverdy T, Peyrin JC. Assessment of an intervention to train teaching hospital care providers in quality management.  Qual Saf Health Care. 2005;14(4):234-239
PubMed   |  Link to Article
Irvine Doran DM, Baker GR, Murray M.  et al.  Achieving clinical improvement: an interdisciplinary intervention.  Health Care Manage Rev. 2002;27(4):42-56
PubMed   |  Link to Article
Saturno PJ. Training health professionals to implement quality improvement activities.  Int J Qual Health Care. 1995;7(2):119-126
PubMed   |  Link to Article
Peters AS, Ladden MD, Kotch JB, Fletcher RH. Evaluation of a faculty development program in managing care.  Acad Med. 2002;77(11):1121-1127
PubMed   |  Link to Article
Palumbo G, Morosini P, Picardi A, Gigantesco A, Geddes J. Educating mental health professionals in clinical epidemiology and continuous quality improvement.  Adm Policy Ment Health. 2004;31(5):421-424
PubMed   |  Link to Article
Rosenthal MS, Lannon CM, Stuart JM, Brown L, Miller WC, Margolis PA. A randomized trial of practice-based education to improve delivery systems for anticipatory guidance.  Arch Pediatr Adolesc Med. 2005;159(5):456-463
PubMed   |  Link to Article
McClellan WM, Hodgin E, Pastan S, McAdams L, Soucie M. A randomized evaluation of two health care quality improvement program (HCQIP) interventions to improve the adequacy of hemodialysis care of ESRD patients.  J Am Soc Nephrol. 2004;15(3):754-760
PubMed   |  Link to Article
Horner JK, Hanson LC, Wood D, Silver AG, Reynolds KS. Using quality improvement to address pain management practices in nursing homes.  J Pain Symptom Manage. 2005;30(3):271-277
PubMed   |  Link to Article
Hanson LC, Reynolds K, Henderson M, Pickard C. A quality improvement intervention to increase palliative care in nursing homes.  J Palliat Med. 2005;8(3):576-584
PubMed   |  Link to Article
O'Connor GT, Plume SK, Olmstead EM.  et al. Northern New England Cardiovascular Disease Study Group.  A regional intervention to improve the hospital mortality associated with coronary artery bypass graft surgery.  JAMA. 1996;275(11):841-846
PubMed   |  Link to Article
O'Connor PJ, Desai J, Solberg LI.  et al.  Randomized trial of quality improvement intervention to improve diabetes care in primary care settings.  Diabetes Care. 2005;28(8):1890-1897
PubMed   |  Link to Article
Solberg LI, Kottke TE, Brekke ML.  et al.  Failure of a continuous quality improvement intervention to increase the delivery of preventive services.  Eff Clin Pract. 2000;3(3):105-115
PubMed
Landon BE, Hicks LS, O'Malley AJ.  et al.  Improving the management of chronic disease at community health centers.  N Engl J Med. 2007;356(9):921-934
PubMed   |  Link to Article
Peterson A, Carlhed R, Lindahl B.  et al.  Improving guideline adherence through intensive quality improvement and the use of a National Quality Register in Sweden for acute myocardial infarction.  Qual Manag Health Care. 2007;16(1):25-37
PubMed
Mangione-Smith R, Schonlau M, Chan KS.  et al.  Measuring the effectiveness of a collaborative for quality improvement in pediatric asthma care.  Ambul Pediatr. 2005;5(2):75-82
PubMed   |  Link to Article
Asch SM, Baker DW, Keesey JW.  et al.  Does the collaborative model improve care for chronic heart failure?  Med Care. 2005;43(7):667-675
PubMed   |  Link to Article
Boushon B, Provost L, Gagnon J, Carver P. Using a virtual breakthrough series collaborative to improve access in primary care.  Jt Comm J Qual Patient Saf. 2006;32(10):573-584
PubMed
Chin MH, Cook S, Drum ML.  et al.  Improving diabetes care in midwest community health centers with the health disparities collaborative.  Diabetes Care. 2004;27(1):2-8
PubMed   |  Link to Article
Wagner EH, Glasgow RE, Davis C.  et al.  Quality improvement in chronic illness care.  Jt Comm J Qual Improv. 2001;27(2):63-80
PubMed
Weeks WB, Mills PD, Waldron J, Brown SH, Speroff T, Coulson LR. A model for improving the quality and timeliness of compensation and pension examinations in VA facilities.  J Healthc Manag. 2003;48(4):252-262
PubMed
Young PC, Glade GB, Stoddard GJ, Norlin C. Evaluation of a learning collaborative to improve the delivery of preventive services by pediatric practices.  Pediatrics. 2006;117(5):1469-1476
PubMed   |  Link to Article
Colón-Emeric C, Schenck A, Gorospe J.  et al.  Translating evidence-based falls prevention into clinical practice in nursing facilities.  J Am Geriatr Soc. 2006;54(9):1414-1418
PubMed   |  Link to Article
Baier RR, Gifford DR, Patry G.  et al.  Ameliorating pain in nursing homes.  J Am Geriatr Soc. 2004;52(12):1988-1995
PubMed   |  Link to Article
Horbar JD, Carpenter JH, Buzas J.  et al.  Collaborative quality improvement to promote evidence based surfactant for preterm infants.  BMJ. 2004;329(7473):1004
PubMed   |  Link to Article
Stoeckle-Roberts S, Reeves MJ, Jacobs BS.  et al.  Closing gaps between evidence-based stroke care guidelines and practices with a collaborative quality improvement project.  Jt Comm J Qual Patient Saf. 2006;32(9):517-527
Institute for Healthcare Improvement.  The Breakthrough Series: IHI's Collaborative Model for Achieving Breakthrough Improvement. Cambridge, MA: Institute for Healthcare Improvement; 2003:1-20
Marinopoulos SS, Dorman T, Ratanawongsa N.  et al.  Effectiveness of Continuing Medical Education. Rockville, MD: Agency for Healthcare Research and Quality; 2007:28
Mittman BS. Creating the evidence base for quality improvement collaboratives.  Ann Intern Med. 2004;140(11):897-901
PubMed   |  Link to Article
Davis DA, Mazmanian PE, Fordis M, Van Harrison R, Thorpe KE, Perrier L. Accuracy of physician self-assessment compared with observed measures of competence.  JAMA. 2006;296(9):1094-1102
PubMed   |  Link to Article
Moore SM, Alemi F, Headrick LA.  et al.  Using learning cycles to build an interdisciplinary curriculum in CI for health professions students in Cleveland.  Jt Comm J Qual Improv. 1996;22(3):165-171
PubMed
Headrick LA, Moore SM, Alemi F.  et al.  Using PDSA (Plan-Do-Study-Act) to establish academic-community partnerships.  Qual Manag Health Care. 1998;6(2):12-20
PubMed
Ellrodt AG. Introduction of total quality management (TQM) into an internal medicine residency.  Acad Med. 1993;68(11):817-823
PubMed   |  Link to Article
Ladden MD, Bednash G, Stevens DP, Moore GT. Educating interprofessional learners for quality, safety and systems improvement.  J Interprof Care. 2006;20(5):497-505
PubMed   |  Link to Article
Weingart SN, Tess A, Driver J, Aronson MD, Sands K. Creating a quality improvement elective for medical house officers.  J Gen Intern Med. 2004;19(8):861-867
PubMed   |  Link to Article
Bravata DM, McDonald KM, Shojania KG, Sundaram V, Owens DK. Challenges in systematic reviews.  Ann Intern Med. 2005;142(12 pt 2):1056-1065
PubMed   |  Link to Article
Reed D, Price EG, Windish DM.  et al.  Challenges in systematic reviews of educational intervention studies.  Ann Intern Med. 2005;142(12 pt 2):1080-1089
PubMed   |  Link to Article
Laffel G, Blumenthal D. The case for using industrial quality management science in health care organizations.  JAMA. 1989;262(20):2869-2873
PubMed   |  Link to Article
Watson WE, Michaelsen LK, Sharp W. Member competence, group interaction, and group decision making: a longitudinal study.  J Appl Psychol. 1991;76(6):803-809
Link to Article
Varkey P, Natt N. The Objective Structured Clinical Examination as an educational tool in patient safety.  Jt Comm J Qual Patient Saf. 2007;33(1):48-53
PubMed

Figures

Figure. Literature Search and Study Selection Process
Graphic Jump Location

Tables

Table Graphic Jump LocationTable 1. Characteristics of Quality Improvement (QI) Curriculum Studies (N = 39)a
Table Graphic Jump LocationTable 2. Characteristics of Quality Improvement (QI) Curricula With Controlled Evaluations
Table Graphic Jump LocationTable 3. Characteristics of Quality Improvement (QI) Curricula With Uncontrolled Evaluations
Table Graphic Jump LocationTable 4. Evaluation Methods and Direction of Educational Outcomes
Table Graphic Jump LocationTable 5. Evaluation Methods and Direction of Clinical Outcomes
Table Graphic Jump LocationTable 6. Summary of Evaluation Results of Quality Improvement Curricula

References

American Association of Colleges of Nursing.  The Essentials of Master's Education for Advanced Practice Nursing. Washington, DC: American Association of Colleges of Nursing; 1996:6-9
Institute of Medicine.  Crossing the Quality Chasm. Washington, DC: National Academy Press; 2001:207-214
Berwick DM. Continuous improvement as an ideal in health care.  N Engl J Med. 1989;320(1):53-56
PubMed   |  Link to Article
Headrick LA, Neuhauser D, Schwab P, Stevens DP. Continuous quality improvement and the education of the generalist physician.  Acad Med. 1995;70(1):(suppl)  S104-S109
PubMed   |  Link to Article
Institute of Medicine; Committee on Health Professions Education Summit.  Health Professions Education: A Bridge to Quality. Washington, DC: National Academy Press; 2003:45-96
Accreditation Council on Graduate Medical Education Outcome Project Web site. http://www.acgme.org/outcome/comp/compFull.asp. Accessed May 9, 2006
Oxman AD, Thomson MA, Davis DA, Haynes RB. No magic bullets: a systematic review of 102 trials of interventions to improve professional practice.  CMAJ. 1995;153(10):1423-1431
PubMed
Gilbody S, Whitty P, Grimshaw J, Thomas R. Educational and organizational interventions to improve the management of depression in primary care: a systematic review.  JAMA. 2003;289(23):3145-3151
PubMed   |  Link to Article
Evans D, Sheares BJ, Vazquez TL. Educating health professionals to improve quality of care for asthma.  Paediatr Respir Rev. 2004;5(4):304-310
PubMed   |  Link to Article
Ogrinc G, Headrick LA, Mutha S, Coleman MT, O'Donnell J, Miles PV. A framework for teaching medical students and residents about practice-based learning and improvement, synthesized from a literature review.  Acad Med. 2003;78(7):748-756
PubMed   |  Link to Article
Kern DE, Thomas PA, Howard DM, Bass EB. Curriculum Development for Medical Education: A Six-Step Approach. Baltimore, MD: Johns Hopkins University Press; 1998:1
Berlin JA.University of Pennsylvania Meta-analysis Blinding Study Group.  Does blinding of readers affect the results of meta-analyses?  Lancet. 1997;350(9072):185-186
PubMed   |  Link to Article
Batalden P, Berwick DM, Bisognano M, Splaine M, Baker G, Headrick L. Knowledge Domains for Health Professional Students Seeking Competency in the Continual Improvement and Innovation of Health Care. Boston, MA: Institute for Healthcare Improvement; 1998:1-2
Shojania KG, McDonald KM, Wachter RM, Owens DK. Closing the Quality Gap: A Critical Analysis of Quality Improvement Strategies. Vol 1. Rockville, MD: Agency for Healthcare Research and Quality; 2006:14-18
Kaufman DM. Applying educational theory in practice.  BMJ. 2003;326(7382):213-216
PubMed   |  Link to Article
Homer CJ, Forbes P, Horvitz L, Peterson LE, Wypij D, Heinrich P. Impact of a quality improvement program on care and outcomes for children with asthma.  Arch Pediatr Adolesc Med. 2005;159(5):464-469
PubMed   |  Link to Article
Margolis PA, Lannon CM, Stuart JM, Fried BJ, Keyes-Elstein L, Moore DEJ. Practice based education to improve delivery systems for prevention in primary care: randomised trial.  BMJ. 2004;328(7436):388
PubMed   |  Link to Article
Landon BE, Wilson IB, McInnes K.  et al.  Effects of a quality improvement collaborative on the outcome of care of patients with HIV infection: the EQHIV study.  Ann Intern Med. 2004;140(11):887-896
PubMed   |  Link to Article
McInnes DK, Landon BE, Wilson IB.  et al.  The impact of a quality improvement program on systems, processes, and structures in medical clinics.  Med Care. 2007;45(5):463-471
PubMed   |  Link to Article
Holmboe ES, Prince L, Green M. Teaching and improving quality of care in a primary care internal medicine residency clinic.  Acad Med. 2005;80(6):571-577
PubMed   |  Link to Article
Ogrinc G, Headrick LA, Morrison LJ, Foster T. Teaching and assessing resident competence in practice-based learning and improvement.  J Gen Intern Med. 2004;19(5 pt 2):496-500
PubMed   |  Link to Article
Varkey P, Reller MK, Smith A, Ponto J, Osborn M. An experiential interdisciplinary quality improvement education initiative.  Am J Med Qual. 2006;21(5):317-322
PubMed   |  Link to Article
Nuovo J, Balsbaugh T, Barton S.  et al.  Development of a diabetes care management curriculum in a family practice residency program.  Dis Manage. 2004;7(4):314-324
Link to Article
Djuricich AM, Ciccarelli M, Swigonski NL. A continuous quality improvement curriculum for residents.  Acad Med. 2004;79(10):(suppl)  S65-S67
PubMed   |  Link to Article
Coleman MT, Nasraty S, Ostapchuk M, Wheeler S, Looney S, Rhodes S. Introducing practice-based learning and improvement ACGME core competencies into a family medicine residency curriculum.  Jt Comm J Qual Saf. 2003;29(5):238-247
PubMed
O'Connell MT, Rivo ML, Mechaber AJ, Weiss BA. A curriculum in systems-based care.  Fam Med. 2004;36:(suppl)  S99-S104
PubMed
Kyrkjebø JM, Hanssen TA, Haugland BO. Introducing quality improvement to pre-qualification nursing students.  Qual Health Care. 2001;10(4):204-210
PubMed   |  Link to Article
Mohr JJ, Randolph GD, Laughon MM, Schaff E. Integrating improvement competencies into residency education.  Ambul Pediatr. 2003;3(3):131-136
PubMed   |  Link to Article
Gould BE, Grey MR, Huntington CG.  et al.  Improving patient care outcomes by teaching quality improvement to medical students in community-based practices.  Acad Med. 2002;77(10):1011-1018
PubMed   |  Link to Article
François P, Vinck D, Labarere J, Reverdy T, Peyrin JC. Assessment of an intervention to train teaching hospital care providers in quality management.  Qual Saf Health Care. 2005;14(4):234-239
PubMed   |  Link to Article
Irvine Doran DM, Baker GR, Murray M.  et al.  Achieving clinical improvement: an interdisciplinary intervention.  Health Care Manage Rev. 2002;27(4):42-56
PubMed   |  Link to Article
Saturno PJ. Training health professionals to implement quality improvement activities.  Int J Qual Health Care. 1995;7(2):119-126
PubMed   |  Link to Article
Peters AS, Ladden MD, Kotch JB, Fletcher RH. Evaluation of a faculty development program in managing care.  Acad Med. 2002;77(11):1121-1127
PubMed   |  Link to Article
Palumbo G, Morosini P, Picardi A, Gigantesco A, Geddes J. Educating mental health professionals in clinical epidemiology and continuous quality improvement.  Adm Policy Ment Health. 2004;31(5):421-424
PubMed   |  Link to Article
Rosenthal MS, Lannon CM, Stuart JM, Brown L, Miller WC, Margolis PA. A randomized trial of practice-based education to improve delivery systems for anticipatory guidance.  Arch Pediatr Adolesc Med. 2005;159(5):456-463
PubMed   |  Link to Article
McClellan WM, Hodgin E, Pastan S, McAdams L, Soucie M. A randomized evaluation of two health care quality improvement program (HCQIP) interventions to improve the adequacy of hemodialysis care of ESRD patients.  J Am Soc Nephrol. 2004;15(3):754-760
PubMed   |  Link to Article
Horner JK, Hanson LC, Wood D, Silver AG, Reynolds KS. Using quality improvement to address pain management practices in nursing homes.  J Pain Symptom Manage. 2005;30(3):271-277
PubMed   |  Link to Article
Hanson LC, Reynolds K, Henderson M, Pickard C. A quality improvement intervention to increase palliative care in nursing homes.  J Palliat Med. 2005;8(3):576-584
PubMed   |  Link to Article
O'Connor GT, Plume SK, Olmstead EM.  et al. Northern New England Cardiovascular Disease Study Group.  A regional intervention to improve the hospital mortality associated with coronary artery bypass graft surgery.  JAMA. 1996;275(11):841-846
PubMed   |  Link to Article
O'Connor PJ, Desai J, Solberg LI.  et al.  Randomized trial of quality improvement intervention to improve diabetes care in primary care settings.  Diabetes Care. 2005;28(8):1890-1897
PubMed   |  Link to Article
Solberg LI, Kottke TE, Brekke ML.  et al.  Failure of a continuous quality improvement intervention to increase the delivery of preventive services.  Eff Clin Pract. 2000;3(3):105-115
PubMed
Landon BE, Hicks LS, O'Malley AJ.  et al.  Improving the management of chronic disease at community health centers.  N Engl J Med. 2007;356(9):921-934
PubMed   |  Link to Article
Peterson A, Carlhed R, Lindahl B.  et al.  Improving guideline adherence through intensive quality improvement and the use of a National Quality Register in Sweden for acute myocardial infarction.  Qual Manag Health Care. 2007;16(1):25-37
PubMed
Mangione-Smith R, Schonlau M, Chan KS.  et al.  Measuring the effectiveness of a collaborative for quality improvement in pediatric asthma care.  Ambul Pediatr. 2005;5(2):75-82
PubMed   |  Link to Article
Asch SM, Baker DW, Keesey JW.  et al.  Does the collaborative model improve care for chronic heart failure?  Med Care. 2005;43(7):667-675
PubMed   |  Link to Article
Boushon B, Provost L, Gagnon J, Carver P. Using a virtual breakthrough series collaborative to improve access in primary care.  Jt Comm J Qual Patient Saf. 2006;32(10):573-584
PubMed
Chin MH, Cook S, Drum ML.  et al.  Improving diabetes care in midwest community health centers with the health disparities collaborative.  Diabetes Care. 2004;27(1):2-8
PubMed   |  Link to Article
Wagner EH, Glasgow RE, Davis C.  et al.  Quality improvement in chronic illness care.  Jt Comm J Qual Improv. 2001;27(2):63-80
PubMed
Weeks WB, Mills PD, Waldron J, Brown SH, Speroff T, Coulson LR. A model for improving the quality and timeliness of compensation and pension examinations in VA facilities.  J Healthc Manag. 2003;48(4):252-262
PubMed
Young PC, Glade GB, Stoddard GJ, Norlin C. Evaluation of a learning collaborative to improve the delivery of preventive services by pediatric practices.  Pediatrics. 2006;117(5):1469-1476
PubMed   |  Link to Article
Colón-Emeric C, Schenck A, Gorospe J.  et al.  Translating evidence-based falls prevention into clinical practice in nursing facilities.  J Am Geriatr Soc. 2006;54(9):1414-1418
PubMed   |  Link to Article
Baier RR, Gifford DR, Patry G.  et al.  Ameliorating pain in nursing homes.  J Am Geriatr Soc. 2004;52(12):1988-1995
PubMed   |  Link to Article
Horbar JD, Carpenter JH, Buzas J.  et al.  Collaborative quality improvement to promote evidence based surfactant for preterm infants.  BMJ. 2004;329(7473):1004
PubMed   |  Link to Article
Stoeckle-Roberts S, Reeves MJ, Jacobs BS.  et al.  Closing gaps between evidence-based stroke care guidelines and practices with a collaborative quality improvement project.  Jt Comm J Qual Patient Saf. 2006;32(9):517-527
Institute for Healthcare Improvement.  The Breakthrough Series: IHI's Collaborative Model for Achieving Breakthrough Improvement. Cambridge, MA: Institute for Healthcare Improvement; 2003:1-20
Marinopoulos SS, Dorman T, Ratanawongsa N.  et al.  Effectiveness of Continuing Medical Education. Rockville, MD: Agency for Healthcare Research and Quality; 2007:28
Mittman BS. Creating the evidence base for quality improvement collaboratives.  Ann Intern Med. 2004;140(11):897-901
PubMed   |  Link to Article
Davis DA, Mazmanian PE, Fordis M, Van Harrison R, Thorpe KE, Perrier L. Accuracy of physician self-assessment compared with observed measures of competence.  JAMA. 2006;296(9):1094-1102
PubMed   |  Link to Article
Moore SM, Alemi F, Headrick LA.  et al.  Using learning cycles to build an interdisciplinary curriculum in CI for health professions students in Cleveland.  Jt Comm J Qual Improv. 1996;22(3):165-171
PubMed
Headrick LA, Moore SM, Alemi F.  et al.  Using PDSA (Plan-Do-Study-Act) to establish academic-community partnerships.  Qual Manag Health Care. 1998;6(2):12-20
PubMed
Ellrodt AG. Introduction of total quality management (TQM) into an internal medicine residency.  Acad Med. 1993;68(11):817-823
PubMed   |  Link to Article
Ladden MD, Bednash G, Stevens DP, Moore GT. Educating interprofessional learners for quality, safety and systems improvement.  J Interprof Care. 2006;20(5):497-505
PubMed   |  Link to Article
Weingart SN, Tess A, Driver J, Aronson MD, Sands K. Creating a quality improvement elective for medical house officers.  J Gen Intern Med. 2004;19(8):861-867
PubMed   |  Link to Article
Bravata DM, McDonald KM, Shojania KG, Sundaram V, Owens DK. Challenges in systematic reviews.  Ann Intern Med. 2005;142(12 pt 2):1056-1065
PubMed   |  Link to Article
Reed D, Price EG, Windish DM.  et al.  Challenges in systematic reviews of educational intervention studies.  Ann Intern Med. 2005;142(12 pt 2):1080-1089
PubMed   |  Link to Article
Laffel G, Blumenthal D. The case for using industrial quality management science in health care organizations.  JAMA. 1989;262(20):2869-2873
PubMed   |  Link to Article
Watson WE, Michaelsen LK, Sharp W. Member competence, group interaction, and group decision making: a longitudinal study.  J Appl Psychol. 1991;76(6):803-809
Link to Article
Varkey P, Natt N. The Objective Structured Clinical Examination as an educational tool in patient safety.  Jt Comm J Qual Patient Saf. 2007;33(1):48-53
PubMed
CME
Meets CME requirements for:
Browse CME for all U.S. States
Accreditation Information
The American Medical Association is accredited by the Accreditation Council for Continuing Medical Education to provide continuing medical education for physicians. The AMA designates this journal-based CME activity for a maximum of 1 AMA PRA Category 1 CreditTM per course. Physicians should claim only the credit commensurate with the extent of their participation in the activity. Physicians who complete the CME course and score at least 80% correct on the quiz are eligible for AMA PRA Category 1 CreditTM.
Note: You must get at least of the answers correct to pass this quiz.
You have not filled in all the answers to complete this quiz
The following questions were not answered:
Sorry, you have unsuccessfully completed this CME quiz with a score of
The following questions were not answered correctly:
Commitment to Change (optional):
Indicate what change(s) you will implement in your practice, if any, based on this CME course.
Your quiz results:
The filled radio buttons indicate your responses. The preferred responses are highlighted
For CME Course: A Proposed Model for Initial Assessment and Management of Acute Heart Failure Syndromes
Indicate what changes(s) you will implement in your practice, if any, based on this CME course.

Multimedia

Data Supplements
Supplemental Content

Some tools below are only available to our subscribers or users with an online account.

Web of Science® Times Cited: 108

Related Content

Customize your page view by dragging & repositioning the boxes below.

See Also...
Related Multimedia

Author in the Room

Articles Related By Topic
Related Collections
PubMed Articles
JAMAevidence.com

Users' Guides to the Medical Literature
What are the Results of the Study?

Users' Guides to the Medical Literature
What Are the Results?