Impact of Formal Continuing Medical Education:  Do Conferences, Workshops, Rounds, and Other Traditional Continuing Education Activities Change Physician Behavior or Health Care Outcomes? FREE

Two apparently conflicting pieces of evidence exist about physicians' continuing medical education (CME). Physicians report spending, on average (and among other activities), 50 hours per year in CME activities,^1- 3 ostensibly geared toward improving their performance and/or optimizing the outcomes of their patients. In addition, producing and accrediting formal, planned CME events and activities are a large enterprise, especially in the United States,^4- 5 intended to bring physicians up-to-date with rapidly expanding medical information. Patterned after undergraduate medical education consisting of lectures, audio visual presentations, and printed materials, CME activities appear underpinned by a belief that gains in knowledge lead physicians to improve how they practice and thus improve patient outcomes. Despite this belief and the level of participation in and resources for CME, many studies have demonstrated a lack of effect on physicians' performance of current practice guidelines^6- 7 or sizable gaps between real and ideal performance.^8- 9 In addition, a relatively weak effect of formal, planned CME on physician performance has been demonstrated in 2 of our previous reviews.^10- 11

Prompted by such evidence, this study (one of a series of reviews of the educational literature and its effect on practicing health professionals^10,12- 14) focuses on the effect of formal CME. This form of continuing education is highly variable, ranging from passive, didactic, large-group presentations to highly interactive learning methods, such as workshops, small groups, and individualized training sessions. Examples of such educational activities include rounds, educational meetings, conferences, refresher courses, programs, seminars, lectures, workshops, and symposia. Given the wide variability of formal CME formats, and the weight given by adult education theorists to interactive learning,^15- 18 our objective in this review was to answer these questions: Overall, is formal CME effective? Under what conditions is formal CME effective? What is particularly effective within formal CME in changing physician performance or health care outcomes?

Building on previous work and methods, we searched the Research and Development Resource Base in CME,¹⁹ a bibliographic database of continuing health professional education literature (Continuing Education, Faculty of Medicine, University of Toronto, Toronto, Ontario). Regular searches of MEDLINE, CINAHL, ERIC, EMBASE, and PsycInfo form the repository of all references cited in this and previous reviews. To update our previous MEDLINE search, 2 authors separately searched from January 1993 (the concluding date of our last major review-focused search¹⁰) to January 31, 1999. We combined the medical subject terms randomized controlled trials and random allocation with randomized controlled trial as both a publication type and text words. We then added to the strategy medical subject terms such as education, medical, continuing; education, continuing; and education, as well as variations of the following text words: lectures, rounds, seminars, meetings, symposia, conferences, courses, workshops, and small groups. We also searched the Specialised Register of the Cochrane Effective Practice and Organisation of Care Group.²⁰ The reference lists and abstracts of all articles that met initial criteria were reviewed independently by 3 authors (D.D., M.A.T.O., A.T.-V.).

We included only those studies that met the following criteria: randomized controlled trials of formal educational interventions (such as conferences, rounds, meetings, symposia, and individualized training sessions) using only educational activities meant to be persuasive rather than those that coerced or provided incentives to the learner-participant; objective determination of health professional performance in the workplace and/or determinations of health care outcomes; and more than 50% of the participants were practicing physicians. Health care outcomes included patient behavior outcomes, such as adherence to medication or smoking cessation rates. Although we recognize that a myriad of complex factors influence patient behavior, we included these outcomes if reported in the primary studies. For purposes of comparison, we included only those formal CME activities that were didactic and/or may have used interactive educational techniques. We excluded interventions that deployed postcourse reminders, audit, and feedback or other measures to change physician performance and studies that included more than 50% nonphysician health professionals or residents, given the marked difference in these populations' regulations, work environments, and expectations.

We developed and applied the following categorization of educational interventions. Didactic sessions were defined as predominantly lectures or presentations with minimal audience interaction or discussion. Interactive sessions included those using techniques to enhance physician participation, such as role-play, discussion groups, hands-on training, problem solving, or case solving. These interventions were termed mixed if they specified the use of both interactive and didactic methods. We classified the interventions as single or in a series if content was delivered on more than 1 occasion.

We also noted, where possible, the size of the groups educated, the presence of needs assessment techniques, and the inclusion of enabling elements within the session, such as those that would assist the physician-learner to make changes in his/her practice environment (eg, patient education materials, flow charts, reminders, and protocols). Further, we classified the outcomes as positive if 1 or more primary outcome measures related to physician performance or patient health care demonstrated a statistically significant change and negative if no such change occurred. Finally, 2 of 3 reviewers (M.A.T.O., N.F., D.D.) independently applied all inclusion and exclusion criteria using specific standards for each study. Once extracted, data were subjected to meta-analytic techniques.

Where available, data were reported and a standardized effect size was estimated for each study group. The standardized effect size is the difference in means divided by the square root of the pooled-group variances. Transforming differences in means to standardized scores enables the comparison of effects in different outcomes on a common dimensionless scale. We used the full random effects approach described by Smith et al,²¹ which takes into account uncertainty in the distribution of observed effects and works well with sparse data. This approach also enabled us to estimate the extent to which the presence of some interactive element in a teaching intervention predicted an improved outcome.

Fourteen studies met our inclusion criteria^22- 35 (Table 1). These were derived from a larger pool of studies retrieved from the Cochrane Effective Practice and Organisation of Care³⁶ module (which yielded 39 studies, of which 13 met our inclusive criteria) and MEDLINE (which yielded 25 potentially relevant studies, of which 1 met our criteria).

All trials studied the effect of formal, planned CME on American,^{23- 25,28- 34} Canadian,^26- 27,35 or French²² physicians. Of these, 1 study reported the effect of formal CME interventions on internists,³¹ 5 on family physicians/general practitioners,^{22,26- 28,35} and 2 on pediatricians.^24,30 Six studies targeted outcomes from a mixture of these physicians.^{23,25,29,32- 33,35} Clinical dimensions of care addressed by the studies included prevention and screening,^{22- 23,25,27} disease management,^24,26,34 counseling or communication skills,^{29- 31,33,35} smoking cessation,²⁸ and, in one instance, manual skills.³²

Using previously published criteria for assessing the quality of randomized controlled trials,³⁶ we determined that, of the 14 trials reviewed, only one²³ described adequate concealment of allocation to randomization indicated in the published report and 10 trials documented adequate follow-up. Outcomes were assessed blindly in only 7 studies.

Within the 14 studies, 24 separate interventions were tested; several studies used more than 1 intervention (Table 2). Of these, we report the outcomes of only 17 interventions; 5 did not conform to our definition of formal CME, 1 intervention itself was didactic and was used as a usual care control,²⁶ and, in a component of 1 trial, physicians were not randomized to a longitudinal formal CME intervention.²⁹ Of these 17 comparisons, the effect on physician performance was measured in all instances and on both performance and health care outcomes in 4 studies. The majority of these interventions were positive: 9 of 17 had an impact on 1 or more measures related to physician performance and 3 of 4 were related to health care outcomes.

Three interventions were found that used didactic measures to carry their educational message. One study focused on screening for cholesterol,²³ 1 on family practice topics,²⁶ and a third study²² attempted to improve screening techniques for breast and cervical cancer. Heale et al²⁶ provided family practice topics by mostly large-group, didactic, case-based presentations, comparing this method with a didactic, traditional, topic-oriented control. Overall, none of the 4 interventions altered physician performance.

We found 6 interventions that described only interactive techniques.^24- 27,32 Of these, 2 were single interventions^25- 26 and 4 used sequenced sessions.^24,27,32 All 6 interventions measured the impact on physician performance, and all but 2 demonstrated a significant impact. These 2 were Heale et al,²⁶ using small group discussions of family practice topics, and Dietrich et al,²⁵ who demonstrated a change in only 1 of 10 outcome measures following an interactive CME session. Clark et al,²⁴ for example, studied the effect of two 2½-hour sessions in pediatric care that used interactive techniques, a video, and case studies over a 2- to 3-week period. Clark et al, conducted parent interviews that demonstrated a positive effect on both physician performance and on child health.

Seven interventions were described as being mixed (ie, including both didactic and interactive elements).^{28- 31,33- 35} All 7 measured their impact on physician performance; of these, 5 were positive.^{25,28,30,33,35} Three measured the impact on health care outcomes,^28,30,33 of which 2 demonstrated positive changes.^30,33 Roter et al,³³ for example, used two 4-hour sessions that included didactic elements, roundtable discussions, and interactive presentations to improve physician communication skills. Positive changes were demonstrated by audiotaped analysis of patient interviews.

Seven interventions were single in nature, held in 1 period, and ranged from 2³⁴ to 6 hours.²⁶ All 7 measured their impact on physician performance and 2 were deemed positive.^25,34 In contrast, there were 8 interventions that used 2 sessions in a series^{24,28,30- 33,35} and 1 that offered a series of formal CME interventions, including small groups and teleconferences²⁷ in 2 separate topic-based interventions. These ranged from 2 hours³⁵ to a total of 48 hours.²⁷ Of these 10 interventions, seven^{24,27- 28,30,32- 33,35} demonstrated positive outcomes on physician performance; 3 of 4 demonstrated a similar impact on health care outcomes.^24,30,33

Group size varied in these studies. One trial studied the effect of individualized training sessions,³² while 3 used small groups of fewer than 10 individuals.^26,29,31 Moderate-sized groups, from 10 to 19 participants, were reported by 6 authors^{23- 25,27,30,35} and 3 interventions^22,26,34 used groups of 20 or more. No relationship between group size and positive outcomes was noted. Four studies^22,24,32,34 reported the development of the intervention following a needs assessment survey. One study²⁷ used a 6-step process to identify and evaluate the learning needs of participating physicians. Of these 5 studies, only one²² generated no change in physician performance or health care outcomes. Several studies used enabling methods, such as patient education materials^23,28,31,35 delivered in the context of the formal CME intervention; of these, three^28,31,35 demonstrated change in physician performance.

Finally, length of time to outcome assessment was reviewed in each of the studies (Table 1), ranging from 1 month²⁹ to 24 months³¹ after the intervention. Those studies that effected a positive change in physician performance varied in the length of time to outcome from 1.5³⁵ to 22 months,²⁴ while those that effected no such change ranged from 1 ²⁹ to 24 months.³¹ Of the 5 studies that analyzed patient outcomes, those that generated negative results assessed outcomes at 12²⁸ and 18 months.²³ In contrast, the 3 studies that effected positive changes included those by Maiman et al³⁰ and Roter et al,³³ each at 6 months, and Clark et al²⁴ at 22 months after intervention.

Only 7 trials provided data for quantitative pooling.^{22- 23,25,28- 29,34- 35} Overall, the pooled, standardized weighted mean difference from these studies was 0.34 (95% confidence interval [CI], −0.22 to 0.97), indicating a nonsignificant overall benefit from formal CME. However, when a random effects factor was added to the model, describing the effects of sessions that included an interactive element, this was associated with a significant positive effect (standardized weighted mean difference, 0.67 [95% CI, 0.01-1.45]).

The use of traditional CME activities such as lectures has been widely criticized.³⁷ This criticism appears justified because didactic interventions analyzed in this review failed to achieve success in changing performance or health care outcomes. While such interventions may change other elements of competence, such as knowledge, skills, or attitudes, or may act as predisposing elements to change, didactic lectures by themselves do not play a significant role in immediately changing physician performance or improving patient care.

In contrast, studies that used interactive techniques such as case discussion, role-play, or hands-on practice sessions were generally more effective changing those outcomes documented in this review. Sessions that were sequenced also appeared to have more impact. Both of these findings match closely those principles promoted by adult educators,^17,38 who describe successful adult education as learner-centered, active rather than passive, relevant to the learner's needs, engaging, and reinforcing—characteristics of CME interventions more frequently found in the interactive rather than the passive educational setting. Further, the learn-work-learn opportunities afforded by sequenced sessions, in which education may be translated into practice and reinforced (or discussed) at a further session, may explain the success of sequenced interventions. Communication theory³⁹ suggests that communication sustained over time may enable ideas to converge across gaps such as those that exist between CME teachers and learner-participants.

Other factors within the purview of the CME provider may also affect the impact of the formal CME intervention. The addition of methods we have termed enabling (eg, patient education materials), which may facilitate adapting to changes in the practice site, appear to be effective in the few trials in this review. These enabling agents conform in general terms to our previous conclusions¹⁴ about the success of multiple interventions, to the broader understanding of the complexity of the environment in which the change is to occur,⁴⁰ and to the promotion of health literature.⁴¹

Other variables appear effective but fall outside the traditional domain of CME providers. The first cluster of these may be termed external, relating to the practice environment, such as the use of after-course office management systems in which nurse-facilitators provide onsite, practice-based suggestions. Such variables also include the use of objective needs assessment methods, which appear to be precursors of effective CME interventions.¹⁰ Both these variables speak to the need for collaboration between CME providers and the practice sector and their data sources, exemplifying, from an educational perspective, the new paradigm for CME,⁴² or from a health service perspective, Berwick's model of quality improvement.⁴³ Such collaboration would appear to provide an environment in which clinical changes could be sustained over time.

The second cluster relates to the internal or intrapersonal aspects of the physician-participant. In this context, it is important to note that what was measured in this review was performance change, not learning. An evolving body of knowledge suggests that physicians attend formal CME events with varying levels of motivation to change, and that the level of their commitment to change may supersede both the immediate clinical value of the information and the method by which it was delivered, as predictors of change in performance.^44- 45 Additionally, the interaction between members of groups in some of these studies may influence individuals' learning and change,⁴⁶ perhaps by producing a level of cognitive dissonance between what peers know and do compared with the learner.⁴⁷ Finally, physicians appear to develop their own learning priorities based on external and internal forces: the CME course or conference may be just one of many such forces.⁴⁸

We offer several cautions about the interpretation of these results. First, as in all such reviews, publication bias may generate a biased sample of studies, increasing the likelihood of finding a false-positive result. This bias may also include the selective publication of data within trials by describing results as nonsignificant (P>.05).⁴⁹ Second, the process of study selection, data extraction, and estimation of trial quality, although performed by 3 independent reviewers, may be flawed. For example, our descriptions of interactive or didactic elements reflect the reporting authors' description of the degree of interaction, not necessarily the learning process. Third, the limited number of randomized controlled trials and settings may undermine the generalizability of these findings. However, the fact that this small number of studies identified a significant result may imply a large and fairly consistent effect from interactive educational methods. This latter point is especially important, given that we used a conservative method for random effects meta-analysis, which takes into account uncertainty in the distribution of the effects of studies.²¹ Fifth, these trials were, for the most part, all performed with primary care physicians, who permitted an examination of their performance data or the health outcomes of their patients, who were volunteers, and who (in some trials) engaged in sequenced CME activities; as such, they may not represent the target population of most CME activities. That is, these studies may be viewed more as efficacy studies conducted under better than normal conditions than as effectiveness studies of everyday CME. Finally, the degree to which this review may be termed a meta-analysis and the extent to which pooled effect sizes may be reported here depends to a large extent on the comparability and likeness of these CME interventions, an issue that continues to be debated.

As in previous reviews, we noted the relatively narrow range of clinical areas studied by these trials. While screening, smoking cessation, and counseling or communication skills are important clinical topic areas and key ingredients of primary care, many issues that involve the complex management of patients in surgery, internal medicine, psychiatry, and other areas of specialty or subspecialty interest were not covered in the studies located. It appears to us that the setting of both the learning experience and the practicing physician, particularly in an era of increasing managed care, warrant careful study. An area ripe for investigation appears to be the changing demographics of the physician population and gender mix, the increasing numbers of graduates from problem-based schools and primary care training, and the inclination toward and skills required for self-directed lifelong learning.³⁸

We provided several comments about the methods and quality of trials of CME interventions. When investigators pursue controlled trial methods, we recommend that they, along with journal editors and referees, adhere more closely to the CONSORT⁵⁰ recommendations for reporting randomized controlled trials. In particular, it is important that investigators choose a suitable design⁵¹ and an appropriate unit of analysis.^52- 53 Although these trials produce quantifiable end points, they cannot fully explain why change does or does not occur as a result of CME participation.⁵⁴ Qualitative methods such as those used by Fox et al⁴⁸ need to be applied, and tools such as those evolving from the commitment to change model^44,55 used.

While these studies evaluated the impact of the CME intervention on physician performance or health care outcomes, we recognized that these measures are more distal to the intervention compared with the more proximal and easier-to-measure components of competence: knowledge, skills, and attitudes. Acknowledging the diminished impact of CME along this continuum (competence to performance to health care outcomes), we urge further research into those factors that accelerate or impede translation from one domain to another.

The ultimate effect of formal CME interventions on the practice of physicians and the health of their patients (as in the case of any intervention) must be understood in the context of the methods by which the CME is delivered, including but not limited to the nature of the interaction and the quality of the enabling resources where available, the environment in which the translated competence is played out, and in the complex intrapersonal, interpersonal, and professional educational variables that affect the physician-learner. Despite this complexity and the cautions one should bear in mind when interpreting these trials, we conclude that where performance change is the immediate goal of a CME activity, the exclusively didactic CME modality has little or no role to play. Knowledge is clearly necessary but not in and of itself sufficient to bring about change in physician behavior and patient outcomes. Such didactic interventions should—as they do in the Canadian Maintenance of Competence Program system of the Royal College of Physicians and Surgeons of Canada⁵⁶—receive less credit than do more effective methods and perhaps no credit. In contrast, variables over which the CME provider has control and appear to have a positive effect are the degree of active learning opportunities, learning delivered in a longitudinal or sequenced manner, and the provision of enabling methods to facilitate implementation in the practice setting.

While numerous questions remain regarding formal CME, including group size, the role of the learning and practice environment, the clinical dimensions of care, the assessment of learner needs, and barriers to change, 1 large question remains. In the face of longstanding knowledge about adult, self-directed learning and the general disinclination to believe that didactic CME works, now coupled with the findings of this review, why would the medical profession persist in delivering such a product and accrediting its consumption? The reasons for the persistence of didactic CME include (but are definitely not limited to) the ease of designing and providing such activities, the substantial pharmaceutical sponsorship that promotes the transfer of information about new medications, and the dependence on traditional undergraduate models of education that are easy-to-mount and revenue generating.

Changing this delivery system carries serious implications for several groups of stakeholders that want to design and deliver effective CME. First, medical licensing boards and others with a genuine interest in assuring the public of physician competence must rethink the value of the CME credit system, including the American Medical Association's Physician Recognition Award⁵⁷ category I credit, the most visible US CME currency exchanged for the privilege of practicing medicine. Second, medical schools, specialty societies, and other providers of CME must reconsider the value of the credit they provide, as well as the type and duration of learning activities they produce. Third, the Accreditation Council for CME in the United States and other organizations intending to ensure the quality of CME must evaluate the services they provide to a large, complex, and expensive CME enterprise that values the production of single-session, teacher-centered activities over learner achievement. Finally, physicians must reflect on what they perceive as the CME experience itself and weigh the costs and lost learning opportunities of attendance at ineffective didactic sessions against participating in interactive, challenging, and sequenced activities that have increased potential for positively affecting their performance and the health of the patients they serve—the most important outcome of all.