Evaluating the Teaching of Evidence-Based Medicine

An increasing number of medical schools and residency programs are instituting curricula for teaching the principles and practice of evidence-based medicine (EBM). For example, 95% of US internal medicine residency programs have journal clubs¹ and 37% of US and Canadian internal medicine residencies have time dedicated for EBM.² Curricula based on EBM are increasingly popular in residency programs in other specialties, including family medicine, pediatrics, obstetrics/gynecology, and surgery.³ Despite the widespread teaching of EBM, however, most of what is known about the outcomes of evidence-based curricula relies on observational data. Although evaluation of the quality of research evidence is a core competency of EBM, the quantity and quality of the evidence for effectively teaching EBM are poor. Ironically, if one were to develop guidelines for how to teach EBM based on these results, they would be based on the lowest level of evidence.

There are several reasons why the quality of the evidence for teaching EBM is so weak. Many of these problems are related to the limitations in educational research in general. First, quantitative research methods may be inadequate to capture the complexity of an educational system. Second, students and residents change frequently, making it difficult to retain a consistent sample. Third, the time allotted for a given intervention may be brief in the context of the overall medical curriculum. Fourth, educational institutions may be hesitant to pay students as research participants or to allocate them to unproved educational interventions. Fifth, because most educational interventions are unique to specific institutions, assessment of their effectiveness is usually limited by small sample sizes. Furthermore, even if such interventions could be instituted across multiple institutions, the problems of standardization and cointervention would be particularly challenging. Sixth, perhaps because they are simplest to measure, the most frequently reported outcomes are subjective variables such as satisfaction or self-reported changes in attitudes or knowledge, rather than more important assessments of objectively measured clinical skills or improved patient outcomes. Finally, granting agencies do not give priority to educational investigations, making it difficult to undertake definitive multicenter studies.

Educators who have struggled to evaluate educational interventions will find these issues all too familiar. With the increasing prevalence of EBM teaching, however, high-quality evidence is more important than ever. Assessment of EBM teaching has also presented some unique problems. For instance, we originally defined evidence-based practice in terms of 4 basic competencies: (1) recognition of a patient problem and construction of a structured clinical question; (2) ability to efficiently and effectively search the medical literature to retrieve the best available evidence to answer the clinical question; (3) critical appraisal of the evidence; and (4) integration of the evidence with all aspects of individual patient decision making to determine the best clinical care for the patient.⁴ Although these 4 skills were the most commonly reported curricular objectives in 99 internal medicine residencies that teach EBM,² almost all the research on EBM education has focused exclusively on the third item: teaching critical appraisal skills.^{5 - 7} Examining this literature may yield useful insights into the difficulties of EBM educational research.

Since critical appraisal skills involve the ability to differentiate strong from weak research methods, one might expect that this research would be of relatively high quality. In fact, most of these studies are methodologically weak. Using broad criteria to identify any reports of a graduate (residency) EBM curricula, Green³ identified 18 reports published between 1980 and 1997. Of these, 72% used a traditional journal club format to teach critical appraisal skills. Only 7 of the 18 studies evaluated the effectiveness of their intervention. Five of these 7 studies compared intervention with control (only 1 with randomized design); only 2 of 7 studies used any blinding. Of these 5 controlled studies, 2 used a validated outcome measure to evaluate critical appraisal skills. Measurement of behavioral change relied on self-report in all 5 studies, and none examined patient outcomes. Most reports did not evaluate their intervention in terms of making inferences about optimal teaching methods. Of the 7 studies that had a quantitative experimental design, 2 failed the most basic design criterion of including a control group to measure the relative effectiveness of the intervention.³

When the Cochrane Collaboration attempted a systematic review of educational interventions to teach critical appraisal to health care professionals, only 1 article met the methodologic criteria of including a control group and measuring any process of care outcome, patient health care outcome, or learner knowledge/awareness using a valid instrument.⁵ Despite the lack of rigorous randomized trials, they are no less necessary to educational research than they are to clinical studies. As with clinical research, rigorous methods would help reduce the impact of other influences that could affect the measured outcomes (such as other teaching sessions, interactions with patients or house staff or faculty, self-directed learning, etc). Such studies should ideally include a pretest of knowledge to measure change from baseline. Similarly, if evaluators are not blinded to the intervention, they could systematically overestimate its effects.

There are 2 general shortcomings of the current literature. First, there is a lack of validated outcome measures. While it is easiest to measure changes in knowledge, only a few critical appraisal instruments have been validated.^{8 - 10} When researchers do not use validated instruments it is difficult to know what to make of the results. Second, the long-term goal of medical education is not merely to impart new knowledge, but rather to change learners' behaviors. In assessing the teaching of EBM, however, researchers continue to struggle to define the specific changes they hope to result from their interventions. Perhaps this is why the critical appraisal educational literature has relied most heavily on self-reports and learner satisfaction questionnaires.^{3 ,6 - 7} We acknowledge, however, that patient outcomes are difficult to measure and are affected by many other unrelated variables.

Despite logistical and methodological challenges, rigorous investigation of EBM teaching remains possible. First, EBM educators need not feel uniquely disadvantaged by a lack of evidence. Little more is known about the optimal ways of teaching other content areas such as primary care or cardiology, skills such as physical examination, or global constructs such professionalism. Despite this lack of high-quality evidence, there is universal agreement that such material is teachable. Similarly, the increase in the number of EBM practitioners suggests that evidence-based practice is a learnable skill. Thus, the question is not whether EBM can be taught, but how best to teach it. Finally, it is no more realistic to think one can teach evidence-based practice in a dozen 1-hour sessions than it is to believe one can teach the practice of cardiology in a similar period. There may be considerable wisdom, as yet unproved, in taking a broad, curricular approach to interventions.¹¹

Methodology must be considered with these perspectives in mind. Although we have focused on the quantitative critical appraisal research, this is simply a reflection of the current state of the literature. Well-designed studies using approaches other than quantitative methods are equally valuable contributions. In a recent commentary, Murray¹² suggested that since education is a complex, multifactorial process that occurs in an environment of numerous confounders, educational researchers could learn from health services research designs that also evaluate complex interventions.¹³ Qualitative research methods are an underused and valuable research tool for evaluating EBM teaching and could help identify learners' needs, effective components of interventions, and barriers to behavioral change.¹⁴

While observational studies may be useful first steps in the research process, definitive insights into optimal educational approaches to EBM must combine both qualitative and quantitative approaches. Quantitative studies are attainable and should include control groups, randomized if possible, and use validated measures of important outcomes. New behavioral outcome measures must be developed and may include tools that track the evidence used by learners in their patient or attending encounters, instruments to audit learners' computer literature searches, or the development of objective structured clinical examinations to assess the quality of learners' structured clinical questions.¹⁵

Randomization of learners is possible^{8 ,16} as is blinding of assessors.^{8 ,17} Investigators have assessed EBM skills other than critical appraisal, including building structured clinical questions and search strategies.¹⁶ There are several validated measurement tools,^{8 - 10 ,18} and several new measures of behavioral change await validation. For example, a recent study objectively measured the behavioral outcomes corresponding to the quality and quantity of literature searches.¹⁶ A preliminary study reported assessing audiotapes of resident and attending encounters to determine the frequency with which residents incorporate evidence in their patient care.¹⁹

Collaborative efforts are under way among numerous institutions to evaluate EBM teaching interventions. While multi-institutional research has been rare until recently, it could provide a giant step forward for research into the optimal ways of teaching EBM.¹⁵ Multicenter trials could address methodological issues such as reducing contamination between interventions (by randomizing at an institutional, hospital, or ward level) and achieving adequate sample size. Educators and educational researchers should continue to develop rigorous studies of medical education in general, and of the teaching of EBM in particular. Research into teaching of EBM must not be immune from the standards that EBM educators seek to convey to their students.