0
We're unable to sign you in at this time. Please try again in a few minutes.
Retry
We were able to sign you in, but your subscription(s) could not be found. Please try again in a few minutes.
Retry
There may be a problem with your account. Please contact the AMA Service Center to resolve this issue.
Contact the AMA Service Center:
Telephone: 1 (800) 262-2350 or 1 (312) 670-7827  *   Email: subscriptions@jamanetwork.com
Error Message ......
Original Contribution |

Effectiveness of a Clinically Integrated e-Learning Course in Evidence-Based Medicine for Reproductive Health Training:  A Randomized Trial FREE

Regina Kulier, MD; Ahmet Metin Gülmezoglu, MD, PhD; Javier Zamora, PhD; M. Nieves Plana, MD; Guillermo Carroli, MD; Jose G. Cecatti, MD, PhD; Maria J. Germar, MD; Lumbiganon Pisake, MD, MS; Sunneeta Mittal, MD, FRCOG; Robert Pattinson, MD, FRCOG; Jean-Jose Wolomby-Molondo, MD; Anne-Marie Bergh, PhD; Win May, MD, PhD; Joao Paulo Souza, MD, PhD; Shawn Koppenhoefer, PhD; Khalid S. Khan, MBBS, MSc
[+] Author Affiliations

Author Affiliations: Department of Research Policy and Cooperation (Dr Kulier) and UNDP/UNFPA/WHO/World Bank Special Programme of Research, Development and Training in Human Reproduction (HRP) (Drs Gülmezoglu and Souza), World Health Organization (WHO), Geneva, Switzerland; Clinical Biostatistics Unit, Hospital Ramón y Cajal (IRYCIS) and CIBER Epidemiologia y Salud Publica (CIBERESP), Spain (Drs Zamora and Plana); Centro Rosarino de Estudios Perinatales (CREP), Rosario, Argentina (Dr Carroli); Department of Obstetrics and Gynecology, School of Medical Sciences, University of Campinas, Campinas, Brazil (Dr Cecatti); Department of Obstetrics and Gynecology, University of the Philippines College of Medicine, Philippine General Hospital, Manila (Dr Germar); Department of Obstetrics and Gynecology, Faculty of Medicine, Khon Kaen University, Khon Kaen, Thailand (Dr Pisake); Department of Obstetrics and Gynecology, All India Institute of Medical Sciences, New Delhi (Dr Mittal); Medical Research Council Maternal and Infant Health Care Research Unit, University of Pretoria, Pretoria, South Africa (Drs Pattinson and Bergh); Departement de Gynecologie et Obstetrique, Cliniques Universitaires de Kinshasa, Democratic Republic of the Congo (Dr Wolomby-Molondo); Department of Medical Education, Keck School of Medicine of the University of Southern California, Los Angeles (Dr May); Ecole Polytechnique Federale de Lausanne, Lausanne, Switzerland (Dr Koppenhoefer); and Women's Health Research Unit, Barts and The London School of Medicine and Dentistry, Queen Mary University of London, London, United Kingdom (Dr Khan).


JAMA. 2012;308(21):2218-2225. doi:10.1001/jama.2012.33640.
Text Size: A A A
Published online

Context For evidence-based practice to embed culturally in the workplace, teaching of evidence-based medicine (EBM) should be clinically integrated. In low-middle–income countries (LMICs) there is a scarcity of EBM-trained clinical tutors, lack of protected time for teaching EBM, and poor access to relevant databases in languages other than English.

Objective To evaluate the effects of a clinically integrated e-learning EBM course incorporating the World Health Organization (WHO) Reproductive Health Library (RHL) on knowledge, skills, and educational environment compared with traditional EBM teaching.

Design, Setting, and Participants International cluster randomized trial conducted between April 2009 and November 2010 among postgraduate trainees in obstetrics-gynecology in 7 LMICs (Argentina, Brazil, Democratic Republic of the Congo, India, Philippines, South Africa, Thailand). Each training unit was randomized to an experimental clinically integrated course consisting of e-modules using the RHL for learning activities and trainee assessments (31 clusters, 123 participants) or to a control self-directed EBM course incorporating the RHL (29 clusters, 81 participants). A facilitator with EBM teaching experience was available at all teaching units. Courses were administered for 8 weeks, with assessments at baseline and 4 weeks after course completion. The study was completed in 24 experimental clusters (98 participants) and 22 control clusters (68 participants).

Main Outcome Measures Primary outcomes were change in EBM knowledge (score range, 0-62) and skills (score range, 0-14). Secondary outcome was educational environment (5-point Likert scale anchored between 1 [strongly agree] and 5 [strongly disagree]).

Results At baseline, the study groups were similar in age, year of training, and EBM-related attitudes and knowledge. After the trial, the experimental group had higher mean scores in knowledge (38.1 [95% CI, 36.7 to 39.4] in the control group vs 43.1 [95% CI, 42.0 to 44.1] in the experimental group; adjusted difference, 4.9 [95% CI, 2.9 to 6.8]; P < .001) and skills (8.3 [95% CI, 7.9 to 8.7] vs 9.1 [95% CI, 8.7 to 9.4]; adjusted difference, 0.7 [95% CI, 0.1 to 1.3]; P = .02). Although there was no difference in improvement for the overall score for educational environment (6.0 [95% CI, −0.1 to 12.0] vs 13.6 [95% CI, 8.0 to 19.2]; adjusted difference, 9.6 [95% CI, −6.8 to 26.1]; P = .25), there was an associated mean improvement in the domains of general relationships and support (−0.5 [95% CI, −1.5 to 0.4] vs 0.3 [95% CI, −0.6 to 1.1]; adjusted difference, 2.3 [95% CI, 0.2 to 4.3]; P = .03) and EBM application opportunities (0.5 [95% CI, −0.7 to 1.8] vs 2.9 [95%, CI, 1.8 to 4.1]; adjusted difference, 3.3 [95% CI, 0.1 to 6.5]; P = .04).

Conclusion In a group of LMICs, a clinically integrated e-learning EBM curriculum in reproductive health compared with a self-directed EBM course resulted in higher knowledge and skill scores and improved educational environment.

Trial Registration anzctr.org.au Identifier: ACTRN12609000198224

Figures in this Article

Evidence-based medicine (EBM) encourages assimilation and implementation of new, valid, and relevant scientific knowledge by health care professionals as part of their daily clinical work.1 To be effective at achieving this, EBM curricula need to be clinically integrated.2 However, EBM teaching and assessment is often stand-alone,3 risking its isolation from clinical practice. Incorporating EBM into practice through teaching during clinical working hours, rather than off-site, is not straightforward. Learning EBM alongside service delivery poses many barriers, foremost of which include scarcity of trained, confident clinical teachers,4,5 lack of protected teaching time in practice, and poor access to clinically relevant evidence sources often in nonlocal languages.3,6 Although these barriers are universally relevant, they pose particular constraints in low-middle–income countries (LMICs).

E-learning can be developed to complement EBM teaching in the workplace.7 This can help augment confidence and reduce time and work-related pressures for clinical tutors. It is also a potentially useful strategy for harmonizing teaching across various languages and clinical settings.8 Video lectures have been shown to be as effective as equivalent face-to-face EBM sessions in improving knowledge and attitudes.911 However, teaching EBM is likely to be more successful in changing health care when its principal steps are integrated into daily clinical practice reflected in an improved educational culture. We developed a clinically integrated e-learning EBM course incorporating the World Health Organization (WHO) Reproductive Health Library (RHL) and evaluated the effects of the course on knowledge, skills, and educational environment in comparison with traditional EBM teaching in a group of LMICs.

We designed an international cluster randomized trial to compare a clinically integrated e-learning course (experimental intervention) with a self-directed course (control intervention) in EBM for reproductive health training incorporating the RHL. Seven LMICs (Argentina, Brazil, Democratic Republic of the Congo, India, Philippines, South Africa, and Thailand) participated in the trial between April 2009 and November 2010. The trial was approved by the WHO research ethics review committee (dated July 4, 2008) and by the local ethics review boards of each of the participating institutions. It was prospectively registered.

Using their local knowledge the lead country investigators approached heads of potentially eligible clinical obstetrics and gynecology training units. To be eligible, the unit had to be delivering EBM courses, defined as opportunities to learn about the techniques of EBM and its application in clinical practice, in the unit's residency program. In addition, units had to have at least 4 residents who had not yet been exposed to formal EBM training and who were available for the duration of the trial to undertake the course and the assessments. They also had to appoint a facilitator, a current clinical staff member knowledgeable about basic EBM principles, to facilitate on-the-job training throughout the trial period. Appropriate computer equipment and access to relevant databases was a precondition. We did not collect denominator data for the total numbers of units initially approached.

Informed consent was sought from the heads of training units before randomization to facilitate allocation concealment. There were 60 clusters (4 in Argentina, 19 in Brazil, 2 in Democratic Republic of the Congo, 13 in India, 8 in the Philippines, 7 in South Africa, and 7 in Thailand). The WHO statistical support unit randomized the clusters, stratified by country, by means of computer-generated random numbers into 2 groups: group 1 received the clinically integrated e-learning EBM teaching package (experimental intervention) (31 clusters, 123 participants); group 2 received a self-directed EBM teaching package (control intervention) (29 clusters, 81 participants). Randomization was stratified by country to control for differing resources. Consent was obtained from all individual trainees taking part in the trial after the clusters had been randomized. Facilitators and participants were informed that an educational evaluation was being conducted within their institutions but were not given any details of the trial to minimize the risk of biases arising from knowledge of group allocation.

Interventions and Data Collection

We developed a learner-centered, clinically integrated course designed to facilitate just-in-time learning through on-the-job-training in reproductive health. The course combined e-learning of EBM principles with a specialist library provided in various languages.7,8,12 The e-learning modules for experimental intervention consisted of 5 recorded video sessions in which basic EBM knowledge was delivered by a speaker. Questions arising in clinical practice prompted trainees to study these questions. The knowledge acquired through e-learning was blended with face-to-face teaching and learning with a clinical trainer. The clinical questions were addressed in formative assignments and signed off by trainers.

The RHL13 is a specialist database for sexual and reproductive health that contains systematic reviews on high-priority topics in areas such as maternal and perinatal health and family planning, with expert commentaries, educational videos, and other material mainly aimed at clinicians in LMICs. It is regularly updated and disseminated by WHO and is available in a range of languages. The e-learning EBM course particularly complemented the RHL in overcoming the barriers related to provision of clinically relevant evidence in local languages.6 Learning activities, assignments, and assessments were incorporated in clinical practice to help integrate EBM teaching into actual patient care and to improve workplace culture, as described in detail elsewhere (Box).7,14

Box. An Overview of the Clinically Integrated e-Learning Course in Evidence-Based Medicine for Reproductive Health Training

Aim

To familiarize course participants with evidence-based medicine (EBM) basics for incorporating evidence from systematic reviews included in the WHO RHL (CD-ROM/Internet version) into practice

Target Participants

Physicians in training (residents, registrars, postgraduate clinical trainees) in obstetrics and gynecology

Learning Objectives (Competencies)

On completion of the course, participants should be competently able to:

Generate structured questions arising from clinical problems in practice

Search relevant literature, identifying systematic reviews from the RHL wherever possible

Assess the quality (validity) of systematic reviews and primary research included within them

Assess the applicability of research findings for use in clinical practice

Identify possible barriers when implementing the output from the above activities into clinical practice and apply strategies to overcome these barriers

Learning Resources

A study guide including course outline, learning exercises and assignments using the WHO RHL, and link to video-based e-learning sessions structured in 5 modules:

Module 1: Asking clinical questions

Module 2: Searching the evidence

Module 3: Critical appraisal of systematic reviews

Module 4: Applicability of the evidence to the patient

Module 5: Implementation of evidence into practice

Learning and Teaching Methods

Participant-initiated (tutor-supported) learning in a clinical setting. Participants pursue independent study using the study guide and e-learning sessions. Tutors facilitate learning by:

Identifying EBM learning opportunities in a clinical setting

Directing participants to appropriate use of learning resources

Providing feedback on learning exercises and assignments

Student-Directed Learning

e-Learning (2-3 hours)

Contact Time

Assignments, feedback, and assessments (total, 20 hours)

Assessments

Feedback on assignments

Multiple-choice questions to test attitudes, knowledge, and skills

Questionnaire on educational environment (EBMEEM)

The course was developed and piloted in a collaborative project involving 8 international partners supported by the European Union's Leonardo da Vinci Vocational Training Action Programme (grant UK/05/B/F/PP-162_349) before being adapted and studied by WHO and its collaborating centers. See “Methods” for details. EBM, evidence-based medicine; EBMEEM, Evidence-Based Medicine Educational Environment Measure; RHL, Reproductive Health Library; WHO, World Health Organization.

The experimental and control interventions both had the same aim and learning objectives. The self-directed learning course (control intervention) consisted of a set of PowerPoint slide presentations made available online from the WHO RHL workshop-based course, with the same learning objectives and with similar content as the e-modules of the experimental intervention. This group also had access to a facilitator who could be consulted whenever necessary. The control course had been provided alongside the RHL for dissemination before the e-learning courses had been developed and had been in use as standard dissemination practice until this trial was launched.14 Following randomization the courses, data collection tools, and the RHL were provided in local languages: Spanish in Argentina, Portuguese (RHL in Spanish or English) in Brazil, French in the Democratic Republic of the Congo, and English in India, the Philippines, South Africa, and Thailand.

Trainees were enrolled in the trial for a period of 12 weeks. Each trainee, at the start, undertook an assessment of attitudes, knowledge, skills, and educational environment at baseline (precourse) to gain access to the training course materials for 8 weeks. Four weeks after the 8-week course (during which access to the course materials was not possible), trainees were asked to complete the postcourse assessment (same as at baseline). The experimental and control interventions were both delivered in addition to the local teaching programs.

Data were entered online using a database built to minimize the risk of errors with built-in range and consistency checks.

Outcomes

The outcome measures captured the effect of teaching on trainee learning in defined areas of EBM competencies including attitudes, knowledge, and skills.15 We measured attitudes toward EBM using a validated tool.16 EBM knowledge and skills were measured by adapting previously validated questionnaires,1618 taking items that mapped to the competencies taught and learned in this trial. Knowledge scores were obtained with multiple-choice questions (score range, 0-62). Gains in EBM skills were evaluated using an objective structured clinical examination administered after course completion only (score range, 0-14).

A new outcome measure for assessing educational environment with respect to evidence-based practice was developed and validated specifically for use in this trial. We considered this outcome measure important because EBM teaching should be about influencing climate in the workplace to facilitate evidence-based practice, not just about imparting knowledge and skills. There were 7 domains in the tool: knowledge and learning materials; learner support; general relationships and support; institutional focus on EBM; education, training, and supervision; EBM application opportunities; and affirmation of EBM environment. Responses to items within each domain were captured on a 5-point Likert scale anchored between 1 (strongly agree) and 5 (strongly disagree). The tool-validation study is provided as an eAppendix.

Sample Size

We used information regarding baseline EBM knowledge, possible gains, and intracluster (intraclass) correlation coefficient from our pilot work8,12,14 to inform the sample size and power calculation. With a 2-sided test and an α level of 5%, we needed 60 clusters (training units) to detect a knowledge gain of 10% (expected effect size) in the experimental intervention compared with the control intervention with 80% power, assuming a standard deviation of 15% and an intracluster (intraclass) correlation coefficient of 0.2, expecting a mean cluster size of about 4. The 10% knowledge gain represented a moderate improvement of 4.3 in multiple-choice questions score above the baseline score (43) observed in our previous trial,12 in which the standard deviation (9.3) was approximately 15% of the maximum possible score (62) on this outcome measurement.

Statistical Analysis

Data for the various outcome measures are presented as means with 95% CIs. Responses to the baseline and postcourse assessments were scored, and comparisons between the 2 intervention groups were made. For evaluating the effect on educational environment, the differences between baseline and postcourse responses were computed on a 5-point Likert scale from 1 (strongly agree) to 5 (strongly disagree). The responses for items within each domain and overall were added up.

Ordinal data were treated as continuous in statistical analysis after testing confirmed that assumptions for parametric data19 were met. Postintervention scores of the outcomes were compared between intervention groups using a 3-level generalized linear mixed model, with intervention group, time of assessment (baseline or postcourse), and intervention × time interaction as fixed effects and cluster and participants as random effects to account for the correlations (clustering within clinical training units and participants) of the data. Mixed-effects models allow the inclusion of all available data, consistent with the intention-to-treat approach. Such models account for correlation within clusters and within participants and are relatively robust to the presence of randomly missing data, rendering imputation routines for missing values unnecessary.

All comparisons were 2-sided and were considered statistically significant at P < .05. We determined the importance of size of educational effect observed by dividing the between-group difference by the within-cluster standard deviation and used Cohen guidelines20 for interpretation, with standardized effect size of 0.2 considered small, 0.5 considered medium, and 0.8 considered large. Stata version 11.0 (StataCorp, College Station, Texas) was used for all analyses.

Of the 60 clinical training units approached and initially randomized, 14 later dropped out (7 in each group) (Figure 1). Three clusters in each group declined participation after randomization. The rest either did not respond to further participation and training requests or none of their trainees completed the trial. Of the remaining clusters, 24 (123 trainees) were in the clinically integrated e-learning group and 22 (81 trainees) in the control group. There were 25 trainees that dropped out in the clinically integrated e-learning group and 13 in the control group. Among participating clusters with complete information on knowledge assessments among residents who dropped out, there was no baseline difference between completers (mean score, 38.4 [SD, 5.2); n = 166) and dropouts (mean score, 37.8 [SD, 4.6]; n = 38) (difference, 0.6 [95% CI, −1.2 to 2.3]; P = .52).

Place holder to copy figure label and caption
Figure 1. Study Flow
Graphic Jump Location

aRandomized clusters by country: Argentina, 2; Brazil, 9; Democratic Republic of the Congo, 1; India, 6; Philippines, 5; South Africa, 3; Thailand, 3.
bRandomized clusters by country: Argentina, 2; Brazil, 10; Democratic Republic of the Congo, 1; India, 7; Philippines, 3; South Africa, 4; Thailand, 4.
cAnalyzed clusters by country: Argentina, 1; Brazil, 8; Democratic Republic of the Congo, 0; India, 4; Philippines, 4; South Africa, 2; Thailand, 3.
dAnalyzed clusters by country: Argentina, 2; Brazil, 9; Democratic Republic of the Congo, 0; India, 5; Philippines, 3; South Africa, 3; Thailand, 2.

At baseline the 2 groups were similar in age, year of training, attitudes, and knowledge (Table 1 and Table 2). There was an excess of women in the intervention group vs the control group, but in analyses adjusted for the possible confounding effect of sex the results were unchanged. The initial ratings of attitude toward EBM were high.

Table Graphic Jump LocationTable 1. Baseline Characteristics of Participants
Table Graphic Jump LocationTable 2. Changes in Knowledge and Skills Evaluated Using Scores Obtained With Multiple-Choice Questions and the Objective Structured Clinical Examinationa

After the educational courses, there were significant gains with the experimental intervention in overall knowledge (postcourse mean, 38.1 [95% CI, 36.7 to 39.4]in the control group vs 43.1 [95% CI, 42.0 to 44.1] in the experimental group; adjusted mean difference, 4.9 [95% CI, 2.9 to 6.8]; P < .001) and skills measured with objective structured clinical examination scores (postcourse mean, 8.3 [95% CI, 7.9 to 8.7] in the control group vs 9.1 [95% CI, 8.7 to 9.4] in the experimental group; adjusted mean difference, 0.7 [95% CI, 0.1 to 1.3]; P = .02) (Figure 2 and Table 2). These gains represented large and moderate effect sizes for knowledge and skills (1.05 and 0.39, respectively). Analyses restricted to knowledge modules showed that there were statistically significant knowledge gains in terms of asking clinical questions (module 1), searching the evidence (module 2), and critical appraisal of systematic reviews (module 3) but not in applicability of evidence to the patient (module 4) and implementation of evidence into practice (module 5) (Table 2).

Place holder to copy figure label and caption
Figure 2. Changes in Knowledge Evaluated Using Multiple-Choice Question Scores
Graphic Jump Location

Horizontal bars in boxes indicate medians; boxes, interquartile ranges; whiskers, range excluding outliers.

There was no effect on educational environment overall (improvement, 6.0 [95% CI, −0.1 to 12.0] in the control group vs 13.6 [95% CI, 8.0 to 19.2] in the experimental group; adjusted mean difference, 9.6 [95% CI, −6.8 to 26.1]; P = .25). There was an effect on 2 domains of the environment. The general relationships and support domain improved (−0.5 [95% CI, −1.5 to 0.4] in the control group vs 0.3 [95% CI, −0.6 to 1.1] in the experimental group; adjusted mean difference, 2.3 [95% CI, 0.2 to 4.3]; P = .03), as did the domain regarding EBM application opportunities (0.5 [95% CI, −0.7 to 1.8] in the control group vs 2.9 [95% CI, 1.8 to 4.1] in the experimental group; adjusted mean difference, 3.3 [95% CI, 0.1 to 6.5]; P = .04) (Table 3). The effect sizes for these domains were 0.53 and 0.54, respectively. However, the domains of knowledge and learning materials; learner support; institutional focus on EBM; education, training, and supervision; and affirmation of EBM environment did not improve.

Table Graphic Jump LocationTable 3. Changes in Educational Environmenta

In LMICs, a specialty-specific clinically integrated e-learning EBM course incorporating an evidence library in the local language was more effective in improving knowledge, skills, and some aspects of the educational environment than a control course of similar aim and content. The size of educational effect was large for knowledge and moderate for skills and 2 domains of educational environment. To our knowledge, this is the first time such an effect has been shown in a randomized trial.

The study adhered to guidelines for reporting of cluster randomized controlled trials (RCTs).21 This design is particularly suitable for evaluation of educational interventions delivered to health professionals because it is designed to generate balanced groups at baseline and to avoid biases and imprecision attributable to contamination, which tends to deviate findings toward a null effect, reducing the power to detect significant differences.2224 On the other hand, because of the inherent similarity between participants within a teaching unit, randomization by clusters requires a large number of teaching units.

Evaluation of educational interventions in EBM may capture a range of outcomes, from rates of course completion to improvement in patient outcomes, taking into account the types of participants and interventions.15,25,26 We focused our study on demonstrating improvements in knowledge, skills, and educational environment. Training in EBM should reinforce the perception that it is of value to professional conduct.27 A favorable atmosphere in the workplace can facilitate changes in behavior and clinical outcomes following EBM training. There were few existing tools to assess educational environment in general2831 and none that captured EBM-related environments. We therefore developed a tool to measure this and demonstrated that changes in knowledge and skills were associated with improvements in some domains of educational environment. We did not assess whether these improvements changed patient care; this is a study limitation.

Cluster RCTs are not without limitations. Clusters are usually randomized all at once rather than one at a time, and entire clusters may drop out after randomization. In our study, there was loss of clusters attributable to technical difficulties, such as interrupted or limited Internet connection; irregular library or computer access; unwillingness to participate; and lack of protected time for the participants to take part because of service load, all of which have implications for generalizability of our findings. Because we had not performed a priori adjustments for anticipated loss of clusters and participants in our original sample-size estimation, study power was reduced and the possibility of type II error was increased. However, because we set the sample size assuming a moderate rather than large effect size such as was observed, the power was preserved at least in part. Moreover, in the face of significant results for the primary outcomes, loss of power is less critical.

In addition, the drop out of training units was not differentially excessive in the experimental intervention group vs the control group. After randomization, participants within the clusters could decline when approached for consent, which could result in a selection bias. However, the comparison of baseline knowledge scores between dropouts and completers showed no difference. Although the numbers of participants in comparison groups were dissimilar in our study, we had taken measures to prevent knowledge of group allocation among unit heads. The measured baseline characteristics of experimental and control groups were similar, or a difference (in the case of sex) did not alter the main conclusion in an adjusted analysis, reducing the likelihood of confounding. In real-time implementation, some of the limits that we enforced for completing the multiple-choice questions and the modules for research purposes will not be needed or may be implemented in a more flexible way.

An earlier educational intervention using the RHL to improve obstetric practices was not found effective in improving clinician behavior.32 In this trial we focused on end points more directly linked to the educational intervention. We found that e-learning made on-the-job teaching of EBM feasible and effective in improving self-reported workplace culture. In low-resource areas where EBM expertise is lacking, teacher training would need to be considered when planning wide dissemination of this electronic EBM-RHL package, which could be achieved through e-learning courses.33

Another limitation of our study is that we did not measure decay of knowledge. However, the e-learning modules viewed, reviewed, stopped, restarted, and completed during our study can be revisited any time at participants' convenience and at their own pace for a refresher whenever required after the course. Although this makes e-learning potentially a cost-effective alternative to face-to-face teaching, contextual factors may interfere with the effectiveness of this clinically integrated course, particularly in settings with limited resources.

One criticism of this study may be that the observed effects were the result of facilitation by tutors rather than the e-learning course. Our starting premise was that teaching EBM is more likely to be successful when it is culturally embedded and that this is best achieved by on-the-job training. Allocation was concealed, so randomization should have led to balance in tutor competence. The e-learning course complemented EBM teaching in the workplace, and our study assessed the value added by e-learning to EBM teaching offered by tutors. Because we selected units for the study that already provided EBM teaching, it is not surprising that trainees in both groups showed positive baseline attitudes toward EBM. We had taken measures to keep facilitators and participants unaware of their group allocation to minimize the risk of performance bias. The inherent awareness of and motivation toward EBM among participating units may have contributed to improvements in the control group. However, because the experimental intervention resulted in better performance than the control intervention, the conclusion that e-learning contributed to improved performance merits consideration.

We conclude that in a group of LMICs, an e-learning EBM curriculum in reproductive health, compared with a self-directed EBM course, resulted in higher knowledge and skill scores. The associated improvements in educational environment suggest that EBM principles that are learned may become culturally embedded in the workplace.

Corresponding Author: Khalid S. Khan, MBBS, MSc, Women's Health Research Unit, Barts and The London School of Medicine and Dentistry, Turner St, London E1 2AD, United Kingdom (k.s.khan@qmul.ac.uk).

Author Contributions: Drs Zamora and Plana had full access to all of the data in the study and take responsibility for the integrity of the data and the accuracy of the data analysis.

Study concept and design: Kulier, Gülmezoglu, Carroli, Cecatti, Germar, Pisake, Mittal, Pattinson, Wolomby-Molondo, Bergh, May, Koppenhoefer, Khan.

Acquisition of data: Kulier, Carroli, Cecatti, Germar, Pisake, Mittal, Pattinson, Wolomby-Molondo, Souza.

Analysis and interpretation of data: Kulier, Gülmezoglu, Zamora, Plana, Carroli, Pisake, Mittal, Bergh, Souza, Khan.

Drafting of the manuscript: Kulier, Gülmezoglu, Zamora, Plana, May, Khan.

Critical revision of the manuscript for important intellectual content: Kulier, Gülmezoglu, Zamora, Plana, Carroli, Cecatti, Germar, Pisake, Mittal, Pattinson, Wolomby-Molondo, Bergh, Souza, Koppenhoefer, Khan.

Statistical analysis: Kulier, Zamora, Plana, Khan.

Obtained funding: Gülmezoglu.

Administrative, technical, or material support: Kulier, Gülmezoglu, Carroli, Cecatti, Germar, Pisake, Pattinson, Bergh, May, Souza, Koppenhoefer.

Study supervision: Kulier, Gülmezoglu, Carroli, Cecatti, Germar, Mittal, Pattinson, Wolomby-Molondo.

Conflict of Interest Disclosures: All authors have completed and submitted the ICMJE Form for Disclosure of Potential Conflicts of Interest and none were reported.

Funding/Support: This trial was funded by the UNDP/UNFPA/WHO/World Bank Special Programme of Research, Development and Research Training in Human Reproduction, Department of Reproductive Health and Research, World Health Organization. The initial development and piloting of the e-learning course was funded by the European Union Leonardo da Vinci Vocational Training Action Programme (project grant UK/05/B/F/PP-162_349). Drs Zamora and Khan used funding from European Union made available to the EBM-CONNECT Collaboration through its Seventh Framework Programme, Marie Curie Actions, International Staff Exchange Scheme (proposal 101377; grant agreement 247613) for data analysis and drafting the manuscript. The EBM-CONNECT (Evidence-Based Medicine COllaboratioN: NEtwork for systematic reviews and guideline development researCh and disseminaTion) Collaboration (in alphabetical order by country): L. Mignini, Centro Rosarino de Estudios Perinatales, Argentina; P. von Dadelszen, L. Magee, D. Sawchuck, University of British Columbia, Canada; E. Gao, Shanghai Institute of Planned Parenthood Research, China; B. W. Mol, K. Oude Rengerink, Academic Medical Centre, the Netherlands; J. Zamora, Ramón y Cajal, Spain; C. Fox, J. Daniels, University of Birmingham; and K. S. Khan, S. Thangaratinam, C. Meads, Barts and the London School of Medicine, Queen Mary University of London, United Kingdom. The EBM-CONNECT Canadian Collaborators receive funding from the Canadian Institutes of Health Research.

Role of Sponsor: The funding sources had no role in the design and conduct of the study; the collection, management, analysis, or interpretation of the data; or the preparation, review, or approval of the manuscript.

Additional Resources: The educational environment tool is available from the authors (anne-marie.bergh@up.ac.za).

Additional Contributions: We would like to thank the EBM Unity project team for initially developing and piloting the e-learning course. We would also like to thank the facilitators in each country: Argentina: Yanina Sguassero (CREP); Hernán Garnica, (Sanatorio de la Mujer, Rosario, Argentina); Juan Manuel Nardin, (Maternidad Martin, Rosario, Argentina), Luciano Mignini, MD (Hospital Escuela Eva Peron, Rosario, Argentina); Marcelo Raffagnini, MD (Hospital Roque Saenz Peña, Rosario, Argentina). Brazil: Denis Jose Nascimento, Adriani Galao, Maria Laura Costa, Adriana Gomes Luz, Gustavo Lobato, Vera Terezinha Medeiros Borges, Ione Rodrigues Brum, Carlos Augusto Menezes, Everardo Macedo Guanabara, Joaquim Luiz Castro Moreira, Aurelio Ribeiro Costa, Ana Maria Feitosa Porto, Elias Melo Jr, Claudio Sergio Paiva, Maria Leticia Sperandeo Macedo, Ricardo Carvalho Cavalli, Marcia Maria Aquino. India: Sudha Salhan, Aruna Batra (Vardhman Medical College & Safdarjang Hospital, New Delhi); Neelam B. Vaid, Rachna Agarwal, Amita Suneja (Guru Teg Bahadur Hospital, Delhi); Swaraj Batra, Reva Tripathi, Deepti Goswami, (Maulan Azad Medical College, New Delhi); Vatsla Dadhwal; Sunesh Kumar (All India Institute of Medical Sciences, New Delhi); Vinita Das, Anjoo Agarwal, Amita Pandey (King Jorge Medical University, Lucknow); Veena Agarwal (Government Medical College, Gwalior). Philippines: Enrico Gil Oblepias (Ospital ng Maynila), Mario Bautista (Makati Medical Center), Christine Dobles Dizon (The Medical City), Joselito Santiago (Manila Doctors Hospital), Emmanuel Ganal (Jose Fabella Memorial Hospital), Carolyn Zalameda Castro (Philippine General Hospital), Christine Joy Garcia (Cardinal Santos Medical Center), Anna Lynn Alvarado Matignas (Jose R. Reyes Memorial Medical Center). South Africa: Sarah Jackson, Wilhelm Steyn, Sue Fawcus, Athol Kent, Louise Smith, Hopolang Maise. Thailand: Krasean Panyakhamlerd (King Chulalongkorn Memorial Hospital), Thitiporn Siriwachirachai (Khon Kaen Hospital), Kritsada Srithana (Pramongkutglao Hospital), Karicha Mairaing (Thammasart University Hospital), Suvanna Asavapiriyanont (Rajvithi Hospital), Chompilas Chongsomchai (Srinagarind Hospital, Khon Kaen University), Vitaya Titapant (Siriraj Hospital, Mahidol University).

Kulier R, Gee H, Khan KS. Five steps from evidence to effect: exercising clinical freedom to implement research findings.  BJOG. 2008;115(10):1197-1202
PubMed   |  Link to Article
Khan KS, Coomarasamy A. A hierarchy of effective teaching and learning to acquire competence in evidenced-based medicine.  BMC Med Educ. 2006;6:59
PubMed   |  Link to Article
Kunz R, Nagy E, Coppus SF,  et al.  How far did we get? how far to go? a European survey on postgraduate courses in evidence-based medicine.  J Eval Clin Pract. 2009;15(6):1196-1204
PubMed   |  Link to Article
Richardson WS. Teaching evidence-based practice on foot.  ACP J Club. 2005;143(2):A10-A12
PubMed
Hatala R, Keitz SA, Wilson MC, Guyatt G. Beyond journal clubs: moving toward an integrated evidence-based medicine curriculum.  J Gen Intern Med. 2006;21(5):538-541
PubMed   |  Link to Article
Oude Rengerink K, Thangaratinam S, Barnfield G,  et al.  How can we teach EBM in clinical practice? an analysis of barriers to implementation of on-the-job EBM teaching and learning.  Med Teach. 2011;33(3):e125-e130
PubMed   |  Link to Article
Coppus SF, Emparanza JI, Hadley J,  et al.  A clinically integrated curriculum in evidence-based medicine for just-in-time learning through on-the-job training: the EU-EBM project.  BMC Med Educ. 2007;7(1):46
PubMed   |  Link to Article
Kulier R, Hadley J, Weinbrenner S,  et al.  Harmonising evidence-based medicine teaching: a study of the outcomes of e-learning in five European countries.  BMC Med Educ. 2008;8:27
PubMed   |  Link to Article
Davis J, Chryssafidou E, Zamora J, Davies D, Khan K, Coomarasamy A. Computer-based teaching is as good as face to face lecture-based teaching of evidence-based medicine: a randomised controlled trial.  BMC Med Educ. 2007;7:23
PubMed   |  Link to Article
Davis J, Crabb S, Rogers E, Zamora J, Khan K. Computer-based teaching is as good as face to face lecture-based teaching of evidence-based medicine: a randomized controlled trial.  Med Teach. 2008;30(3):302-307
PubMed   |  Link to Article
Hadley J, Kulier R, Zamora J,  et al.  Effectiveness of an e-learning course in evidence-based medicine for foundation (internship) training.  J R Soc Med. 2010;103(7):288-294
PubMed   |  Link to Article
Kulier R, Coppus SF, Zamora J,  et al.  The effectiveness of a clinically integrated e-learning course in evidence-based medicine: a cluster randomised controlled trial.  BMC Med Educ. 2009;9:21
PubMed   |  Link to Article
World Health Organization (WHO).  RHL: The WHO Reproductive Health Library. WHO website. http://apps.who.int/rhl/en/index.html. Accessed October 29, 2012
Kulier R, Khan KS, Gülmezoglu AM,  et al.  A cluster randomized controlled trial to evaluate the effectiveness of the clinically integrated RHL evidence-based medicine course.  Reprod Health. 2010;7:8
PubMed   |  Link to Article
Straus SE, Green ML, Bell DS,  et al; Society of General Internal Medicine Evidence-Based Medicine Task Force.  Evaluating the teaching of evidence-based medicine: conceptual framework.  BMJ. 2004;329(7473):1029-1032
PubMed   |  Link to Article
Taylor R, Reeves B, Mears R,  et al.  Development and validation of a questionnaire to evaluate the effectiveness of evidence-based practice teaching.  Med Educ. 2001;35(6):544-547
PubMed   |  Link to Article
Ramos KD, Schafer S, Tracz SM. Validation of the Fresno test of competence in evidence-based medicine.  BMJ. 2003;326(7384):319-321
PubMed   |  Link to Article
Fritsche L, Greenhalgh T, Falck-Ytter Y, Neumayer HH, Kunz R. Do short courses in evidence-based medicine improve knowledge and skills? validation of Berlin questionnaire and before and after study of courses in evidence-based medicine.  BMJ. 2002;325(7376):1338-1341
PubMed   |  Link to Article
Pell G. Use and misuse of Likert scales.  Med Educ. 2005;39(9):970-971
PubMed   |  Link to Article
Cohen J. Statistical Power Analysis for the Behavioural Sciences. 2nd ed. New York, NY: Academic Press; 1977
Campbell MK, Elbourne DR, Altman DG.CONSORT Group.  CONSORT statement: extension to cluster randomised trials.  BMJ. 2004;328(7441):702-708
PubMed   |  Link to Article
Cook TD. Randomized experiments in educational policy research: a critical examination of the reasons the educational evaluation community has offered for not doing them.  Educ Eval Policy Anal. 2002;24(3):175-199Link to Article
Link to Article
Keogh-Brown MR, Bachmann MO, Shepstone L,  et al.  Contamination in trials of educational interventions.  Health Technol Assess. 2007;11(43):iii, ix-107
PubMed
Torgerson CJ. Educational research and randomised trials.  Med Educ. 2002;36(11):1002-1003
PubMed   |  Link to Article
Belfield C, Thomas H, Bullock A, Eynon R, Wall D. Measuring effectiveness for best evidence medical education: a discussion.  Med Teach. 2001;23(2):164-170
PubMed   |  Link to Article
Malick SM, Hadley J, Davis J, Khan KS. Is evidence-based medicine teaching and learning directed at improving practice?  J R Soc Med. 2010;103(6):231-238
PubMed   |  Link to Article
Hutchinson L. Educational environment.  BMJ. 2003;326(7393):810-812
PubMed   |  Link to Article
Holt MC, Roff S. Development and validation of the Anaesthetic Theatre Educational Environment Measure (ATEEM).  Med Teach. 2004;26(6):553-558
PubMed   |  Link to Article
Soemantri D, Herrera C, Riquelme A. Measuring the educational environment in health professions studies: a systematic review.  Med Teach. 2010;32(12):947-952
PubMed   |  Link to Article
Vieira JE. The Postgraduate Hospital Educational Environment Measure (PHEEM) questionnaire identifies quality of instruction as a key factor predicting academic achievement.  Clinics (Sao Paulo). 2008;63(6):741-746
PubMed   |  Link to Article
Roff S. The Dundee Ready Educational Environment Measure (DREEM)—a generic instrument for measuring students' perceptions of undergraduate health professions curricula.  Med Teach. 2005;27(4):322-325
PubMed   |  Link to Article
Gülmezoglu AM, Langer A, Piaggio G, Lumbiganon P, Villar J, Grimshaw J. Cluster randomised trial of an active, multifaceted educational intervention based on the WHO Reproductive Health Library to improve obstetric practices.  BJOG. 2007;114(1):16-23
PubMed   |  Link to Article
Weberschock T, Sorinola O, Thangaratinam S, Oude RK, Arvanitis TN, Khan KS. How to confidently teach EBM on foot: development and evaluation of a web-based e-learning course [published online August 4, 2012].  Evid Based MedLink to Article

Figures

Place holder to copy figure label and caption
Figure 1. Study Flow
Graphic Jump Location

aRandomized clusters by country: Argentina, 2; Brazil, 9; Democratic Republic of the Congo, 1; India, 6; Philippines, 5; South Africa, 3; Thailand, 3.
bRandomized clusters by country: Argentina, 2; Brazil, 10; Democratic Republic of the Congo, 1; India, 7; Philippines, 3; South Africa, 4; Thailand, 4.
cAnalyzed clusters by country: Argentina, 1; Brazil, 8; Democratic Republic of the Congo, 0; India, 4; Philippines, 4; South Africa, 2; Thailand, 3.
dAnalyzed clusters by country: Argentina, 2; Brazil, 9; Democratic Republic of the Congo, 0; India, 5; Philippines, 3; South Africa, 3; Thailand, 2.

Place holder to copy figure label and caption
Figure 2. Changes in Knowledge Evaluated Using Multiple-Choice Question Scores
Graphic Jump Location

Horizontal bars in boxes indicate medians; boxes, interquartile ranges; whiskers, range excluding outliers.

Tables

Table Graphic Jump LocationTable 1. Baseline Characteristics of Participants
Table Graphic Jump LocationTable 2. Changes in Knowledge and Skills Evaluated Using Scores Obtained With Multiple-Choice Questions and the Objective Structured Clinical Examinationa
Table Graphic Jump LocationTable 3. Changes in Educational Environmenta

References

Kulier R, Gee H, Khan KS. Five steps from evidence to effect: exercising clinical freedom to implement research findings.  BJOG. 2008;115(10):1197-1202
PubMed   |  Link to Article
Khan KS, Coomarasamy A. A hierarchy of effective teaching and learning to acquire competence in evidenced-based medicine.  BMC Med Educ. 2006;6:59
PubMed   |  Link to Article
Kunz R, Nagy E, Coppus SF,  et al.  How far did we get? how far to go? a European survey on postgraduate courses in evidence-based medicine.  J Eval Clin Pract. 2009;15(6):1196-1204
PubMed   |  Link to Article
Richardson WS. Teaching evidence-based practice on foot.  ACP J Club. 2005;143(2):A10-A12
PubMed
Hatala R, Keitz SA, Wilson MC, Guyatt G. Beyond journal clubs: moving toward an integrated evidence-based medicine curriculum.  J Gen Intern Med. 2006;21(5):538-541
PubMed   |  Link to Article
Oude Rengerink K, Thangaratinam S, Barnfield G,  et al.  How can we teach EBM in clinical practice? an analysis of barriers to implementation of on-the-job EBM teaching and learning.  Med Teach. 2011;33(3):e125-e130
PubMed   |  Link to Article
Coppus SF, Emparanza JI, Hadley J,  et al.  A clinically integrated curriculum in evidence-based medicine for just-in-time learning through on-the-job training: the EU-EBM project.  BMC Med Educ. 2007;7(1):46
PubMed   |  Link to Article
Kulier R, Hadley J, Weinbrenner S,  et al.  Harmonising evidence-based medicine teaching: a study of the outcomes of e-learning in five European countries.  BMC Med Educ. 2008;8:27
PubMed   |  Link to Article
Davis J, Chryssafidou E, Zamora J, Davies D, Khan K, Coomarasamy A. Computer-based teaching is as good as face to face lecture-based teaching of evidence-based medicine: a randomised controlled trial.  BMC Med Educ. 2007;7:23
PubMed   |  Link to Article
Davis J, Crabb S, Rogers E, Zamora J, Khan K. Computer-based teaching is as good as face to face lecture-based teaching of evidence-based medicine: a randomized controlled trial.  Med Teach. 2008;30(3):302-307
PubMed   |  Link to Article
Hadley J, Kulier R, Zamora J,  et al.  Effectiveness of an e-learning course in evidence-based medicine for foundation (internship) training.  J R Soc Med. 2010;103(7):288-294
PubMed   |  Link to Article
Kulier R, Coppus SF, Zamora J,  et al.  The effectiveness of a clinically integrated e-learning course in evidence-based medicine: a cluster randomised controlled trial.  BMC Med Educ. 2009;9:21
PubMed   |  Link to Article
World Health Organization (WHO).  RHL: The WHO Reproductive Health Library. WHO website. http://apps.who.int/rhl/en/index.html. Accessed October 29, 2012
Kulier R, Khan KS, Gülmezoglu AM,  et al.  A cluster randomized controlled trial to evaluate the effectiveness of the clinically integrated RHL evidence-based medicine course.  Reprod Health. 2010;7:8
PubMed   |  Link to Article
Straus SE, Green ML, Bell DS,  et al; Society of General Internal Medicine Evidence-Based Medicine Task Force.  Evaluating the teaching of evidence-based medicine: conceptual framework.  BMJ. 2004;329(7473):1029-1032
PubMed   |  Link to Article
Taylor R, Reeves B, Mears R,  et al.  Development and validation of a questionnaire to evaluate the effectiveness of evidence-based practice teaching.  Med Educ. 2001;35(6):544-547
PubMed   |  Link to Article
Ramos KD, Schafer S, Tracz SM. Validation of the Fresno test of competence in evidence-based medicine.  BMJ. 2003;326(7384):319-321
PubMed   |  Link to Article
Fritsche L, Greenhalgh T, Falck-Ytter Y, Neumayer HH, Kunz R. Do short courses in evidence-based medicine improve knowledge and skills? validation of Berlin questionnaire and before and after study of courses in evidence-based medicine.  BMJ. 2002;325(7376):1338-1341
PubMed   |  Link to Article
Pell G. Use and misuse of Likert scales.  Med Educ. 2005;39(9):970-971
PubMed   |  Link to Article
Cohen J. Statistical Power Analysis for the Behavioural Sciences. 2nd ed. New York, NY: Academic Press; 1977
Campbell MK, Elbourne DR, Altman DG.CONSORT Group.  CONSORT statement: extension to cluster randomised trials.  BMJ. 2004;328(7441):702-708
PubMed   |  Link to Article
Cook TD. Randomized experiments in educational policy research: a critical examination of the reasons the educational evaluation community has offered for not doing them.  Educ Eval Policy Anal. 2002;24(3):175-199Link to Article
Link to Article
Keogh-Brown MR, Bachmann MO, Shepstone L,  et al.  Contamination in trials of educational interventions.  Health Technol Assess. 2007;11(43):iii, ix-107
PubMed
Torgerson CJ. Educational research and randomised trials.  Med Educ. 2002;36(11):1002-1003
PubMed   |  Link to Article
Belfield C, Thomas H, Bullock A, Eynon R, Wall D. Measuring effectiveness for best evidence medical education: a discussion.  Med Teach. 2001;23(2):164-170
PubMed   |  Link to Article
Malick SM, Hadley J, Davis J, Khan KS. Is evidence-based medicine teaching and learning directed at improving practice?  J R Soc Med. 2010;103(6):231-238
PubMed   |  Link to Article
Hutchinson L. Educational environment.  BMJ. 2003;326(7393):810-812
PubMed   |  Link to Article
Holt MC, Roff S. Development and validation of the Anaesthetic Theatre Educational Environment Measure (ATEEM).  Med Teach. 2004;26(6):553-558
PubMed   |  Link to Article
Soemantri D, Herrera C, Riquelme A. Measuring the educational environment in health professions studies: a systematic review.  Med Teach. 2010;32(12):947-952
PubMed   |  Link to Article
Vieira JE. The Postgraduate Hospital Educational Environment Measure (PHEEM) questionnaire identifies quality of instruction as a key factor predicting academic achievement.  Clinics (Sao Paulo). 2008;63(6):741-746
PubMed   |  Link to Article
Roff S. The Dundee Ready Educational Environment Measure (DREEM)—a generic instrument for measuring students' perceptions of undergraduate health professions curricula.  Med Teach. 2005;27(4):322-325
PubMed   |  Link to Article
Gülmezoglu AM, Langer A, Piaggio G, Lumbiganon P, Villar J, Grimshaw J. Cluster randomised trial of an active, multifaceted educational intervention based on the WHO Reproductive Health Library to improve obstetric practices.  BJOG. 2007;114(1):16-23
PubMed   |  Link to Article
Weberschock T, Sorinola O, Thangaratinam S, Oude RK, Arvanitis TN, Khan KS. How to confidently teach EBM on foot: development and evaluation of a web-based e-learning course [published online August 4, 2012].  Evid Based MedLink to Article
CME
Also Meets CME requirements for:
Browse CME for all U.S. States
Accreditation Information
The American Medical Association is accredited by the Accreditation Council for Continuing Medical Education to provide continuing medical education for physicians. The AMA designates this journal-based CME activity for a maximum of 1 AMA PRA Category 1 CreditTM per course. Physicians should claim only the credit commensurate with the extent of their participation in the activity. Physicians who complete the CME course and score at least 80% correct on the quiz are eligible for AMA PRA Category 1 CreditTM.
Note: You must get at least of the answers correct to pass this quiz.
Please click the checkbox indicating that you have read the full article in order to submit your answers.
Your answers have been saved for later.
You have not filled in all the answers to complete this quiz
The following questions were not answered:
Sorry, you have unsuccessfully completed this CME quiz with a score of
The following questions were not answered correctly:
Commitment to Change (optional):
Indicate what change(s) you will implement in your practice, if any, based on this CME course.
Your quiz results:
The filled radio buttons indicate your responses. The preferred responses are highlighted
For CME Course: A Proposed Model for Initial Assessment and Management of Acute Heart Failure Syndromes
Indicate what changes(s) you will implement in your practice, if any, based on this CME course.

Multimedia

Supplementary Online Content

Kulier R, Gulmezoglu AM, Zamora J, et al. Effectiveness of a clinically integrated e-learning course in evidence-based medicine for reproductive health training: a randomized trial. JAMA. doi:10.1001/jama.2012.33640

eAppendix.

Supplemental Content

Some tools below are only available to our subscribers or users with an online account.

Web of Science® Times Cited: 4

Related Content

Customize your page view by dragging & repositioning the boxes below.

See Also...
Articles Related By Topic
Related Collections
PubMed Articles
JAMAevidence.com