Trends in Study Methods Used in Undergraduate Medical Education Research, 1969-2007 FREE

Evidence-based medicine, an accepted construct that asserts that medical interventions should be guided by data from rigorous studies, was described at least 15 years ago.¹ The ideal that medical educators should choose teaching approaches based on evidence soon followed.^2- 4 However, educators cannot be guided by evidence if the studies providing that evidence have significant shortcomings.

Concerns about the rigor of medical education research have been identified, but previous reviews have been limited in scope. Limitations identified in medical education research include insufficient sample sizes, lack of generalizability when interventions are assessed at only 1 institution or are conducted only once, lack of appropriate control groups, use of subjective and unvalidated instruments to assess outcomes, and assessing only short-term outcomes.^5- 12

Previous reviews of medical education that examine study methods have been limited to specific educational topics^6,13- 15; have looked at only 1 aspect of studies, such as funding or outcomes^12,16- 17; or have reviewed only studies published in specific journals.^12,16- 18 None have examined the timing of outcome assessments, whether interventions were conducted more than once, whether statistical power was estimated, or whether participation of medical education departments or centers had an impact on methods. Comprehensive reviews have not included non–English-language research or explored whether study methods are improving over time.

We conducted a structured longitudinal review of the undergraduate medical education literature from nearly 4 decades to characterize historical and current study methods and to evaluate whether participation of medical education departments or centers is associated with more rigorous methods. We hypothesized that we would observe a trend for increasingly rigorous studies over time. We further hypothesized that publications demonstrating participation of medical education departments or centers would exhibit more features of rigorous methods.

We analyzed interventions involving medical students rather than medical education more generally because interventions for this group are most similar internationally and, thus, most comparable. Many curricular topics are relevant to all medical students, whereas much educational content for residents and practicing physicians is specialty-specific. We used PubMed, the Cochrane Controlled Trials Registry, the Campbell Collaboration, and the ERIC databases to identify publications indexed under the exploded Medical Subject Heading terms students, medical or education, medical or equivalent terms. We crossed these search terms with descriptors of relevant study designs (Box). We included publications in all languages. To assess trends over time, we selected publications from academic years (July 1–June 30) 1969-1970, 1974-1975, 1979-1980, 1984-1985, 1989-1990, 1994-1995, 1999-2000, and 2004-2005. We then added the most recent 12 months (April 1, 2006–March 31, 2007).

^aMedical Subject Headings (MeSH).
^bPublication types.

Publications were excluded if they (1) were not about undergraduate medical students; (2) were opinion pieces, editorials, or critiques of prior studies; (3) were reviews summarizing or interpreting preexisting data; or (4) did not evaluate an educational intervention. Examples of publications in this last group include studies that examined characteristics of medical students such as health status or demographics, analyses of predictors of student outcomes such as comparisons of Medical College Admission Test scores to specialty choice, and studies that assessed the performance of testing instruments.

We developed and piloted a standardized data abstraction form that documented the number of medical student participants, number and nationality of institutions involved, whether a control or comparison group was used, methods for assigning controls, outcome variables assessed, use of validated assessment instruments, timing of outcome assessments, number of times the intervention was assessed, whether statistical power was estimated, and cost of the intervention. Sources of funding were also recorded. Author affiliations and acknowledgments were examined to identify whether a medical education department or center had participated in the study.

Features of rigorous study methods were defined based on published guidelines for appraising the effectiveness of medical education curricula.^10,19- 20 More rigorous study methods were defined as (1) greater number of student participants; (2) multi-institutional enrollment of participants; (3) having a control or appropriate comparison group; (4) measuring an objective outcome; (5) measuring a validated outcome; (6) measuring some outcome at least 1 month after the intervention; (7) conducting the intervention more than once; and (8) estimating statistical power.

Publications were categorized as multi-institutional if they recruited participants from more than 1 medical school (including studies that used one school as the intervention group and another as the control). Objective outcome measures were defined as any evaluation other than self-assessment by students, including (1) tests of knowledge; (2) course grades; (3) objective structured clinical examinations or observations of standardized patient interactions; (4) assessment of performance with a real patient, such as graded observation of patient interactions or patient feedback; (5) objective clinical outcomes such as blood pressure control in patients; (6) performance with mannequins, computer simulations, or laboratory animals; and (7) psychological inventories. We also noted assessment of subjective outcomes, such as student satisfaction, self-assessment of competence, or attitude. Specialty choice or success in obtaining desired residency position was recorded as an outcome but was not categorized as either objective or subjective. The study outcomes were further classified into the levels of behavioral change of the Kirkpatrick²¹ hierarchy: (1) participation; (2a) modification of attitudes or perceptions; (2b) modification of knowledge and skills; (3) behavior change in the workplace; (4a) change in organizational practice; and (4b) benefit to patients.

Outcome measures were considered to be validated if the authors stated that their evaluation tool was validated beyond face validity or used tests that are generally known to be validated, such as United States Medical Licensing Examinations. Funding was noted if the publication's text, acknowledgments, or separate disclosure section reported any financial support for the study. Sources of funding were also recorded. Participation of a medical education department or center was defined as having any author or acknowledged person listed with an affiliation with a medical education department or center or if a medical education department or center was acknowledged.

All publications were abstracted by 1 author (A.B. or H.K.L.). To assess the validity of data abstraction, 10% of publications were selected using a random number generator and abstracted by 2 authors (A.B. and H.K.L.). Variables collected were compared for consistency. Discrepancies in coding of dual-reviewed publications were resolved by consensus. Each abstractor was blinded to the other's abstraction but not to our study hypotheses.

Descriptive analysis compared categorical variables over time using the χ² test for trend. Continuous variables were evaluated with the t test (if compared with dichotomous variables) or 1-way analysis of variance (if comparing over time). Data from the earliest years were grouped according to an a priori scheme when sample sizes were too small (<5 expected counts per cell) to allow analyses; this was necessary because there were few publications in the earliest periods. Percentage agreement and the Cohen κ statistic were used to assess interrater agreement between abstractors. Statistical significance was set at 2-sided P < .05. The sample size of 472 articles was large enough to provide 80% power to detect small differences (effect size, 0.12) for the variable “participation of medical education departments and centers” with an α level of .05 for a 2-tailed χ² test.²² The analysis was performed using SPSS software, version 14.0 (SPSS Inc, Chicago, Illinois).

The PubMed search strategy identified 10 213 publications spanning a 38-year period (Figure). Of these, 2974 publications were within the defined periods of our sampling strategy and were selected for review. The PubMed search yielded 459 publications appropriate for full abstraction. The ERIC, Cochrane, and Campbell databases were searched using equivalent strategies, resulting in 13 nonduplicate publications appropriate for full abstraction. Thus, 472 publications were included in the analysis. Reasons for exclusion are shown in the Figure. Between 9% and 19% of publications retrieved by PubMed from a given year met inclusion criteria, and there was no change over time in the percentage included.

Approximately half of included publications appeared in journals dedicated to medical education, such as Academic Medicine and Medical Education (46%). The rest were published in specialty journals, such as Annals of Internal Medicine (36%); other types of journals, including basic science journals (11%); and general medical journals, such as JAMA and BMJ (7%).

Eighteen publications suitable for inclusion were published in non-English languages, including German (9), Spanish (5), Arabic (1), Japanese (1), Portuguese (1), and Swedish (1). These publications were translated prior to inclusion in the analysis. Studies were conducted in many regions of the world, including the United States (263), Europe (101), Asia (36), Canada (27), Australia (23), Latin America (11), Africa (10), and multiple regions (1).

Interrater agreement on the decision to include or exclude publications was 98% (κ = 0.91). Agreement on data from included publications ranged from 77% to 100% (mean [SD], 92% [5.8%]), with κ values ranging from 0.57 to 1.00 (mean [SD], 0.81 [0.12]), indicating excellent agreement.²³

Of the publications, 147 were published between April 2006 and March 2007 (Table 1). The mean number of medical student participants was 145 (median, 81; range, 4-1663). Recruitment of participants from more than 1 institution was present in 6% of publications. A little more than half of the publications (n = 80 [54%]) used a comparison group with the following designs: randomized controls, 37 (25%); nonrandomized concurrent controls, 28 (19%); historical controls, 11 (7%); comparison with national norm, 2 (1%); and more than 1 kind of comparison group, 2 (1%). Publications with no comparison group were divided between 24 (16%) with pretest/posttest designs and 43 (29%) with posttest data only.

The outcomes assessed in recent publications were subjective in 56 (38%), objective in 38 (26%), and both in 53 (36%), with types of objective outcomes shown in Table 2. Validated outcomes were uncommon (n = 23 [16%]). The interval between the intervention and the last assessment was less than 1 month in the majority of publications (n = 102 [69%]). More than 1 presentation of the intervention was evaluated in 71 (51%) of the publications (Table 3). It was rare to have reporting of the statistical power of the study (n = 21 [14%]) or the cost of the intervention (n = 5 [3%]).

Among the recent publications, 109 (74%) measured an outcome at level 1 or 2a of the Kirkpatrick hierarchy²¹ (participation; modification of attitudes or perceptions), 75 (51%) measured an outcome at level 2b (modification of knowledge and skills), 5 (3%) measured an outcome at level 3 (behavior change in the workplace), and none measured an outcome at level 4b (benefit to patients). Five (3%) examined subsequent practice among an underserved population, which may fit Kirkpatrick level 4a (change in organizational practice).

Sixty-six (45%) of the recent publications reported receiving funding (Table 3). Sources of funding included government agencies (n = 31 [47% of funded studies]), private foundations (n = 28 [42%]), internal grants and awards (n = 25 [38%]), and pharmaceutical or device companies (n = 10 [15%]). Of the publications, 22 reported multiple funding sources.

The annual number of publications increased over time from 1 (1969-1970) to 147 (2006-2007). Study size was stable over time, with the mean number of medical student participants ranging from 105 (1969-1970) to 387 (1984-1985). The number of publications demonstrating the other features of rigorous methods, reporting funding, and reporting participation of medical education departments and centers increased over time (Table 1, Table 2, and Table 3).

An increasing percentage of publications per year estimated statistical power (range, 0% in 1969-1985 to 14% in 2006-2007; P = .003) and reported funding (range, 0% in 1969-1970 to 45% in 1999-2000 and 2006-2007; P = .002) (Table 3). However, the percentage of published studies incorporating most of the features of rigorous study methods has not changed over time (Table 1, Table 2, and Table 3).

Percentage of publications with participation of medical education departments or centers did not change over time (Table 3). Participation, documented in 139 (29%) of the publications, was associated with measuring an objective outcome (76% of publications with participation vs 66% of publications without participation; P = .048) (Table 4). This association was not significant when examining only 2006-2007 publications (P = .14). However, 2006-2007 publications enrolled more medical student participants if a medical education department or center participated (mean number of medical students, 218 vs 111; P = .04). Publications in medical education journals compared with other types of journals enrolled more medical student participants (mean number of medical students, 217 vs 137; P = .007) and were more likely to measure a validated outcome (22% vs 13%; P = .02).

This review, which quantifies the methods used to evaluate the efficacy of undergraduate medical education, found that educational researchers use experimental and epidemiologic study designs, measure objective and validated outcomes, and assess long-term outcomes. We found that the number of publications evaluating undergraduate medical education interventions has increased over time, and every feature of stronger methods we examined was used in more studies each year. Even though the percentage of studies demonstrating many of the features of strong methods has not changed over time, educators have an ever larger pool of rigorous published studies to guide their curricular decision making.

Participation of medical education departments and centers was associated with use of objective outcomes and number of participants in our analysis. Detecting associations with other features of study rigor may have been limited by our inability to detect either affiliations with medical education departments and centers that were not documented in the publications or informal consultations. Physicians who teach face conflicting pressures and expectations among their clinical duties, teaching responsibilities, and scholarly requirements; they may be able to produce more rigorous evaluations of curricular innovations when supported by medical education departments and centers.^6,11,24- 25 Creating national centers for educational research excellence that can facilitate collaboration, review emerging evidence, disseminate key findings, and advise credentialing agencies may improve the quality of educational research.^7,26

The majority of published undergraduate medical education research examines local processes and relies on student satisfaction and short-term acquisition of knowledge to measure educational efficacy. There was a paucity of research linking educational interventions to improved patient outcomes. While it may be difficult to demonstrate improvements in patient outcomes because of confounding factors, including individual student characteristics, the multitude of intervening experiences between medical school and practice, and the influences of varying practice environments, the analytic techniques that help overcome such obstacles in clinical research can also be applied in educational settings.²⁷

An essential step toward making the medical education research endeavor more relevant to society is to identify patient-level outcome measures that can be linked to educational interventions.^{11,26,28- 29} In biomedical research, outcomes such as mortality, quality-adjusted life-years, and glycemic control are recognized as meaningful. Equivalent measures should be used in medical education. Educators currently use validated knowledge-based outcome measures (such as board examinations) as intermediary end points. However, whether they are appropriate surrogate markers for the larger goal of improving the health of a population is still unclear.

A research agenda for medical education that is based on patient-level clinical outcomes has been described.⁷ Collaboration with experts in health services has been identified as essential to establish outcome-driven education.^11,29 It is not practical to assess all undergraduate medical education interventions using patient-level outcomes, but identifying deficiencies and opportunities for improvement in current health care delivery should play a much larger role in guiding curriculum development.²⁸

Of the publications we reviewed, 2 used an objective clinical outcome to demonstrate the effectiveness of their intervention. These studies, which demonstrated that it is possible to link education to patient outcomes, used patients' weight loss in Japan³⁰ and adherence with tuberculosis therapy in South Africa³¹ to judge the effectiveness of a practicum in public health. Most clinical outcomes will not be evident until students have finished training and moved on to practice, but only 5% of recent publications assessed an outcome more than 1 year after the intervention. The effect of a curriculum on health care delivery or patient outcomes cannot be ascertained in that time frame; the length of follow-up must increase markedly to measure such educational effects. Medical education research can be made more meaningful by (1) planning prospective studies with mechanisms for following participants over many years, (2) making better use of rigorous retrospective studies, and (3) clarifying linkages between educational processes and patient health outcomes.

There is significant unrealized potential for medical education research to improve the quality of future physicians. For example, medical schools periodically undertake complete overhauls of their undergraduate curricula,^32- 36 providing opportunities to evaluate the effect of curricula on participant learners. While major curricular reforms such as those at Harvard,³⁷ University of Missouri–Columbia,³⁸ McMaster,^39- 41 Maastricht,⁴² and the University of New Mexico^43- 44 have been evaluated on long-term outcomes, they have not included objective measures of graduates' practice behavior and performance.

Educational efficacy can be evaluated with study designs other than randomized controlled trials.^{9- 11,45- 48} Different study designs are appropriate for different types of research questions. For example, qualitative research^9,19,49 can explore why an intervention does or does not work and is an effective way to generate hypotheses. Investigators can reduce bias and confounding while testing hypotheses with nonexperimental quantitative designs, such as cohort and case-control studies.²⁷ Crossover designs eliminate the ethical concern of depriving one group of a potentially better educational experience.

We found that a greater percentage of publications over time reported receiving funding. This is encouraging, as high-quality research in any field is challenging without funding, and increasing funding has been proposed as a specific strategy for improving medical education research.^{6,9- 10,16- 17,26} Research funding buys faculty time, statistical expertise, and administrative support for long-term studies that have sufficient power to measure meaningful outcomes. Reed et al¹⁷ investigated medical education studies conducted from 2002-2003 and found that 30% were funded, whereas 45% of recent publications in our sample reported funding. The difference may be due to inclusion criteria: our sample was international and limited to undergraduate medical education, whereas their sample was US researchers and spanned the continuum of undergraduate, graduate, and continuing medical education. The most common sources of funding in both samples were government agencies and private foundations. We were not able to determine the amount of funding for each publication or measure association of funding with study methods.

Educators considering a curricular change need to know the cost as well as the potential benefits of the program they are considering.^10,12,14,20 Although calculating the cost of an educational program requires considerable effort,^17,50 some costs, such as equipment or standardized patient salaries, can be estimated. However, we found that only 3% of recent publications reported the cost of any part of their program, reducing their usefulness to educators making cost-benefit analyses before adopting new curricula.

Limitations of our study need to be considered. We conducted a current review, updating our list of studies 3 months after the latest publication date, although some articles may not yet have been indexed. The criteria we used to define rigorous methods are important but are not the only ones relevant for generating meaningful data. We did not examine whether a study asked an important question, chose apt study methods to address the question, or used an appropriate conceptual framework.^10,19,51 However, because such criteria cannot be objectively determined, they were not appropriate for this analysis.

Strengths of our study include reviewing a large sample of publications, examining all topics in the undergraduate curriculum, and reviewing publications from 4 databases and with no language restrictions. Other authors have systematically reviewed specific educational topics and have reached similar conclusions about insufficient rigor of research in medical education.^{5- 6,13- 15} We have added to these findings by demonstrating that these problems are present throughout the undergraduate medical education literature and by showing ways in which important research features have changed over time. While the percentage of published research with strong methods has not changed appreciably over the years, more education studies are being conducted every year; therefore, the absolute number of rigorous studies available is increasing. Appropriate application of study design and development of clinically meaningful outcomes have the potential to make medical education research more relevant to the health needs of society.