Costs and Funding for Published Medical Education Research FREE

The Institute of Medicine and medical education leaders have called for increased rigor of education research.^1- 5 In a recent report on the future of academic health centers, the institute emphasized the need for academic health centers to reform medical education through the development and integration of new education curricula.¹ Included among the core recommendations of the report were calls for the development of educational programs that can be shown to improve the health of patients and new approaches to recognize and reward teachers and education researchers. Medical education researchers have also recognized the need for improved quality of education research,^4,6- 7 increased emphasis on theory building,^5,8- 9 larger multi-institutional studies,^10- 11 and demonstration of links between educational interventions and patient outcomes.^12- 15 Likewise, journal editors have acknowledged that education studies with sound methodologies and rigorous outcomes pertaining to health care delivery and quality of care are most likely to be accepted for publication.^12,16- 17

Despite the need to improve the rigor of education research, national funding for such research is limited. Less than 0.04% of federal spending in graduate medical education is used for education research.¹⁸ The Institute of Medicine has therefore recommended that Congress create an “ongoing fund that provides competitive grants to support educational innovation,”¹ and others have advocated creation of a national center for research in health professions education.¹⁹ Unfortunately, such national funding mechanisms have not yet been realized.^19- 20

How medical education research is currently funded and the costs of conducting studies in medical education have not been systematically evaluated. The objectives of this study were (1) to determine how medical education research studies that were recently published were funded, (2) to approximate the costs of conducting these studies, and (3) to elicit the opinions of the authors regarding funding for medical education research.

We conducted a cross-sectional survey of first authors of original medical education research studies published from September 1, 2002, to December 31, 2003. Studies conducted at US institutions and published in 13 prominent peer-reviewed journals were included. This study was approved by the Johns Hopkins Medical Institutions review board.

We selected studies from 2 journals encompassing all medical specialties (JAMA, New England Journal of Medicine), 4 medical education journals (Academic Medicine, Medical Education, Teaching and Learning in Medicine, Medical Teacher), and 7 journals representing the core specialty areas of internal medicine, general surgery, pediatrics, family medicine, obstetrics and gynecology, and emergency medicine (Annals of Internal Medicine, Journal of General Internal Medicine, The American Journal of Surgery, Pediatrics, Family Medicine, American Journal of Obstetrics and Gynecology, Academic Emergency Medicine).

The 7 specialty journals were selected based on a MEDLINE search for original medical education studies published within the study period. The search strategy included the keywords medical education and medical education research and the Medical Subject Heading term education, medical. The search was limited to publication dates September 1, 2002, to December 31, 2003, and to 6 publication types: randomized controlled trials, clinical trials, evaluation studies, validation studies, multicenter studies, or journal articles. The 7 specialty journals with the greatest number of articles resulting from this search were selected for review.

Medical education research was defined as any original research study pertaining to medical students, residents, fellows, faculty development, or continuing medical education for physicians. Studies pertaining to nonphysician health care professionals and patient education were excluded. Original research was defined as an educational intervention or trial, curriculum evaluation with subjective or objective outcomes, evaluation of an educational instrument or tool, surveys, and qualitative studies. Secondary analyses of existing data sets were included. Meta-analyses, systematic reviews, clinical reviews, letters, editorials, and reports of educational interventions without any evaluation or outcomes were excluded. If a first author published more than 1 report during the study period that met the inclusion criteria, the most recent report was included. Every issue of each of the 13 selected journals was hand searched for medical education studies meeting the inclusion criteria.

The survey instrument was developed by a team of medical education researchers and modified using an iterative process and repeated rounds of pilot testing among groups of medical education research authors. The survey assessed demographic information about authors, attainment of funding for the study, study duration, percentage of the authors’ total work commitment devoted to the study (“percentage effort”), resources used, and the costs of these resources. Author opinions regarding education research were measured using 5-point Likert scales (strongly disagree through strongly agree). The survey was sent to authors by mail in January 2004. To encourage full participation, nonrespondents were contacted by repeat mailings and e-mail.

The survey included a series of questions to help authors recall and more accurately report study costs. First, we asked authors to report the duration of their study from inception to first submission for publication. Second, we asked them to estimate the average percentage effort invested in the study per year by the first, second, and last authors. Examples such as “2 hours per week equals 5% effort” were given to provide respondents with a frame of reference to estimate their effort. Third, the authors reported the academic rank of the first, second, and last authors at the time the study was conducted. Fourth, we asked authors to estimate the cost of resources used, such as secretarial support, research assistants, data entry assistants, statisticians, equipment, and postage. Finally, we asked authors to estimate the total cost of their study, taking into account study duration, effort invested by each author, and resources used.

We calculated the cost of each study by multiplying the percentage effort dedicated by the first, second, and last authors by the median national salary for each author’s specialty according to that author’s academic rank.^21- 22 We then multiplied this number by the duration of the study in years and added the cost of the resources used. The calculation of study costs included the efforts of first, second, and last authors of each study because these authors most often contribute the majority of effort to studies. Salaries used were median salaries for physicians and nonphysician academicians in the United States according to specialty and academic rank. Salary data represent total compensation, which included fixed salary, medical practice supplement, bonus and incentive pay, and outside earnings.²² Benefits were not included. Study duration was recorded in intervals of less than 6 months, 6 months to less than 1 year, 1 year to less than 2 years, 2 years to less than 3 years, 3 years to less than 4 years, and 4 or more years. The midpoint of each interval in years was used for calculation of study costs.

We used descriptive statistics to summarize responses to all questions. Responses to 5-point Likert scales were dichotomized (agree and strongly agree vs neutral, disagree, and strongly disagree) and analyzed as proportions. Because the costs of studies were not normally distributed, estimated and calculated costs are reported using medians and interquartile ranges (IQRs).

We performed logistic regression to identify first-author factors associated with the acquisition of funding. The primary outcome was whether funding was obtained for the study. Variables included in the model were academic rank, fellowship training, training in grant writing, number of studies published, number of medical education studies published, and percentage effort dedicated to research. Model variables were examined for evidence of colinearity. In model building, we applied a user-defined stepwise approach. Variables were added based on level of significance in bivariate analysis. Model fit was assessed using the Pearson goodness-of-fit test and the Hosmer-Lemeshow χ² goodness-of-fit test.²³

We used the methods described by Hsieh et al²⁴ to adjust the sample size by a variance inflation factor accounting for the correlation among the 6 independent predictor variables included in the multivariable logistic regression analysis. The sample size of this investigation provides statistical power of approximately 80% to detect an association consistent with an odds ratio (OR) in the range of 2.25 to 2.50 or greater. Data were analyzed using STATA version 8.0 (Stata Corp, College Station, Tex). P<.05 was considered statistically significant.

A total of 665 medical education articles were published in the 13 peer-reviewed journals from September 1, 2002, to December 31, 2003. Forty-seven studies were excluded because they were not conducted in the United States and 328 because they did not meet the definition of original research. Of studies conducted in the United States, the most frequent reason for exclusion was that the report described a curricular intervention but lacked evaluation or outcome data. A total of 290 medical education studies published during the study period met all inclusion criteria.

Eighty-eight of the 290 studies were published in Academic Medicine, 51 in Family Medicine, 29 in Medical Education, 25 in Teaching and Learning in Medicine, 19 in Academic Emergency Medicine, 18 in American Journal of Obstetrics and Gynecology, 16 in Journal of General Internal Medicine, 15 in Medical Teacher, 12 in The American Journal of Surgery, 9 in Pediatrics, 6 in JAMA, 2 in Annals of Internal Medicine, and none in New England Journal of Medicine. Authors of 243 studies responded (response rate, 84%). The 243 studies included a wide range of study types: 30.4% program description, 30.0% survey, 13.2% qualitative, and 5.8% evaluation of a scale or tool (Table 1). Twenty-one percent of studies did not fit into any of these categories and were thus labeled as “other.” They included reports of residency match results, trends in career choices among graduating medical students, correlation of standardized test scores with clinical performance, chart audits, trends in medical school admissions, and reports of medical errors made by students and residents. Fifty-three percent of studies had 3 or fewer authors. The majority of studies (63.8%) were completed within 2 years. Seventy-four percent of studies were accepted to the first journal to which they were submitted.

Sixty-one percent of first authors were men, and the mean age of all authors was 44 years (Table 2). Nearly half of first authors held an academic rank of associate professor or professor. Students, residents, or fellows were first authors on 15.2% of studies. Fifty-four percent of authors had received formal training in grant writing. Authors spent a mean of 23.6% of their professional time in research activities. Respondents had published a mean of 23.3 manuscripts in peer-reviewed journals, with a mean of 8.5 of these being medical education studies. First authors invested an average of 12.1% effort in their studies; second and last authors invested 5.3% and 4.2%, respectively.

The median calculated cost of conducting these 243 medical education studies was $24 471 (interquartile range [IQR], $11 531-$63 808) (Table 3). Effort invested by first, second, and last authors comprised 79.9% of total study costs. Secretarial support (43.2%), postage (35.4%), and statisticians (32.9%) were the most frequently used resources. Research assistants were the most expensive resource used (median cost, $2750; IQR, $670-$7750). The median of the authors’ estimate of costs was $10 000 (IQR, $4000-$25 000), only 41% of the calculated costs.

Of the 243 studies, authors of 72 (29.6%) successfully obtained funding. The median amount of funding obtained was $15 000 (IQR, $5000-$66 500). The median calculated cost for studies that obtained funding was $37 315 (IQR, $18 731-$82 393), resulting in a deficit of $22 315 per funded study. Private foundation grants were the most common funding source (41.7% of funded studies) (Table 4).

In bivariate analysis, the only factors that were associated with the attainment of funding were training in grant writing (OR, 2.37; 95% confidence interval [CI], 1.33-4.22) and publication of a high number (greater than the mean) of peer-reviewed medical education studies by the first author (OR, 2.28; 95% CI, 1.23-4.19). In multivariable regression analysis, training in grant writing (adjusted OR, 2.05; 95% CI, 1.11-3.79) and number of published medical education studies (adjusted OR, 2.40; 95% CI, 1.24-4.63) remained independently associated with attainment of funding. There were no statistically significant associations between attainment of funding and academic rank, percentage effort dedicated to research, total number of studies published, or fellowship training in the bivariate or multivariable analyses.

A majority (79%) of authors agreed that there is inadequate national funding available for medical education research. Ninety-four percent agreed that, compared with national funding for clinical research, medical education research is moderately or vastly underfunded. Eighty-eight percent believed that increased funding would improve the quality of education research. Most authors (77%) reported difficulty in obtaining funding for medical education research. One third of authors had considered changing their research focus to fields other than medical education based on the dearth of funding opportunities.

This study suggests that the majority of published medical education research is not formally funded. Even those studies that were successful in obtaining support were substantially underfunded. Although the paucity of funding for research in education is frequently discussed in the literature,^{5- 6,19,25- 26} there has been a lack of empirical evidence to confirm these perceptions. Carline¹⁰ performed a review of the Acknowledgments sections of medical education studies published in 5 issues of Academic Medicine and Teaching and Learning in Medicine. Of 70 studies reviewed, 25 included an acknowledgment of funding. Twenty of these reported external funding, and 5 acknowledged internal funding. Our study, using a more systematic approach, confirmed a similar proportion of funding for medical education studies. Furthermore, by surveying first authors, we were able to estimate the costs of conducting the studies.

Our calculation of study costs was designed to provide a conservative approximation of actual study costs, and the gap between study costs and funding received is likely wider than we report. Our calculation included only resources used and time invested by first, second, and last authors. Indirect costs and benefit packages that are connected with the salaries of the authors were not included. While the first, second, and last authors likely contributed the majority of effort, time invested by other authors was not considered. Likewise, the costs of work performed by students, residents, and fellows were not included, even if they were first, second, or last authors.

Despite a conservative estimate of study costs, our results indicate that the majority of studies did not receive adequate support to cover costs. For studies that received funding, there was a discrepancy of at least $20 000 between study costs and funding, and the median calculated cost for the 171 published studies that were not funded was $18 454. In addition, for every study that achieves publication, there are likely many others that are not published. This raises the question of who is paying for this research. In many cases, academic health centers may be shouldering the costs through their support of protected time for faculty who are conducting educational research.^27- 28 These costs may be particularly onerous, as the financial pressures facing academic divisions have become increasingly challenging.^29- 30

First authors considerably underestimated the cost of their studies. Even when prompted to consider all costs, including faculty time and resources, the median cost estimated by authors was less than half that of the median calculated cost. While authors’ underestimation of study costs may be a result of poor recall, careful prompting provided by the survey instrument should have helped minimize this. Our results show that 79.9% of the costs of conducting these medical education studies was attributable to effort contributed by first, second, and last authors. Authors may not recognize the substantial cost associated with investment of faculty time, even their own, particularly over a prolonged period. The relevance of this finding is that if authors cannot accurately estimate the cost of their studies after the fact, they will probably not be able to appropriately budget for proposed future studies.

Even among the successfully published authors surveyed in this study, one third have considered changing their research pursuits to fields other than medical education due to the lack of funding opportunities. It is alarming, particularly at a time when there is a call for higher-quality education research,^1- 5,7,13 that some of the most talented investigators are contemplating leaving this field. Even the most passionate and committed researchers may choose alternate domains of study if they are unable to support the protected time they need to carry out their scholarly activities. The creation of new funding opportunities^1,19 will have an important role in retaining talented medical education investigators.³¹

Although in this article we do not evaluate the relationship of funding to the quality of the included papers, almost 90% of authors believed that increased funding would lead to improved quality of medical education research. Acceptance for publication may be a proxy for study quality, and among manuscripts submitted to the medical education issue of 1 noneducation journal, those accepted were almost twice as likely to have received at least partial funding compared with those rejected.³² Increased funding may encourage the use of stronger study designs, including controlled educational interventions and multi-institutional collaborations.

Several limitations of this study should be considered. First, we relied on self-report of first authors about their studies. Second, as some studies were conducted a year or more before they were published, responses may be subject to inaccurate recall. Third, our calculations of costs are rough approximations that are intentionally conservative. Fourth, certain grants may support a body of work rather than a single study, making it difficult for researchers to estimate the funding associated with a particular study. Fifth, the power analysis suggests that, given the sample size, we may have had limited power to detect associations between the attainment of funding and first-author characteristics in the multivariable analysis. Finally, an external comparator for research funding for other types of medical scholarship is lacking, so that it is difficult to judge what amount of funding or research effort is appropriate. Research comparing funding between medical education and clinical research studies of similar scope would be a valuable next step, as would be studying the relationship between study quality and funding.

These limitations notwithstanding, this study provides empirical data on funding for medical education research. These findings add support to the Institute of Medicine’s call for increased funding in this area. At present, academic medical centers appear to be paying for much of the costs associated with scholarly work in medical education. Additional sources of support from national funding agencies might further promote high-quality medical education research.

The calculated cost of conducting this study was $23 083. The cost of author effort for first, second, and last authors was $20 953. The cost of resources was $850 for data entry, $640 for secretarial assistance, $480 for statistical consultation, and $160 for postage. Dr Reed received financial support as a Mayo Foundation Scholar and as a Ruth L. Kirschstein National Research Service Award trainee; Dr Wright received support as an Arnold P. Gold Foundation Associate Professor of Medicine. Funding for this study from these 3 sources totaled $17 625.