0
We're unable to sign you in at this time. Please try again in a few minutes.
Retry
We were able to sign you in, but your subscription(s) could not be found. Please try again in a few minutes.
Retry
There may be a problem with your account. Please contact the AMA Service Center to resolve this issue.
Contact the AMA Service Center:
Telephone: 1 (800) 262-2350 or 1 (312) 670-7827  *   Email: subscriptions@jamanetwork.com
Error Message ......
Original Investigation |

Association of MCAT Scores Obtained With Standard vs Extra Administration Time With Medical School Admission, Medical Student Performance, and Time to Graduation FREE

Cynthia A. Searcy, PhD1; Keith W. Dowd, MA1; Michael G. Hughes, PhD2; Sean Baldwin, PhD2; Trey Pigg, BA1
[+] Author Affiliations
1Association of American Medical Colleges, Washington, DC
2Human Resources Research Organization, Alexandria, Virginia
JAMA. 2015;313(22):2253-2262. doi:10.1001/jama.2015.5511.
Text Size: A A A
Published online

Importance  Individuals with documented disabilities may receive accommodations on the Medical College Admission Test (MCAT). Whether such accommodations are associated with MCAT scores, medical school admission, and medical school performance is unclear.

Objective  To determine the comparability of MCAT scores obtained with standard vs extra administration time with respect to likelihood of acceptance to medical school and future medical student performance.

Design, Setting, and Participants  Retrospective cohort study of applicants to US medical schools for the 2011-2013 entering classes who reported MCAT scores obtained with standard time (n = 133 962) vs extra time (n = 435), and of students who matriculated in US medical schools from 2000-2004 who reported MCAT scores obtained with standard time (n = 76 262) vs extra time (n = 449).

Exposures  Standard or extra administration time during MCAT.

Main Outcomes and Measures  Primary outcome measures were acceptance rates at US medical schools and graduation rates within 4 or 5 years after matriculation. Secondary outcome measures were pass rates on the United States Medical Licensing Examination (USMLE) Step examinations and graduation rates within 6 to 8 years after matriculation.

Results  Acceptance rates were not significantly different for applicants who had MCAT scores obtained with standard vs extra time (44.5% [59 585/133 962] vs 43.9% [191/435]; difference, 0.6% [95% CI, −4.1 to 5.3]). Students who tested with extra time passed the Step examinations on first attempt at significantly lower rates (Step 1, 82.1% [344/419] vs 94.0% [70 188/74 668]; difference, 11.9% [95% CI, 9.6% to 14.2%]; Step 2 CK, 85.5% [349/408] vs 95.4% [70 476/73 866]; difference, 9.9% [95% CI, 7.8% to 11.9%]; Step 2 CS, 92.0% [288/313] vs 97.0% [60 039/61 882]; difference, 5.0% [95% CI, 3.1% to 6.9%]). They also graduated from medical school at significantly lower rates at different times (4 years, 67.2% [285/424] vs 86.1% [60 547/70 305]; difference, 18.9% [95% CI, 15.6% to 22.2%]; 5 years, 81.6% [346/424] vs 94.4% [66 369/70 305]; difference, 12.8% [95% CI, 10.6% to 15.0%]; 6 years, 85.4% [362/424] vs 95.8% [67 351/70 305]; difference, 10.4% [95% CI, 8.5% to 12.4%]; 7 years, 88.0% [373/424] vs 96.2% [67 639/70 305]; difference, 8.2% [95% CI, 6.4% to 10.1%]; 8 years, 88.4% [375/424] vs 96.5% [67 847/70 305]; difference, 8.1% [95% CI, 6.3% to 9.8%]). These differences remained after controlling for MCAT scores and undergraduate grade point averages.

Conclusions and Relevance  Among applicants to US medical schools, those with MCAT scores obtained with extra test administration time, compared with standard administration time, had no significant difference in rate of medical school admission but had lower rates of passing the USMLE Step examinations and of medical school graduation within 4 to 8 years after matriculation. These findings raise questions about the types of learning environments and support systems needed by students who test with extra time on the MCAT to enable them to succeed in medical school.

The Medical College Admission Test (MCAT) is a standardized examination that measures knowledge of undergraduate introductory-level biology, chemistry, and physics concepts along with higher-order thinking skills important for success in medical school. MCAT scores are required for admission by most medical schools in the United States and Canada and predict a variety of local and national student outcomes across all 4 years of medical school.14

Individuals with documented mental disabilities (eg, learning disabilities, attention-deficit disorder or attention-deficit/hyperactivity disorder [ADD/ADHD], psychiatric disorders) or physical disabilities (eg, impairments related to vision, mobility, hearing) may receive testing accommodations on the MCAT and other admissions tests in accordance with professional testing standards and federal law.5(pp67,68)6 Testing accommodations are alterations to standard test administration procedures7 designed to enable test takers to show their proficiencies rather than the extent of their disabilities.5(p67) Examples of accommodations include presentation of testing materials in large print, extra testing time, a separate testing room, or authorization to bring an inhaler into the testing room.

Ideally, testing accommodations are unrelated to the knowledge and skills being assessed by a test8 so that scores obtained under accommodated conditions have the same meaning as scores obtained under standard conditions.5(pp65,66) Comparable scores from standard and accommodated testing conditions should predict comparable levels of student performance in medical school. Extra administration time (extra time) is a common accommodation for a variety of mental and physical disabilities913 and is the most frequently provided accommodation on the MCAT.14 This study investigated the comparability of MCAT scores in relation to medical school admission and medical student performance for individuals who had scores obtained with standard vs extra time.

This study was approved by the institutional review board of the American Institutes for Research (FWA00001666) as part of the MCAT program’s psychometric research protocol. Individuals provided explicit consent for their data to be used for research purposes when they registered for the MCAT. All data were deidentified before analysis by removing all personally identifying information.

Study Populations

We studied data from 2 time intervals using data maintained by the Association of American Medical Colleges (AAMC). For each time interval, data for each application or matriculation year were constructed independently and then combined into a single data set. In both data sets, only individuals whose medical school applications included both MCAT scores and total cumulative undergraduate grade point averages (UGPAs) were included.

To examine acceptance rates reflecting current admissions practices, MCAT test administration and accommodations procedures, extra-time conditions, and disability status of recent applicants, we studied all applicants to 140 US MD-granting medical schools for the 2011-2013 entering classes. For each application year, a record included application and acceptance data for a single applicant to a single medical school, including demographic data (sex, age at application), MCAT data at the time of application (the most recent MCAT section and total scores, the amount or type of extra time [25%, 50%, 100%, nonstandard breaks], an indicator denoting if the MCAT scores were from the first or a repeat attempt), disability status, UGPA, medical school institution code, and an admission acceptance indicator. Applicants who applied to more than 1 medical school within the same application cycle were included once for each medical school to which they applied. Applicants who applied in more than 1 year were included in each year they applied; each record reflects the application and acceptance data that were relevant for that year and school only.

To examine medical student performance outcomes, we studied all students matriculating to US MD-granting medical schools from 2000-2004. All students in these analyses had at least 8 years to graduate, twice the time associated with a typical 4-year medical school curriculum. For each year, a record included application and outcome data for a single medical student, including medical school institution code; demographic data collected in the application (sex, age at application); MCAT data at the time of application (the most recent MCAT section and total scores, an indicator denoting if the administration included extra time, an indicator denoting if the MCAT scores were from the first or a repeat attempt); UGPA; pass/fail outcomes on the United States Medical Licensing Examination (USMLE) Step 1, Step 2 Clinical Knowledge (Step 2 CK), and Step 2 Clinical Skills (Step 2 CS) examinations as of May 22, 2014 (first and final attempt); and graduation year. Complete data on disability status and the amount or type of extra time received on the MCAT were not available for this time interval.

Admissions Variables

The MCAT total score was the sum of an individual’s scores on the 3 multiple-choice sections of the MCAT in place from 1991 to January 2015: Verbal Reasoning, Biological Sciences, and Physical Sciences. The Verbal Reasoning section assesses the ability to understand, evaluate, and apply information and arguments presented in text. The Biological Sciences and Physical Sciences sections assess the ability to apply introductory-level knowledge of biology, chemistry, and physics to solve scientific problems. Scores for each multiple-choice section are reported on a 15-point scale, resulting in an MCAT total score ranging from 3 to 45. As part of the MCAT accommodations determination process, individuals submitted information demonstrating functional impairment and past accommodations history, and approval to receive extra time was based on a careful review of this evidence. Individuals granted extra administration time were provided with 25%, 50%, or 100% extra testing time or nonstandard breaks; the accommodation of 25% extra time was added in 2007 and is relevant for the 2011-2013 cohort only.

The UGPA, ranging from 0.00 to 4.00, is the mean of an individual’s grades from all undergraduate courses and was standardized to account for the different academic calendars and grading systems used by different undergraduate institutions. It includes grades from postbaccalaureate programs but excludes grades from any graduate courses.

Outcome Variables

Primary outcome measures were acceptance offers by US medical schools and graduation within 4 or 5 years after matriculation. Secondary outcome measures were passing Step 1, Step 2 CK, and Step 2 CS on the first attempt and eventually (defined as the last attempt on record as of May 22, 2014, when the data sets were constructed) and graduation within 6 to 8 years after matriculation. When analyzing graduation outcomes, we excluded students enrolled in joint programs (eg, MD-PhD) or other special programs that typically require more than 4 years of study, as well as students who were deceased prior to degree completion. All outcome measures were dichotomous; a score of 1 reflects success (eg, received an acceptance offer; graduated within 4 years), and a score of 0 reflects absence of success (eg, did not receive an acceptance offer; did not graduate within 4 years).

Statistical Analysis

The unit of analysis was the individual applicant or medical student within a US medical school. The admissions and student outcome data were compared for individuals who took the MCAT with standard or extra time, evaluating differences with t tests, χ2 tests, and effect sizes estimates.

In addition to computing descriptive statistics on the admissions variables and student outcomes, multivariable logistic regressions were conducted to examine the predictive relationship between the set of admissions variables (MCAT total scores and UGPAs) and each of the primary and secondary outcomes. Both MCAT total scores and UGPAs were included in the regression analyses because admissions decisions are influenced by both measures of academic preparation15 and because research has shown that together, MCAT scores and UGPAs predict future performance better than either one alone.1

The association between the set of admissions variables and each outcome was examined within school to allow for differences in admissions processes, academic support, curricula, and evaluation across medical schools. For each regression model, the logit coefficients for MCAT total scores and UGPAs were estimated for individuals who tested with standard time only, entering MCAT total scores and UGPAs in a single step (the baseline regression model). Model discrimination was evaluated with the C statistic, defined as the proportion of times the model correctly discriminates a pair of successful and unsuccessful individuals. A C statistic of 0.50 indicates that the model performs no better than chance, a C statistic of 0.70 to 0.80 indicates modest or acceptable discriminative ability, and a C statistic of greater than 0.80 indicates good discriminative ability. Goodness of fit was evaluated with the G statistic, and P values less than .05 indicate that 1 or both predictors contributed significantly to the model.16

The resulting parameter estimates from the within-school regression analyses were applied to the data for students who tested with extra time, allowing a comparison of the predictive meaning of MCAT total scores obtained with extra time vs that of scores obtained with standard time at each medical school. By applying parameter estimates from the baseline regressions to data for individuals who tested with extra time, estimates of the probability of success (eg, graduating within 4 years) were obtained for each individual who tested with extra time.

To estimate overall prediction error, predicted and observed outcomes from the within-school analyses were aggregated separately for individuals who took the MCAT with standard vs extra time. For each group, the number (and percentage) who were predicted to succeed were compared with the observed number (and percentage) of students who succeeded. Positive differences indicated that fewer students who tested with extra time succeeded than predicted, whereas negative differences indicated that more students succeeded than predicted. Prediction error was evaluated with 95% confidence intervals and t tests (2-tailed), and differences in prediction error for students who tested with standard vs extra time were evaluated with effect size estimates (Cohen d).

All statistical tests were 2-sided and considered significant at P < .05. All analyses were conducted using SAS version 9.3 (SAS Institute).

Study Participants for Admission Analyses

In total, 98.0% (134 397/137 199) of the applicants to the 2011-2013 entering classes applied with both MCAT scores and UGPAs and were included in the analysis of admission decisions. Demographic characteristics, MCAT scores, and UGPAs for these applicants are shown in Table 1.

Table Graphic Jump LocationTable 1.  Characteristics of Applicants to the 2011-2013 Entering Classes of Medical School

The vast majority of these applicants took the MCAT with standard time (99.7% [133 962/134 397]) compared with extra time (0.3% [435/134 397]). Of the applicants who tested with standard time, 53.5% (71 680/133 962) were male, and the mean age at application was 24.3 years. Of applicants who tested with extra time, 56.3% (245/435) were male, and the mean age was 25.9 years. For these same applicants, 130 (29.9%) tested with 25% extra time, 236 (54.3%) tested with 50% extra time, and 38 (8.7%) tested with 100% extra time. A total of 59 applicants (13.6%) testing with extra time also received nonstandard breaks, and another 23 (5.3%) received nonstandard breaks that added time to their testing day but did not receive 25%, 50%, or 100% extra time in addition to their nonstandard breaks. Eight applicants (1.8%) were missing information about the amount of extra time. Of the applicants who tested with extra time, 337 (77.5%) had mental impairments (13 with psychiatric disabilities only and 324 with ≥1 impairments related to ADD/ADHD, learning, or psychiatric disabilities), 61 (14.0%) had physical impairments, 27 (6.2%) had both physical and mental impairments, 2 (0.5%) had other impairments, and 8 (1.8%) were missing information about type of impairment. A large percentage (80.2% [349/435]) had 1 or more impairments related to ADD/ADHD and learning disability. Of the applicants, 51.9% (69 797/134 397) applied with a single set of MCAT scores. The mean number of applications submitted to medical schools was 14.34.

There were no differences between MCAT section or total scores for applicants who tested with standard vs extra time (total: 28.3 vs 28.3; mean difference, 0.03 [95% CI, −0.49 to 0.55]; P = .92; d = 0.00; Biological Sciences: 9.9 vs 9.9; mean difference, −0.02 [95% CI, −0.22 to 0.18]; P = .84; d = −0.00; Physical Sciences: 9.4 vs 9.5; mean difference, −0.01 [95% CI, −0.23 to 0.21]; P = .93; d = −0.04; Verbal Reasoning: 9.0 vs 9.0; mean difference, 0.06 [95% CI, −0.14 to 0.26]; P = .57; d = 0.00), but mean UGPAs were significantly lower for applicants who tested with extra time (3.45 vs 3.53; mean difference, 0.08 [95% CI, 0.05 to 0.11]; P < .001; d = 0.24).

Study Participants for Medical Student Outcomes Analyses

In total, 93.2% (76 711/82 343) of the medical students in the 2000-2004 entering classes applied with both MCAT scores and UGPAs and were included in the analyses about medical student outcomes. Descriptive statistics for these students are shown in Table 2.

Table Graphic Jump LocationTable 2.  Characteristics of Students in the 2000-2004 Entering Classes of Medical School

The vast majority of these students took the MCAT with standard time (99.4% [76 262/76 711]) compared with extra time (0.6% [449/76 711]). Of the students who tested with standard time, 51.9% (39 561/76 262) were male, and the mean age at application was 23.7 years. Of applicants who tested with extra time, 56.6% (254/449) were male, and the mean age was 25.7 years. Sample sizes for analyses of specific student outcomes ranged from 62 195 to 75 087 and varied because each analysis was conducted only on students for whom outcome data were available. Of the applicants, 59.1% (45 342/76 711) applied with a single set of MCAT scores.

There were no differences between MCAT section or total scores for students who tested with standard vs extra time (total: 29.7 vs 29.5; mean difference, 0.23 [95% CI, −0.17 to 0.62]; P = .26; d = 0.05; Biological Sciences: 10.2 vs 10.2; mean difference, 0.06 [95% CI, −0.09 to 0.21]; P = .46; d = 0.00; Physical Sciences: 10.0 vs 10.0; mean difference, 0.02 [95% CI, −0.16 to 0.21]; P = .85; d = 0.00; Verbal Reasoning: 9.6 vs 9.4; mean difference, 0.14 [95% CI, −0.02 to 0.31]; P = .08; d = 0.11), but mean UGPAs were significantly lower for students who tested with extra time (3.44 vs 3.61; mean difference, 0.17 [95% CI, 0.14 to 0.20]; P < .001; d = 0.59).

Admissions Outcomes

The number and percentage of medical students who received acceptance offers are summarized separately for students who tested with standard vs extra time in Table 3. In total, 43.9% (191/435) of applicants who tested with extra time in 2011, 2012, or 2013 were accepted into medical school, compared with 44.5% (59 585/133 962) of those who tested with standard time (difference, 0.6% [95% CI, −4.1 to 5.3]; P = .81; h = 0.01).

Table Graphic Jump LocationTable 3.  Success Rates on Admissions Decisions, USMLE Step Examinations, and Medical School Graduation

Within-school regression analyses predicting acceptance offers from MCAT total scores and UGPAs for applicants who took the MCAT with standard time were conducted on application data from 140 medical schools and included a total of 1 920 155 applications from 133 962 applicants; applicants applied to a mean (SD) of 14.1 (11.23) medical schools. The median and interquartile ranges (IQRs) of the C statistics and G statistic P values for these analyses are summarized in Table 4. Across the 140 schools, the median C statistic was 0.77 (IQR, 0.72-0.81), indicating modest discriminative ability, and the median G statistic P value was P < .001 (IQR, P < .001-P < .001).

Table Graphic Jump LocationTable 4.  Discrimination and Goodness of Fit for Regression Models

A comparison of prediction error for applicants testing with standard vs extra time is shown in Table 5. Applicants taking the MCAT with extra time (n = 435) submitted 7853 applications to 138 of 140 medical schools in the study; applicants applied to a mean (SD) of 18.1 (13.20) medical schools. The total number of observed acceptance offers was 379, while the number of offers predicted based on the logistic regression analyses was 383.5, a difference of 4.5, or 0.1% (4.5/7853; [95% CI, −0.5% to 0.6%]; P = .83). Differences in prediction error for applicants who tested with standard vs extra time resulted in a d value of 0, indicating that there was no difference in acceptance decisions between the 2 groups after controlling for MCAT total scores and UGPAs.

Table Graphic Jump LocationTable 5.  Predicted vs Observed Success Rates for Students Testing With Standard Time vs Extra Time on the MCATa
Medical Student Performance Outcomes

The number and percentage of medical students who succeeded on outcomes related to USMLE Step examinations and graduation from medical school are summarized separately for students who tested with standard vs extra time in Table 3. Students who took the MCAT with extra time passed the USMLE Step examinations on their initial attempt at significantly lower rates than students who took the MCAT with standard time. For Step 1, the comparison was 82.1% (344/419) vs 94.0% (70 188/74 668) (difference, 11.9% [95% CI, 9.6% to 14.2%]; P < .001; h = 0.38). For Step 2 CK, the comparison was 85.5% (349/408) vs 95.4% (70 476/73 866) (difference, 9.9% [95% CI, 7.8% to 11.9%]; P < .001; h = 0.35). For Step 2 CS, the comparison was 92.0% (288/313) vs 97.0% (60 039/61 882) (difference, 5.0% [95% CI, 3.1% to 6.9%]; P < .001; h = 0.23). For the eventual outcome on each of the Step examinations, between 0.5% and 1.5% fewer of the students who took the MCAT with extra time than standard time passed the USMLE Step examinations eventually (Table 3).

Medical students who tested with extra time also graduated at significantly lower rates than students who tested with standard time. For example, the comparison for graduation within 4 years was 67.2% (285/424) vs 86.1% (60 547/70 305) (difference, 18.9% [95% CI, 15.6% to 22.2%]; P < .001; h = 0.46). For graduation within 5 years, the comparison was 81.6% (346/424) vs 94.4% (66 369/70 305) (difference, 12.8% [95% CI, 10.6% to 15.0%]; P < .001; h = 0.41). More than 10% (45/424) of students who tested with extra time did not graduate within the time frame of the study, compared with approximately 3% (2320/70 305) of students who tested with standard time (difference, 7.3% [95% CI, 5.6% to 9.0%]; P < .001; h = −0.30).

The results of the within-school regression analyses predicting medical student outcomes from MCAT total scores and UGPAs for students who took the MCAT with standard time are shown in Table 4. Across the different outcomes, the number of medical schools included in the analyses ranged from 47 to 126 and varied because analyses were not conducted on any outcomes for which all students in a medical school who took the MCAT with standard time succeeded. Median and IQR ranges for the C statistics were greater than 0.5 for all outcomes, but the discriminative ability of the models were greater for passing Step 1 and Step 2 CK on the first attempt (Step 1, median, 0.77 [IQR, 0.72-0.80]; Step 2 CK, median, 0.72 [IQR, 0.66-0.77]) or eventually (Step 1, median, 0.82 [IQR, 0.72-0.91]; Step 2 CK, median, 0.80 [IQR, 0.70-0.89]) and for passing Step 2 CS eventually (median, 0.84 [IQR, 0.73-0.94]). Median goodness of fit was greater for the outcomes passing Step 1 and Step 2 CK on the first attempt (passed on first attempt: Step 1, median, P < .001 [IQR, P < .001-P < .001]; Step 2 CK, median, P < .001 [IQR, P = .03-P < .001]) and graduating within 4 or 5 years of matriculation (graduating within 4 years, median, P = .003 [IQR, P = .13-P < .001]; 5 years, median, P = .03 [IQR, P = .24-P = .003]).

A comparison of prediction error for medical students testing with standard vs extra time is shown in Table 5. MCAT total scores and UGPAs overpredicted the likelihood of passing the USMLE Step examinations for medical students who took the MCAT with extra time; observed success rates for these students were significantly less than the predicted success rates. The overprediction rate for passing the Step 1 examination on the first attempt was 10.6% (44.6/419 [95% CI, 8.5% to 12.8%]; P < .001); for Step 2 CK, 8.4% (34.3/408 [95% CI, 6.4%-10.4%]; P < .001); and for Step 2 CS, 4.5% (14.1/312 [95% CI, 2.6% to 6.4%]; P < .001). Overprediction rates for eventually passing the Step examinations were between 0.3% and 1.7% (Table 5). d values associated with the differences in prediction error for students who tested with standard vs extra time ranged from 0.27 to 0.47 for the first attempt outcomes and 0.06 to 0.28 for the eventual outcomes, representing small effect sizes17 for all results except for eventually passing Step 1 and Step 2 CS.

MCAT total scores and UGPAs also overpredicted graduation outcomes of students who tested with extra time. The amount of overprediction ranged from 17.0% (71.9/424 [95% CI, 13.8% to 20.1%]; P < .001) for graduating within 4 years to 7.7% (32.7/424 [95% CI, 6.0% to 9.4%]; P < .001) for graduating within 8 years, twice the number of years associated with a typical 4-year undergraduate medical education program. d values associated with the differences in prediction error for students who tested with standard vs extra time ranged from 0.42 to 0.53 for the graduation outcomes, representing small to medium effect sizes.17

This study investigated the rate of admission to medical school, medical student performance, and time to graduation for individuals who took the MCAT with standard vs extra administration time. Mean MCAT scores and rates of acceptance to medical school were not significantly different for applicants who had MCAT scores obtained with standard vs extra time, yet students who tested with extra time performed less well on the USMLE Step examinations and graduated at significantly lower rates at various times. They passed their USMLE Step examinations at lower rates on the first attempt and eventually and were less likely to graduate within 4 to 8 years after graduation compared with students who had MCAT scores obtained with standard time.

These performance differences persisted after controlling for MCAT scores and UGPAs; MCAT scores and UGPAs overpredicted students’ academic performance in medical school. Research has shown that MCAT scores obtained with extra time overpredicted Step 1 scores.18 The current study extends these findings, showing that MCAT scores obtained with extra time overpredicted students’ first and eventual pass/fail outcomes on the USMLE Step 1, Step 2 CK, and Step 2 CS examinations and likelihood of graduating from medical school within 4 to 8 years after graduation. Other educational testing programs have reported similar results.19,20

These findings should be considered in light of changes to the MCAT21,22 that take effect in 2015, which include testing a broader set of knowledge and skills than the prior MCAT did. In addition to testing information processing skills and knowledge of biology, chemistry, and physics, the new MCAT will also test knowledge of biochemistry, psychology, and sociology, with an increased emphasis on scientific reasoning skills. However, the expanded focus of the new MCAT may not reduce the overprediction of future performance for medical students who take the MCAT with extra time.

The MCAT redesign drew from a wide evidence base, including research about the effect of additional testing time on scores from standardized tests. Research on the MCAT has shown that extra time is beneficial for most examinees.23 As a result of these and subsequent analyses, and in accordance with professional testing standards,5(p90) the redesigned MCAT will increase the amount of working time per question by between 13% and 20% per section (S. H. Oppler, PhD, Vice President, Certification Research and Analysis, Society for Human Resources Management, oral communication, November 2014) to reduce potential time barriers that may make it difficult for examinees to demonstrate their proficiency. Providing more working time may decrease the need for extra testing time and reduce the differences in predictive meaning between scores obtained under standard vs extra time conditions.

Even with these changes, more research is needed to carefully tailor the amount of extra time needed by individuals with disabilities who take the MCAT. Currently, it is common for individuals with disabilities to receive 50% or 100% additional time on standardized examinations.24 In addition to these extra-time accommodations, in 2007 the AAMC began offering 25% extra time in response to research suggesting that a smaller amount of additional time may be sufficient in some cases.25 Future research should compare the amount of overprediction for scores obtained with 25%, 50%, and 100% extra time to assess whether a smaller amount of extra time results in MCAT scores that are more comparable with scores obtained with standard testing time, which would support the practice of more carefully tailoring the amount of extra time provided to the needs of examinees.

The poorer performance on the USMLE Step examinations, and the longer time needed to graduate from medical school for individuals who have received extra testing time on the MCAT, also suggest that medical schools should examine their learning environments and support systems for individuals with disabilities. Medical school educators and administrators need a better understanding of potential barriers to medical education for students with disabilities in order to develop evidence-based policies, procedures, and resources.2629 Research on the performance of medical students with disabilities at one medical school found that although most of the students with disabilities at that school performed well academically and clinically, overall, medical students with disabilities performed less well than students without disabilities on measures of academic performance and were less likely to graduate.30 This research also found that, for this institution’s medical students with disabilities, the magnitude of differences in clinical performance varied by type of disability. More research is needed to understand the interplay between type of disability and performance differences to improve the medical education process for students with disabilities.3133

The limitations of this study include lack of access to other admissions data that might influence the results of the multivariable analysis predicting medical student outcomes from admissions data, as well as information about the accommodations provided on the USMLE Step examinations and how they compare with MCAT accommodations for each individual. Admissions decisions are based on a wide range of other academic, experiential, and demographic/biographic information15 not examined in the present study. Including these variables might have improved the ability to discriminate successful from unsuccessful performance on more distal medical student outcomes such as graduation within 5 to 8 years. We also do not know whether or how each medical school weighed the annotation on the MCAT score report that an applicant tested under nonstandard conditions in making admissions decisions. We also lacked complete data on disability status and the amount of extra time received on the MCAT for medical students entering from 2000-2004, information that might allow for a more careful consideration of the disability and accommodation types that lead to more accurate prediction of medical student performance.

Among applicants to US medical schools, those with MCAT scores obtained with extra test administration time, compared with standard administration time, had no significant difference in rate of medical school admission but had lower rates of passing the USMLE Step examinations and of medical school graduation within 4 to 8 years after matriculation. These findings raise questions about the types of learning environments and support systems needed by students who test with extra time on the MCAT to enable them to succeed in medical school.

Corresponding Author: Cynthia A. Searcy, PhD, MCAT Research and Development, Association of American Medical Colleges, 655 K St, NW, Ste 100, Washington, DC 20001 (csearcy@aamc.org).

Author Contributions: Dr Searcy and Mr Dowd had full access to all of the data in the study and take responsibility for the integrity of the data and the accuracy of the data analysis.

Study concept and design: Searcy, Hughes.

Acquisition, analysis, or interpretation of data: Searcy, Dowd, Hughes, Baldwin, Pigg.

Drafting of the manuscript: Searcy, Hughes, Baldwin.

Critical revision of the manuscript for important intellectual content: Searcy, Dowd, Hughes, Pigg.

Statistical analysis: Dowd, Hughes, Baldwin, Pigg.

Administrative, technical, or material support: Searcy, Dowd.

Study supervision: Searcy.

Conflict of Interest Disclosures: All authors have completed and submitted the ICMJE Form for Disclosure of Potential Conflicts of Interest. Drs Hughes and Baldwin reported that their organization (Human Resources Research Organization) received funds from the Association of American Medical Colleges as consulting fees to conduct analyses for this research in collaboration with Dr Searcy, Mr Dowd, and Mr Pigg. No other disclosures were reported.

Additional Contributions: We thank the National Board of Medical Examiners (NBME) and the following personnel for reviewing earlier drafts of the manuscript: Heather Alarcon, JD; Gabrielle Campbell, MBA, LLM; Atul Grover, MD, PhD; Robert Jones, PhD; Karen Mitchell, PhD; Scott Oppler, PhD; Elisa Siegal, AB; Frank Trinity, JD; and Geoffrey Young, PhD, all with the Association of American Medical Colleges. We also thank the following external reviewers for their review of the manuscript: Robert Linn, PhD, University of Colorado at Boulder; Paul Sackett, PhD, University of Minnesota; and Lauress Wise, PhD, Human Resources Research Organization. None of these persons was compensated for their contributions. Dr Wise’s organization (Human Resources Research Organization) received funds from the Association of American Medical Colleges as consulting fees to conduct analyses for this research.

Julian  ER.  Validity of the Medical College Admission Test for predicting medical school performance. Acad Med. 2005;80(10):910-917.
PubMed   |  Link to Article
Dunleavy  DM, Kroopnick  MH, Dowd  KW, Searcy  CA, Zhao  X.  The predictive validity of the MCAT exam in relation to academic performance through medical school: a national cohort study of 2001-2004 matriculants. Acad Med. 2013;88(5):666-671.
PubMed   |  Link to Article
Callahan  CA, Hojat  M, Veloski  J, Erdmann  JB, Gonnella  JS.  The predictive validity of three versions of the MCAT in relation to performance in medical school, residency, and licensing examinations: a longitudinal study of 36 classes of Jefferson Medical College. Acad Med. 2010;85(6):980-987.
PubMed   |  Link to Article
Donnon  T, Paolucci  EO, Violato  C.  The predictive validity of the MCAT for medical school performance and medical board licensing examinations: a meta-analysis of the published research. Acad Med. 2007;82(1):100-106.
PubMed   |  Link to Article
American Educational Research Association, American Psychological Association, National Council on Measurement in Education. Standards for Educational and Psychological Testing. Washington, DC: American Educational Research Association; 2014.
Americans with Disabilities Act of 1990, 42 USC §12101 et seq.
Thompson  S, Blount  A, Thurlow  M. A summary of research on the effects of test accommodations: 1999 through 2001 [technical report 34]. University of Minnesota, National Center on Educational Outcomes. http://www.cehd.umn.edu/nceo/onlinepubs/Technical34.htm. Accessed May 14, 2015.
Thurlow  ML, Thompson  SJ, Lazarus  SS. Considerations for the administration of tests to special needs students: accommodations, modifications, and more. In: Downing  SM, Haladyna  TM, eds. Handbook of Test Development. Mahwah, NJ: Lawrence Erlbaum; 2006:653-673.
Ballard  R, Elwork  A.  Learning disability and professional licensing examinations: what accommodations are reasonable under the ADA? J Psychiatry Law. 2003;31(1):43-66.
Hollenbeck  K. Determining when test alterations are valid accommodations or modifications for large-scale assessment. In: Tindal  G, Haladyna  TM, eds. Large-scale Assessment Programs for All Students: Validity, Technical Adequacy, and Implementation. Mahwah, NJ: Lawrence Erlbaum Associates; 2002.
Zuriff  GE.  Extra examination time for students with learning disabilities: an examination of the maximum potential thesis. Appl Meas Educ. 2000;13(1):99-117.
Link to Article
Bolt  S, Thurlow  M.  Five of the most frequently allowed testing accommodations in state policy: synthesis of research. Remedial Spec Educ. 2004;25(3):141-152.
Link to Article
Stretch  L, Osborne  J.  Extended time test accommodation: directions for future research and practice. Pract Assess Res Eval. 2005;10(7):1-8.
Julian  ER, Ingersoll  DJ, Etienne  PM, Hilger  AE.  The impact of testing accommodations on MCAT scores: descriptive results. Acad Med. 2004;79(4):360-364.
PubMed   |  Link to Article
Monroe  A, Quinn  E, Samuelson  W, Dunleavy  DM, Dowd  KW.  An overview of the medical school admission process and use of applicant data in decision making: what has changed since the 1980s? Acad Med. 2013;88(5):672-681.
PubMed   |  Link to Article
Hosmer  D, Lemeshow  S. Applied Logistic Regression. New York, NY: John Wiley & Sons; 2000.
Cohen  J.  A power primer. Psychol Bull. 1992;112(1):155-159.
PubMed   |  Link to Article
Oppler  S, Mueller  L, Dunleavy  E, Mitchell  K, Dunleavy  D. A comparison of MCAT validity across standard and accommodated administrations. Paper presented at: 25th Annual Conference of the Society for Industrial and Organizational Psychology; April 8-10, 2010; Atlanta, GA.
Amodeo  A, Marcus  L, Thornton  A, Pashley  P. Predictive validity of accommodated LSAT scores for 2002-2006 entering law school classes [LSAC technical report 09-01]. Law School Admission Council. http://www.lsac.org/docs/default-source/research-%28lsac-resources%29/tr-09-01.pdf. Accessed May 14, 2015.
Cahalan  C, Mandinach  E, Camara  W. Predictive validity of SAT I: reasoning test for test-takers with learning disabilities and extended time accommodations [research report No. 2002-5]. The College Board. http://www.ets.org/Media/Research/pdf/RR-02-11-Cahalan.pdf. Accessed May 14, 2015.
Schwartzstein  RM, Rosenfeld  GC, Hilborn  R, Oyewole  SH, Mitchell  K.  Redesigning the MCAT exam: balancing multiple perspectives. Acad Med. 2013;88(5):560-567.
PubMed   |  Link to Article
Mann  S. AAMC approves new MCAT exam with increased focus on social, behavioral sciences. Association of American Medical Colleges. https://www.aamc.org/newsroom/reporter/march2012/276588/mcat2015.html. Accessed November 24, 2014.
Oppler  S, Davies  S, Lyons  B, Nathanson  L. The effect of speededness on MCAT scores: an initial examination. Paper presented at: 19th Annual Conference of the Society for Industrial and Organizational Psychology; April 2-4, 2004; Chicago, IL.
Lewandowski  L, Cohen  J, Lovett  B.  Effects of extended time allotments on reading comprehension performance of college students with and without learning disabilities. J Psychoed Assess. 2013;31(3):326-336.
Link to Article
Mandinach  E, Bridgeman  B, Cahalan-Laitusis  C, Trapani  C. The impact of extended time on SAT test performance [research report No. 2005-8]. The College Board. http://www.ets.org/Media/Research/pdf/RR-05-20.pdf. Accessed May 14, 2015.
Altchuler  SI.  Commentary: granting medical licensure, honoring the Americans with Disabilities Act, and protecting the public: can we do all three? Acad Med. 2009;84(6):689-691.
PubMed   |  Link to Article
Smith  WT, Allen  WL.  Implications of the 2008 amendments to the Americans with Disabilities Act for medical education. Acad Med. 2011;86(6):768-772.
PubMed   |  Link to Article
Melnick  DE.  Commentary: balancing responsibility to patients and responsibility to aspiring physicians with disabilities. Acad Med. 2011;86(6):674-676.
PubMed   |  Link to Article
DeLisa  J, Silverstein  R, Thomas  P.  Commentary: more implications of the 2008 amendments to the Americans with Disabilities Act: influencing institutional policies, practices, and procedures. Acad Med. 2011;86(6):677-679.
PubMed   |  Link to Article
Teherani  A, Papadakis  MA.  Clinical performance of medical students with protected disabilities. JAMA. 2013;310(21):2309-2311.
PubMed   |  Link to Article
Watson  J, Hutchens  S. Medical Students With Disabilities: A Generation of Practice. Washington, DC: Association of American Medical Colleges; 2005.
Latham  P, Hosterman  J. Medical Students With Disabilities: Resources to Enhance Accessibility. Washington, DC: Association of American Medical Colleges; 2010.
Group on Diversity and Inclusion. GDI Navigator to Excellence: Summaries of Disability Articles in the Journal of Academic Medicine 2001-2012. Washington, DC: Association of American Medical Colleges; 2013.

Figures

Tables

Table Graphic Jump LocationTable 1.  Characteristics of Applicants to the 2011-2013 Entering Classes of Medical School
Table Graphic Jump LocationTable 2.  Characteristics of Students in the 2000-2004 Entering Classes of Medical School
Table Graphic Jump LocationTable 3.  Success Rates on Admissions Decisions, USMLE Step Examinations, and Medical School Graduation
Table Graphic Jump LocationTable 4.  Discrimination and Goodness of Fit for Regression Models
Table Graphic Jump LocationTable 5.  Predicted vs Observed Success Rates for Students Testing With Standard Time vs Extra Time on the MCATa

References

Julian  ER.  Validity of the Medical College Admission Test for predicting medical school performance. Acad Med. 2005;80(10):910-917.
PubMed   |  Link to Article
Dunleavy  DM, Kroopnick  MH, Dowd  KW, Searcy  CA, Zhao  X.  The predictive validity of the MCAT exam in relation to academic performance through medical school: a national cohort study of 2001-2004 matriculants. Acad Med. 2013;88(5):666-671.
PubMed   |  Link to Article
Callahan  CA, Hojat  M, Veloski  J, Erdmann  JB, Gonnella  JS.  The predictive validity of three versions of the MCAT in relation to performance in medical school, residency, and licensing examinations: a longitudinal study of 36 classes of Jefferson Medical College. Acad Med. 2010;85(6):980-987.
PubMed   |  Link to Article
Donnon  T, Paolucci  EO, Violato  C.  The predictive validity of the MCAT for medical school performance and medical board licensing examinations: a meta-analysis of the published research. Acad Med. 2007;82(1):100-106.
PubMed   |  Link to Article
American Educational Research Association, American Psychological Association, National Council on Measurement in Education. Standards for Educational and Psychological Testing. Washington, DC: American Educational Research Association; 2014.
Americans with Disabilities Act of 1990, 42 USC §12101 et seq.
Thompson  S, Blount  A, Thurlow  M. A summary of research on the effects of test accommodations: 1999 through 2001 [technical report 34]. University of Minnesota, National Center on Educational Outcomes. http://www.cehd.umn.edu/nceo/onlinepubs/Technical34.htm. Accessed May 14, 2015.
Thurlow  ML, Thompson  SJ, Lazarus  SS. Considerations for the administration of tests to special needs students: accommodations, modifications, and more. In: Downing  SM, Haladyna  TM, eds. Handbook of Test Development. Mahwah, NJ: Lawrence Erlbaum; 2006:653-673.
Ballard  R, Elwork  A.  Learning disability and professional licensing examinations: what accommodations are reasonable under the ADA? J Psychiatry Law. 2003;31(1):43-66.
Hollenbeck  K. Determining when test alterations are valid accommodations or modifications for large-scale assessment. In: Tindal  G, Haladyna  TM, eds. Large-scale Assessment Programs for All Students: Validity, Technical Adequacy, and Implementation. Mahwah, NJ: Lawrence Erlbaum Associates; 2002.
Zuriff  GE.  Extra examination time for students with learning disabilities: an examination of the maximum potential thesis. Appl Meas Educ. 2000;13(1):99-117.
Link to Article
Bolt  S, Thurlow  M.  Five of the most frequently allowed testing accommodations in state policy: synthesis of research. Remedial Spec Educ. 2004;25(3):141-152.
Link to Article
Stretch  L, Osborne  J.  Extended time test accommodation: directions for future research and practice. Pract Assess Res Eval. 2005;10(7):1-8.
Julian  ER, Ingersoll  DJ, Etienne  PM, Hilger  AE.  The impact of testing accommodations on MCAT scores: descriptive results. Acad Med. 2004;79(4):360-364.
PubMed   |  Link to Article
Monroe  A, Quinn  E, Samuelson  W, Dunleavy  DM, Dowd  KW.  An overview of the medical school admission process and use of applicant data in decision making: what has changed since the 1980s? Acad Med. 2013;88(5):672-681.
PubMed   |  Link to Article
Hosmer  D, Lemeshow  S. Applied Logistic Regression. New York, NY: John Wiley & Sons; 2000.
Cohen  J.  A power primer. Psychol Bull. 1992;112(1):155-159.
PubMed   |  Link to Article
Oppler  S, Mueller  L, Dunleavy  E, Mitchell  K, Dunleavy  D. A comparison of MCAT validity across standard and accommodated administrations. Paper presented at: 25th Annual Conference of the Society for Industrial and Organizational Psychology; April 8-10, 2010; Atlanta, GA.
Amodeo  A, Marcus  L, Thornton  A, Pashley  P. Predictive validity of accommodated LSAT scores for 2002-2006 entering law school classes [LSAC technical report 09-01]. Law School Admission Council. http://www.lsac.org/docs/default-source/research-%28lsac-resources%29/tr-09-01.pdf. Accessed May 14, 2015.
Cahalan  C, Mandinach  E, Camara  W. Predictive validity of SAT I: reasoning test for test-takers with learning disabilities and extended time accommodations [research report No. 2002-5]. The College Board. http://www.ets.org/Media/Research/pdf/RR-02-11-Cahalan.pdf. Accessed May 14, 2015.
Schwartzstein  RM, Rosenfeld  GC, Hilborn  R, Oyewole  SH, Mitchell  K.  Redesigning the MCAT exam: balancing multiple perspectives. Acad Med. 2013;88(5):560-567.
PubMed   |  Link to Article
Mann  S. AAMC approves new MCAT exam with increased focus on social, behavioral sciences. Association of American Medical Colleges. https://www.aamc.org/newsroom/reporter/march2012/276588/mcat2015.html. Accessed November 24, 2014.
Oppler  S, Davies  S, Lyons  B, Nathanson  L. The effect of speededness on MCAT scores: an initial examination. Paper presented at: 19th Annual Conference of the Society for Industrial and Organizational Psychology; April 2-4, 2004; Chicago, IL.
Lewandowski  L, Cohen  J, Lovett  B.  Effects of extended time allotments on reading comprehension performance of college students with and without learning disabilities. J Psychoed Assess. 2013;31(3):326-336.
Link to Article
Mandinach  E, Bridgeman  B, Cahalan-Laitusis  C, Trapani  C. The impact of extended time on SAT test performance [research report No. 2005-8]. The College Board. http://www.ets.org/Media/Research/pdf/RR-05-20.pdf. Accessed May 14, 2015.
Altchuler  SI.  Commentary: granting medical licensure, honoring the Americans with Disabilities Act, and protecting the public: can we do all three? Acad Med. 2009;84(6):689-691.
PubMed   |  Link to Article
Smith  WT, Allen  WL.  Implications of the 2008 amendments to the Americans with Disabilities Act for medical education. Acad Med. 2011;86(6):768-772.
PubMed   |  Link to Article
Melnick  DE.  Commentary: balancing responsibility to patients and responsibility to aspiring physicians with disabilities. Acad Med. 2011;86(6):674-676.
PubMed   |  Link to Article
DeLisa  J, Silverstein  R, Thomas  P.  Commentary: more implications of the 2008 amendments to the Americans with Disabilities Act: influencing institutional policies, practices, and procedures. Acad Med. 2011;86(6):677-679.
PubMed   |  Link to Article
Teherani  A, Papadakis  MA.  Clinical performance of medical students with protected disabilities. JAMA. 2013;310(21):2309-2311.
PubMed   |  Link to Article
Watson  J, Hutchens  S. Medical Students With Disabilities: A Generation of Practice. Washington, DC: Association of American Medical Colleges; 2005.
Latham  P, Hosterman  J. Medical Students With Disabilities: Resources to Enhance Accessibility. Washington, DC: Association of American Medical Colleges; 2010.
Group on Diversity and Inclusion. GDI Navigator to Excellence: Summaries of Disability Articles in the Journal of Academic Medicine 2001-2012. Washington, DC: Association of American Medical Colleges; 2013.
CME
Also Meets CME requirements for:
Browse CME for all U.S. States
Accreditation Information
The American Medical Association is accredited by the Accreditation Council for Continuing Medical Education to provide continuing medical education for physicians. The AMA designates this journal-based CME activity for a maximum of 1 AMA PRA Category 1 CreditTM per course. Physicians should claim only the credit commensurate with the extent of their participation in the activity. Physicians who complete the CME course and score at least 80% correct on the quiz are eligible for AMA PRA Category 1 CreditTM.
Note: You must get at least of the answers correct to pass this quiz.
Please click the checkbox indicating that you have read the full article in order to submit your answers.
Your answers have been saved for later.
You have not filled in all the answers to complete this quiz
The following questions were not answered:
Sorry, you have unsuccessfully completed this CME quiz with a score of
The following questions were not answered correctly:
Commitment to Change (optional):
Indicate what change(s) you will implement in your practice, if any, based on this CME course.
Your quiz results:
The filled radio buttons indicate your responses. The preferred responses are highlighted
For CME Course: A Proposed Model for Initial Assessment and Management of Acute Heart Failure Syndromes
Indicate what changes(s) you will implement in your practice, if any, based on this CME course.

Multimedia

Some tools below are only available to our subscribers or users with an online account.

5,893 Views
1 Citations
×

Related Content

Customize your page view by dragging & repositioning the boxes below.

See Also...
Articles Related By Topic
Related Collections
PubMed Articles
Jobs