Tools for Direct Observation and Assessment of Clinical Skills of Medical Trainees:  A Systematic Review FREE

Direct observation of medical trainees with actual patients by clinical supervisors is critical for teaching and assessing clinical and communication skills. A recent Institute of Medicine report calls for improved supervision of trainees to enhance patient safety and quality of clinical education.¹ The Liaison Committee on Medical Education and Accreditation Council for Graduate Medical Education require ongoing assessment that includes direct observation of trainees' clinical skills.^2- 3 By observing and assessing learners with patients and providing feedback, faculty help trainees to acquire and improve skills and help patients through better supervision of clinical care.⁴

Direct observation of medical trainees occurs infrequently and inadequately.^5- 6 End-of-rotation global rating forms are often completed by supervisors who have not directly observed trainees with patients.⁷ However, assessment based on direct observation should be an essential component of outcomes-based education and certification.^8- 9 With current interest in establishing an outcomes-based medical education system that enhances trainee development and patient safety, there is a great need for robust work-based evaluation tools. To our knowledge, a rigorous systematic review has not been performed of the utility and quality of the numerous existing tools for direct observation and assessment of medical trainees with actual patients. We therefore systematically reviewed the literature to determine available tools for direct observation by supervisors of trainees' clinical skills with actual patients. The aim was to describe existing tools and the evidence of their validity and outcomes to provide medical educators with evidence-based assessment measures and an understanding of areas for further research.

A systematic literature search was conducted using specific eligibility criteria, electronic searching, and hand searching to minimize risk of bias in selecting articles. The search, conducted with the assistance of a library science expert, included relevant English-language studies published between January 1965 and March 2009 using the PubMed, Education Resource Information Center (ERIC), Cumulative Index to Nursing and Allied Health Literature (CINAHL), and Web of Science electronic literature databases. Combinations of terms were used related to competence (clinical competence; clinical skills), medical education (education; students, education, medical; clinical clerkship, internship and residency/methods; preceptorship), and learner level (student; intern; resident). Tables of contents of medical education journals not indexed in PubMed (Teaching and Learning in Medicine, 1986-1996; Medical Teacher, 1979-1980) were hand-searched. The reference lists of all included articles and identified review articles were examined. A key word search of instruments identified in the included articles was conducted. A more detailed search strategy is available on request.

Studies were included if they described a tool designed (1) for direct observation of skills in clinical settings with actual patients (observer in the room or observing by remote camera) and (2) for use by educational supervisors (interns, residents, fellows, faculty, nurses, nurse practitioners, other trained observers) with medical trainees (medical students, interns, residents, fellows). Studies were excluded that described tools intended (1) for use with standardized patients, (2) for use in simulated settings (eg, without actual patients), or (3) to assess surgical or procedural skills; and (4) without a full article available for review.

The initial search identified 10 672 citations (Figure). All 3 authors independently reviewed citation titles and abstracts to assess eligibility for review, with each title/abstract reviewed by at least 2 authors. Of those, 199 were appropriate for detailed review to determine if they met inclusion criteria. Review articles were excluded. When reviewers disagreed or an abstract was insufficient to determine study eligibility, the full article was retrieved.

A Best Evidence in Medical Education abstraction form¹⁰ was modified to focus on the settings, learners, tool content, and outcomes described in studies. Every article was independently abstracted by 2 authors (J.R.K. and K.E.H.). Each reviewer then reconciled half of the abstractions for completeness and accuracy. Differences in data abstraction were resolved through consensus adjudication. Extracted information included tool characteristics and implementation, validity, and outcomes. Abstracted items characterized tool characteristics (assessed skills, number of items and how they were evaluated, space for open-ended comments or action plan) and implementation (research study design,¹¹ setting [country, single/multi-institution, specialty, inpatient/outpatient, trainee level], observer characteristics, use for formative/summative evaluation).

Information on reliability and validity was extracted. Although many frameworks to evaluate assessment tools exist,^12- 14 the unitary theory of Messick¹³ was used. In this approach, validity evidence is used to support the overarching framework of construct validity, the degree to which an assessment measures the underlying construct.^13,15- 16 Validity evidence was sought in 5 areas:

A modified version of Kirkpatrick's hierarchy was used to evaluate outcomes of implementing a tool.¹⁷ Outcome levels abstracted included:

Information regarding cost of tool development and implementation was also extracted.¹⁸

Due to study heterogeneity, a meta-analysis was not possible. After ascertaining tools used for direct observation, we specifically identified those with evidence of internal structure validity and validity based on relationship to other variables. We determined whether these tools had an educational outcome beyond learners' or observers' attitudes about the tool or its implementation.

The Figure summarizes the results of the review process. Of 10 672 citations, 85 met inclusion criteria after title, abstract, and full article review. Fifty-five unique tools were identified. The 85 studies were heterogeneous in their populations, methods, and outcomes (Table 1). The most common study design was a prospective cohort without a comparison group. Randomized controlled trials were used in 6 studies in internal medicine,^19- 24 1 in pediatrics,²⁵ and 1 in an unspecified discipline.²⁶ Of the studies, 64 (75%) occurred within single institutions. Twenty-seven studies mentioned institutional review board approval.^{20- 24,27- 48} Costs of tool implementation, mentioned infrequently,^{37,39,49- 57} usually focused on faculty time. One article specifically mentioned administrative costs⁵⁶ but none included cost calculations. eTable 1 presents additional information about the characteristics of each study (objective, design, country, learner, specialty, observation location, assessment type [formative/summative], and how observations of trainees occurred).

Details about each of the tools are provided in Table 2A and Table 2B. Of the 55 unique tools identified, 21 (38%) were implemented with students, 32 (58%) with residents or fellows, and 2 (the Mini Clinical Evaluation Exercise [Mini-CEX] and 1 unnamed⁵⁸) with both. The largest number of tools (17) were developed or tested in internal medicine settings. The Mini-CEX was the most studied, with adaptations for palliative care,³⁷ ophthalmology,^59- 60 and cardiology^41,61- 62 and implementation in multispecialty settings.⁶³ Most tools contained items on history taking, physical examination, and communication (eTable 2). Eleven tools (20%) contained scales with behavioral anchors.^{40,59- 60,64- 73} Twenty tools (36%) solicited open-ended comments or written action plans. Thirty-two tools (58%) were implemented for formative assessment, 7 (13%) for summative assessment, and 3 (5%) for both, although this distinction was not always clear (eTable 1). Many tools were used once per trainee, although some were used up to 10 times (eTable 1).

The frequency of reported validity evidence across tools is summarized in eTable 2. Table 2A and Table 2B describe whether validity was studied for each tool. Actual evidence by study is presented in eTable 3.

Descriptions of tool content selection (content validity) were mentioned for 20 tools (36%)^{20- 21,27,29- 30,33- 34,38- 40,52,56,59,74- 81} and typically involved expert or consensus groups reviewing educational competencies and literature.

Observers were infrequently trained to use assessment tools. Rater training, described for 47% of tools,^{19- 23,27- 29,33- 39,41- 42,44- 45,47,49- 51,55,61- 62,65- 66,70,73- 75,77,80,82- 87} usually occurred once and was brief (10 minutes to 3 hours). Training usually included orienting observers to the tool or discussing feedback principles via e-mail, workshops, or preexisting institutional faculty/resident lectures and meetings.^{19- 22,27- 29,33- 39,41- 42,44- 45,47,49- 50,55,61- 62,64- 66,70,75,77,80,82,85- 87} Training sessions that incorporated rater practice using the tool or review of videotaped performances of different competency levels were described for 8 tools.^{20,22- 23,34- 35,49,55,70,74,85} For 2 tools, observers were either given examples of effective feedback^21,85 or trained to provide feedback using role play.^23,49

Interrater reliability, reported for 22 tools (40%), was the most commonly reported reliability assessment^{19- 20,22,24- 25,28,30- 31,33- 34,39- 40,52,60,63,66,69,73,75,77,79,81,88- 93} and was often suboptimal (<0.70).⁹⁴ Intrarater reliability⁹³ and test-retest reliability⁸⁸ were reported for 1 tool each. Interitem correlations (correlations between items on the form) and item-total correlations (correlations between items and the overall rating) were reported for 2^{22,42,52,95- 96} and 4 tools,^{22,42,45,77,80,95- 96} respectively. Internal consistency was described for only 8 tools^{30,36,40- 42,45,60,68,75,77,97} but was usually high (Cronbach α approximately ≥0.70).⁹⁴ Generalizability/reproducibility coefficients were reported for 8 tools.^{22,28,42,47,61,63,66,69,73- 75,77,95- 96} Three studies, 1 describing the minicard and the other 2 a modified Mini-CEX, compared performance characteristics of 2 different tools.^20,22,48

Correlation of direct observation scores with other assessments was described for 17 tools (31%) in 22 studies.^{21,28,30,36,41- 42,52- 53,63,68- 69,73,75,79- 80,84,89,97- 101} Assessments were compared to written examination scores^{21,28,41- 42,73,75,84,89,97- 99,101} and clinical performance ratings.^{21,28,30,36,42,52- 53,69,84,89,97,99- 101} Comparisons with objective structured clinical examinations/standardized patient examinations,^{28,41,63,73,75,101} chart audits,⁷⁹ patient write-ups,^42,68 or patient ratings³⁰ were less common. In general, correlations were low (r = 0.1) or modest (r = 0.3).¹⁰² Correlations were disattenuated in 3 studies.^41,73,75

Performance scores were also compared across training level or other learner characteristics.^{24,28,35,38- 39,41- 42,51,58,61,69,72,80,83,92- 93,95- 97,103} Eight tools (10 studies) had scores that increased with training level^{35,38,42,51,58,61,69,80,83,95}; with 4 tools this trend was not seen.^51,72,83,97 The Mini-CEX had evidence both supporting^{24,41- 42,61,95- 96} and refuting⁹⁷ score improvement with training level. With 4 tools, learners' performance improved after clinical skills training and/or feedback.^{39,72,92- 93}

Surveying trainees and observers about their experiences with a tool was the most common method for assessing outcomes, used with 19 tools (35%).^{21,23,30,37,41- 42,44- 45,47,49- 50,54- 57,61- 62,65- 67,70,76,86- 89,93,95- 96,100,104} Trainees generally rated observation experiences positively.

Modification of trainees' self-assessed knowledge, attitudes, or skills was reported for 9 tools (16%).^{21,27,30,37,50,76,89,91,100,104} Transfer of trainee learning (objectively measured skill or behavior change) was described for 5 tools.^{25- 26,39,49,93} Studies describing these changes were often nonblinded and failed to control for baseline clinical skills.^26,39

Outcomes of tool implementation on observer feedback or the effect of observer training on rating behaviors was described for 6 tools.^{22- 23,27,49,56,70,88} Tool implementation increased the frequency,^27,56 specificity,^70,88 and timeliness⁷⁰ of observation and feedback. Training increased confidence using the tool^22- 23 but inconsistently improved rater stringency and accuracy.^22- 23

Organizational change was described for 2 tools (Modified Leicester Assessment Package⁶⁴; Patient Evaluation Assessment Form³⁸). For both, it was suggested that deficiencies identified on assessments inspired curricular change.^38,64 No tool had evidence that use affected patient care outcomes.

Eleven tools had evidence of internal structure validity and validity based on relationships to other variables. These included the Direct Observation Clinical Encounter Examination⁷⁵ (multispecialty), Clinical Encounter Card^27- 28 (surgery), Direct Psychiatric Clinical Examination⁸⁹ (psychiatry), Revised Infant Video Questionnaire³⁹ (pediatrics), a 360-degree evaluation described by Wood et al³⁰ (radiology), Davis Observation Code^79,101 (family medicine), Mini-CEX,^{41- 42,45,47,61,63,95- 97} and unnamed tools described by Woolliscroft et al (unspecified discipline),⁶⁸ Brennan and Norman⁷³ (obstetrics), Beckman et al⁹² (internal medicine), and Nørgaard et al⁸⁰ (internal medicine). Only 3 had evidence of learning. Use of the Revised Infant Video Questionnaire increased learning using a noncontrolled study design.³⁹ Residents self-assessed improved communication and counseling skills with a 360-degree evaluation.³⁰ Students reported improved understanding of their history-taking, physical examination, and decision-making skills using the Clinical Encounter Card.^27- 28

Direct observation of medical trainees by faculty remains a vital component of assessment across specialties. Assessment through observation provides ongoing data on trainee performance with actual patients, and effective assessment helps medical educators meet their professional obligation to self-regulate effectively.¹⁰⁵ Enhanced supervision (with observation) can be associated with better patient care and faster acquisition of clinical skills by trainees,¹⁰⁶ and the 2008 Institute of Medicine report recommends greater supervision in medical education to improve patient safety and education.¹ The development of expertise depends on accurate and detailed assessment and feedback.¹⁰⁷ However, faculty and training institutions may not be held accountable for ensuring trainees' clinical competence, and high-quality direct observation of trainees should augment the quality of supervision.¹⁰⁸

Although we identified many tools available for direct observation of clinical skills, few have been thoroughly evaluated and tested. One tool, the Mini-CEX, has been implemented repeatedly with medical students, residents, and fellows across specialties. The 20 Mini-CEX studies illustrate how validity evidence can accrue and tool implementation can be manipulated (ie, adding behavioral anchors to increase score reliability and accuracy).²⁰ Multiple publications suggest the validity of Mini-CEX scores. Ten other tools (Table 2A and Table 2B) possessing at least 2 levels of validity evidence have potential for wider use with additional research on implementation and consequential validity.^{27- 28,30,39,68,73,75,79- 80,89,92,101}

Although many studies measured trainees' or observers' attitudes about the observation process, few demonstrated improved clinical skills or patient care quality with tool implementation in an educational program. Outcomes such as learning, transfer of skills to new situations, or improved patient care are important and relatively unstudied. Whether these tools are associated with health care system improvements remains an area for future research.

In many studies, rater training (the response process component of validity) was minimally described or did not occur. Whether this omission was related to perceived cost, time constraints, or unawareness of the importance of rater training is unknown. However, observers need training to rate learners' performance reliably and discriminate between performance levels.⁸ Randomized trials highlight the value of rater training and its effect on scores.^22- 23 Brief training is likely to be ineffective.^{19,22- 23,77} Although rater training may initially be resource- and time-intensive, these costs should be weighed against potential benefits gained in teaching quality and learning.¹⁸ Given the relative inattention to implementation in the studies we reviewed, as well as the high expense associated with current assessment strategies such as simulation and standardized patient examinations, faculty development that enhances trainees' clinical skills and increases faculty supervision through observation could enhance care and may be cost-effective.

Our findings also suggest several next steps to improve the quality of research in this area. To enhance the quality of evidence in medical education, published research should include the assessment or intervention; methods of implementation; and evidence for reliability, validity, and educational outcomes.¹⁰⁶ However, current research generally does not adhere to these recommendations. After utility of a tool has been demonstrated (validity evidence) and guidelines for implementation developed, randomized study designs should follow whenever possible to assess whether the tool affects educational outcomes.^109- 110 More multi-institutional studies could help improve generalizability of findings. However, these larger, complex studies will require more resources, often lacking for educational research,¹¹¹ and may benefit from more streamlined institutional review board approval processes.¹¹²

A strength of this study is that the review included more than 10 000 abstracts and hand-searching of bibliographies from published studies. However, several limitations should be considered. Publication bias is possible; there are likely tools that have not been described in publications, although they may have relatively poor psychometric characteristics.¹¹³ The search strategy was limited to English-language studies and did not include unpublished abstracts from conference proceedings or nonindexed open-access journals. Although a library science expert assisted with the search, the lack of a specific Medical Subject Heading for direct observation and variability of terms used in the medical education literature may have limited the ability to identify all studies. The literature search may have missed relevant international studies because the search strategy did not include some terms commonly used in non-US countries (eg, registrar).

In conclusion, this systematic review identified and described a large number of tools designed for direct observation of medical trainees' clinical skills with actual patients. Of these, only a few have demonstrated sufficient evidence of validity to warrant more extensive use and testing.