The Education and Training of Future Physicians: Title and subTitle BreakWhy Coaches Can't Be Judges

A physician must be able to diagnose and treat patients. The clinical skills required to be successful include gathering data, differentiating important from unimportant facts, making decisions about further investigations and treatments, implementing therapy, and providing follow-up, education, and counseling. These skills cannot be learned through reading or in classrooms alone; practical experience is required. The present method of exposing physicians-in-training to practical experience involves a hierarchical team approach with graded levels of responsibility whereby the decisions of the most junior members of the team are reviewed by physicians with more experience and seniority. These practical experiences impart content knowledge and also allow trainees to become comfortable with decision making and to learn the consequences of these decisions. Although there may be better ways to train future physicians, this apprenticeship method seems to work, as evidenced by the relatively low failure rate in medical schools and training programs.

Part of the apprenticeship experience also includes having members at higher levels of the hierarchy evaluate those at lower levels. As such, the supervising individuals are both coaches (instructing and assisting trainees in improving their clinical skills) and judges (responsible for performance assessment of the same trainees).¹ In this Commentary we discuss some of the problems created by this dual role and offer potential solutions.

The purpose of assessing trainees is to motivate and direct learning, as well as to provide an evaluation of competence. In most settings, in-training evaluation reports (ITERs) form the standard method of evaluation.^{2 - 4} ITERs usually involve assessment of multiple attributes and an overall global rating of competence. They commonly are based on ordinal Likert scales with 5 points ranging from failure to outstanding.

Studies of the validity of ITERs have shown that they correlate poorly with other assessment measures.^{5 - 6} Poor reproducibility has been widely documented and explained by many factors including leniency of raters, restriction of range of scores (toward the top end), the halo effect (the tendency to rate all aspects of performance based on 1 trait), inadequate rater training, and lack of direct observation of trainees while performing clinical skills.^{1 ,3 - 4} In addition, supervisors are reluctant to fail trainees.⁷

Despite the perception that trainees are intimidated by their medical supervisors, these same supervisors find it especially difficult to provide negative feedback, even if that feedback may be presented constructively.⁴ The reasons given include a lack of receptivity by trainees, adverse consequences to the faculty such as extra work and stress, and the knowledge that trainees in turn evaluate faculty members. In particular, giving a trainee a failing grade has important consequences for both the trainee (having to repeat the rotation) and supervisor (there may be appeals and need for extensive documentation).

In addition, several cognitive biases may contribute to measurement error in ITERs.¹ For example, during a clinical rotation, early impressions of a trainee's performance in either a positive or negative way may affect subsequent observations through confirmation, anchoring, or other biases. Confirmation bias refers to the phenomenon by which evaluators seek and interpret information to confirm initial impressions, both discounting contradictory observations and overvaluing confirmatory ones. Anchoring bias results from overreliance on 1 piece of information to guide overall impression, for example, relying solely on an excellent presentation on a single topic or a brilliant diagnosis in 1 case. These biases also can be influenced by memory effects when ITERs are performed at the end of a rotation.⁸

Trainees are able to affect their evaluations by techniques such as turning a discussion about their patients toward a moderately obscure topic of which they have detailed knowledge, especially if they know their supervisor lacks expertise in that area.⁹ More recently, trainees have become able to use electronic resources to quickly find answers that may give the evaluator a false impression of their knowledge.⁹

There are 2 main problems with the dual role of coach and judge. First, the dual role provides a suboptimal setting for trainees to seek help in gaining skills in areas in which they perceive deficiency. No trainee would want to show an evaluator his or her weaknesses. Therefore, the teacher has to try to expose what the trainee needs to learn or guess from experience with previous trainees. Second, the dual role creates a conflict of interest for the evaluator because, as coach, he or she will also have a vested interest in the trainee's success.

The simple solution to this problem would be to ensure that all assessments of trainees' knowledge and performance are conducted by individuals who have no vested interest in the success of the candidate. However, this approach is unrealistic. First, the coach will have firsthand direct information about how well the trainee learns skills. In fact, since the current approach to medical education is to develop lifelong learners, the level of competency in any specific area at one point in time may be less important than a physician's ability to acquire competence in other areas in the future. The coach will be able to evaluate performance over time and thereby assess a trainee's ability to acquire new knowledge and develop skills. An external evaluator at a fixed point in time can only assess competency once.

Second, because members at the top of an educational hierarchy have more seniority and expertise, members at the lower levels will always be threatened in some way by their position. For example, clinical supervisors do not simply provide assessments on formal ITERs but also make both formal and informal recommendations in reference letters or by serving as members of residency selection committees. This phenomenon cannot be counteracted. Nevertheless, by focusing on high stakes, pass-fail type assessment from the same individuals who teach clinical skills, the current structure of medical training programs rewards trainees for showcasing their strengths without properly addressing their weaknesses.

Clinical supervisors may be hampered in their ability to be both coach and judge of medical trainees. In particular, decisions to provide a pass or fail grade should not be placed in the hands of clinical tutors. At the same time, coaches will play a valuable role in determining trainees' learning skills but their primary role should be as coaches not judges. In sports, coaches do not just reinforce what an athlete does well; their key role is in addressing areas of relative weakness by identifying them and providing feedback.

In contrast, medical training programs have evolved a system in which strengths are highlighted and praised but weaknesses may be hidden. By shifting the focus of clinical training to feedback and coaching (for the acquisition of expertise¹⁰ ), tutors will help trainees improve the areas in which they need the most help. Until this change occurs, the current dual role of clinical supervisors will continue to fail the needs of physicians-in-training by not helping them to reach their full potential.