Defining and Assessing Professional Competence FREE

Medical schools, postgraduate training programs, and licensing bodies conduct assessments to certify the competence of future practitioners, discriminate among candidates for advanced training, provide motivation and direction for learning, and judge the adequacy of training programs. Standards for professional competence delineate key technical, cognitive, and emotional aspects of practice, including those that may not be measurable.^1- 2 However, there is no agreed-upon definition of competence that encompasses all important domains of professional medical practice. In response, the Accreditation Council for Graduate Medical Education defined 6 areas of competence and some means of assessing them³: patient care (including clinical reasoning), medical knowledge, practice-based learning and improvement (including information management), interpersonal and communication skills, professionalism, and systems-based practice (including health economics and teamwork).³

In this article, we will advance a definition of professional competence of physicians and trainees that expands on these 6 areas, perform an evidence-based critique of current methods of assessing these areas of competence, and propose new means for assessing residents and medical students.

Building on prior definitions,^1- 3 we propose that professional competence is the habitual and judicious use of communication, knowledge, technical skills, clinical reasoning, emotions, values, and reflection in daily practice for the benefit of the individual and community being served. Competence builds on a foundation of basic clinical skills, scientific knowledge, and moral development. It includes a cognitive function—acquiring and using knowledge to solve real-life problems; an integrative function—using biomedical and psychosocial data in clinical reasoning; a relational function—communicating effectively with patients and colleagues; and an affective/moral function—the willingness, patience, and emotional awareness to use these skills judiciously and humanely (BOX 1). Competence depends on habits of mind, including attentiveness, critical curiosity, self-awareness, and presence. Professional competence is developmental, impermanent, and context-dependent.

Cognitive
Core knowledge
Basic communication skills
Information management
Applying knowledge to real-world situations
Using tacit knowledge and personal experience
Abstract problem-solving
Self-directed acquisition of new knowledge
Recognizing gaps in knowledge
Generating questions
Using resources (eg, published evidence, colleagues)
Learning from experience

Technical
Physical examination skills
Surgical/procedural skills

Integrative
Incorporating scientific, clinical, and humanistic judgment
Using clinical reasoning strategies appropriately (hypothetico-deductive, pattern-recognition, elaborated knowledge)
Linking basic and clinical knowledge across disciplines
Managing uncertainty

Context
Clinical setting
Use of time

Relationship
Communication skills
Handling conflict
Teamwork
Teaching others (eg, patients, students, and colleagues)

Affective/Moral
Tolerance of ambiguity and anxiety
Emotional intelligence
Respect for patients
Responsiveness to patients and society
Caring

Habits of Mind
Observations of one's own thinking, emotions, and techniques
Attentiveness
Critical curiosity
Recognition of and response to cognitive and emotional biases
Willingness to acknowledge and correct errors

Evidence-based medicine is an explicit means for generating an important answerable question, interpreting new knowledge, and judging how to apply that knowledge in a clinical setting.⁴ But Polanyi⁵ argues that competence is defined by tacit rather than explicit knowledge. Tacit knowledge is that which we know but normally do not explain easily, including the informed use of heuristics (rules of thumb), intuition, and pattern recognition. The assessment of evidence-based medicine skills is difficult because many of the heuristics used by novices are replaced by shortcuts in the hands of experts,⁶ as are other clinical skills.⁷

Personal knowledge is usable knowledge gained through experience.⁸ Clinicians use personal knowledge when they observe a patient's demeanor (such as a facial expression) and arrive at a provisional diagnosis (such as Parkinson disease) before eliciting the specific information to confirm it. Because experience does not necessarily lead to learning and competence,⁹ cognitive and emotional self-awareness is necessary to help physicians question, seek new information, and adjust for their own biases.

Professional competence is more than a demonstration of isolated competencies¹⁰; "when we see the whole, we see its parts differently than when we see them in isolation."¹¹ For example, the student who can elicit historical data and physical findings, who can suture well, who knows the anatomy of the gallbladder and the bile ducts, and who can draw the biosynthetic pathway of bilirubin may not accurately diagnose and manage a patient with symptomatic gallstones. A competent clinician possesses the integrative ability to think, feel, and act like a physician.^6,12- 15 Schon¹⁶ argues that professional competence is more than factual knowledge and the ability to solve problems with clear-cut solutions: it is defined by the ability to manage ambiguous problems, tolerate uncertainty, and make decisions with limited information.

Competence depends on using expert scientific, clinical, and humanistic judgment to engage in clinical reasoning.^{14- 15,17- 18} Although expert clinicians often use pattern recognition for routine problems¹⁹ and hypothetico-deductive reasoning for complex problems outside their areas of expertise, expert clinical reasoning usually involves working interpretations¹² that are elaborated into branching networks of concepts.^20- 22 These networks help professionals initiate a process of problem solving from minimal information and use subsequent information to refine their understanding of the problem. Reflection allows practitioners to examine their own clinical reasoning strategies.

The quality of the patient-physician relationship affects health and the recovery from illness,^23- 24 costs,²⁵ and outcomes of chronic diseases^26- 29 by altering patients' understanding of their illnesses and reducing patient anxiety.²⁶ Key measurable patient-centered²⁸ (or relationship-centered)^30- 31 behaviors include responding to patients' emotions and participatory decision making.²⁹

Medical errors are often due to the failure of health systems rather than individual deficiencies.^32- 34 Thus, the assessment of teamwork and institutional self-assessment might effectively complement individual assessments.

Moral and affective domains of practice may be evaluated more accurately by patients and peers than by licensing bodies or superiors.³⁵ Only recently have validated measures captured some of the intangibles in medicine, such as trust^36- 37 and professionalism.^38- 39 Recent neurobiological research indicates that the emotions are central to all judgment and decision making,¹³ further emphasizing the importance of assessing emotional intelligence and self-awareness in clinical practice.^1,40- 42

Competence depends on habits of mind that allow the practitioner to be attentive, curious, self-aware, and willing to recognize and correct errors.⁴³ Many physicians would consider these habits of mind characteristic of good practice, but they are especially difficult to objectify. A competent physician, for example, should be able to judge his or her level of anxiety when facing an ambiguous clinical presentation and be aware of how the anxiety of uncertainty may be influencing his or her clinical judgment. Errors in medicine, according to this view, may result from overcertainty that one's impressions are beyond doubt.^41,43- 44

Competence is context-dependent. Competence is a statement of relationship between an ability (in the person), a task (in the world),⁴⁵ and the ecology of the health systems and clinical contexts in which those tasks occur.^46- 47 This view stands in contrast to an abstract set of attributes that the physician possesses—knowledge, skills, and attitudes—that are assumed to serve the physician well in all the situations that he or she encounters. For example, rather than assessing a student's competence in diagnosing and treating heart disease (a disease-specific domain) by dividing it into competencies (physical examination, interpretation of electrocardiogram, and pharmacology of β-blockers), our view is that competence is defined by the interaction of the task (the concrete process of diagnosing and treating Mrs Brown, a 52-year-old business executive who is now in the emergency department because of new-onset chest pain), the clinician's abilities (eliciting information, forming a therapeutic relationship, performing diagnostic maneuvers, and making judgments about treatment), and the health system (good insurance and ready access to care). Caring for Mrs Brown requires different skills than caring for Ms Hall, a 52-year-old uninsured homeless woman who has similar symptoms and receives episodic care at an inner-city clinic.

Competence is developmental. There is debate about which aspects of competence should be acquired at each stage of training. For example, early clinical experiences and problem-based learning formats encourage clinical reasoning skills formerly relegated to the final years of medical school. But students tend to use the same cognitive strategy for solving all problems, whereas experts draw on several strategies,⁶ which raises the question of whether assessment of practicing physicians should be qualitatively different from the assessment of a student. Determining how and at what level of training the patient-physician relationship should be assessed is also difficult. For example, participatory decision making correlates with clinical outcomes,^25,29 but it is unclear when trainees should be assessed on this skill. Although a third-year resident might be expected to counsel a fearful diabetic patient about the need to start insulin, a third-year student might be expected only to elicit the patient's preferences, emotions, and expectations. Changes in medical practice and the context of care invite redefinitions of competence; for example, the use of electronic communication media⁴⁸ and changes in patient expectations.^49- 50

Assessment must take into account what is assessed, how it is assessed, and the assessment's usefulness in fostering future learning. In discussing validity of measures of competence in an era when reliable assessments of core knowledge, abstract problem solving, and basic clinical skills have been developed,^45,51- 56 we must now establish that they encompass the qualities that define a good physician: the cognitive, technical, integrative, contextual, relational, reflective, affective, and moral aspects of competence. We distinguish between expert opinion, intermediate outcomes, and the few studies that show associations between results of assessments and actual clinical performance.^57- 60

We consider how the process of assessment might foster future learning. Too often, practitioners select educational programs that are unlikely to influence clinical practice.⁶¹ Good assessment is a form of learning and should provide guidance and support to address learning needs. Finally, we address concerns that the medical profession still lacks adequate accountability to the public⁶² and has not done enough to reduce medical errors.^32,63

Within each domain of assessment, there are 4 levels at which a trainee might be assessed (Figure 1).⁶⁴ The knows level refers to the recall of facts, principles, and theories. The knows how level involves the ability to solve problems and describe procedures. The shows how level usually involves human (standardized patient), mechanical, or computer simulations that involve demonstration of skills in a controlled setting. The does level refers to observations of real practice. For each of these levels, the student can demonstrate the ability to imitate or replicate a protocol, apply principles in a familiar situation, adapt principles to new situations, and associate new knowledge with previously learned principles.⁶⁵

Using the MEDLINE database for 1966 to 2001, we searched for articles that studied the reliability or validity of measures of clinical or professional competence of physicians, medical students, and residents. An initial search using the following Medical Subject Headings of the National Library of Medicine yielded 2266 references: educational measurement, patient simulation, clinical competence OR professional competence AND reproducibility of results, validity OR research, OR the text word reliability. This set was reduced by including any of 20 text words describing assessment techniques; we used words such as OSCE, oral examination, peer assessment, triple jump, essay, portfolio, and CEX (clinical evaluation exercise), yielding 430 references. Articles of a purely descriptive nature, reviews that offered no new data, and opinions and position statements were excluded, yielding 101 English-language references. We surveyed the first 200 of the 2165 references excluded and found none that met our search criteria. Quality criteria for inclusion were broad, given the small number of controlled trials of assessment interventions and the complexity of outcome measures. Because we knew that MEDLINE search strategies would not capture all relevant studies, we searched reference lists in the 101 articles, other review articles, and books and did additional literature searches using the key authors of recent reviews; we gathered 94 additional relevant references. Of the 195 references, 124 presented new data on assessment of physicians.

The 3 most commonly used assessment methods are subjective assessments by supervising clinicians, multiple-choice examinations to evaluate factual knowledge and abstract problem solving,⁶⁶ and standardized patient assessments of physical examination and technical and communication skills.^67- 69 Although curricular designs increasingly integrate core knowledge and clinical skills, most assessment methods evaluate these domains in isolation. Few assessments use measures such as participatory decision making⁷⁰ that predict clinical outcomes in real practice. Few reliably assess clinical reasoning, systems-based care, technology, and the patient-physician relationship.^3,69 The literature makes important distinctions between criteria for licensing examinations and program-specific assessments with mixed formative and summative goals.

Evaluation of factual knowledge and problem-solving skills by using multiple-choice questions offers excellent reliability^71- 75 and assesses some aspects of context and clinical reasoning. Scores on Canadian licensing examinations, which include standardized patient assessment and multiple-choice tests, correlated positively with subsequent appropriate prescribing, mammographic screening, and referrals,⁵⁸ and multiple-choice certification examination scores correlated with subsequent faculty⁷⁶ and peer⁷⁷ ratings. Many have questioned the validity of multiple-choice examinations, though.^78- 81 For example, compared with Florida family physicians who are not board-certified, those who are have nearly twice the risk of being sued.⁸² Standardized test scores have been inversely correlated with empathy, responsibility, and tolerance.⁸³ Also, because of lack of expertise and resources, few medical school examinations can claim to achieve the high psychometric standards of the licensing boards.

The Objective Structured Clinical Examination (OSCE) is a timed multistation examination often using standardized patients (SPs) to simulate clinical scenarios. The roles are portrayed accurately^56,84 and simulations are convincing; the detection rate of unannounced SPs in community practice is less than 10%.^{57,59,85- 89} Communication, physical examination, counseling, and technical skills can be rated reliably if there is a sufficiently large number of SP cases^67,90- 100 and if criteria for competence are based on evidence.¹⁰¹ Although few cases are needed to assess straightforward skills, up to 27 cases may be necessary to assess interpersonal skills reliably in high-stakes examinations.^102- 103 Although SPs' ratings usually correlate with those of real patients,¹⁰⁴ differences have been noted.^105- 107

Defining pass/fail criteria for OSCEs has been complex.^54,108- 111 There is debate about who should rate student performance in an OSCE.¹¹² Ratings by the SP are generally accurate⁵² but may be hampered by memory failure, whereas external raters, either physicians or other SPs, may be less attuned to affective aspects of the interview and significantly increase the cost of the examination.

Checklist scores completed by physician-examiners in some studies improve with expertise of the examinees¹¹³ and with the reputation of the training program.^90,114 But global rating scales of interpersonal skills may be more valid than behavioral checklists.^7,115- 116 The OSCE scores may not correlate with multiple-choice examinations and academic grades,^90,100,117 suggesting that these tools measure different skills. Clinicians may behave differently in examination settings than in real practice,^106,118 and short OSCE stations can risk fragmentation and trivialization of isolated elements of what should be a coherent whole.¹¹⁹ The OSCE also has low test reliability for measuring clinical ethics.¹²⁰

There are few validated strategies to assess actual clinical practice, or Miller's does level. Subjective evaluation by residents and attending physicians is the major form of assessment during residency and the clinical clerkships and often includes the tacit elements of professional competence otherwise overlooked by objective assessment instruments. Faculty ratings of humanism predicted patient satisfaction in one study.¹²¹ However, evaluators often do not observe trainees directly. They often have different standards^122- 123 and are subject to halo effects¹²⁴ and racial and sex bias.^125- 126 Because of interpatient variability and low interrater reliability, each trainee must be subject to multiple assessments for patterns to emerge. Standardized rating forms for direct observation of trainees^127- 132 and structured oral examination formats have been developed in response to this criticism.^133- 134

The Royal College of General Practitioners, dissatisfied with the capability of the OSCE to evaluate competence for the final professional licensing examination, developed a format in which candidates for certification present several best-case videotapes of their performance in real clinical settings to a trained examiner who uses specified criteria for evaluation.¹³⁵ Although the face validity of such a measure is high and the format is well accepted by physicians,¹³⁶ the number of cases that should be presented to achieve adequate reliability is unclear.^137- 139

Profiling by managed-care databases is increasingly used as an evaluation measure of clinical competence. However, data abstraction is complex¹⁴⁰ and defining competence in terms of cost and value is difficult. The underlying assumptions driving such evaluation systems may not be explicit. For example, cost analyses may favor physicians caring for more highly educated patients.¹⁴¹

Peer ratings are accurate and reliable measures of physician performance.^77,142 Peers may be in the best position to evaluate professionalism; people often act differently when not under direct scrutiny.¹⁴³ Anonymous medical student peer assessments of professionalism have raised awareness of professional behavior, fostered further reflection, helped students identify specific mutable behaviors, and been well accepted by students.³⁵ Students should be assessed by at least 8 of their classmates. The composite results should be edited to protect the confidentiality of the raters.

Self-assessments have been used with some success in standardized patient exercises¹⁴⁴ and in programs that offer explicit training in the use of self-assessment instruments.¹⁴⁵ Among trainees who did not have such training, however, self-assessment was neither valid nor accurate. Rather, it was more closely linked to the trainee's psychological sense of self-efficacy and self-confidence than to appropriate criteria, even among bright and motivated individuals.

Aside from the need to protect the public by denying graduation to those few trainees who are not expected to overcome their deficiencies, the outcomes of assessment should foster learning, inspire confidence in the learner, enhance the learner's ability to self-monitor, and drive institutional self-assessment and curricular change. Given the difficulty in validating tests of basic skills, it is not surprising that there is scant literature on the assessment of learning, professionalism, teamwork, and systems-based care or on the ability of assessment programs to drive curricular change or reduce medical errors.

Assessment serves personal, institutional, and societal goals (BOX 2). Distinctions between these goals often are blurred in practice. For example, formative feedback is intended to foster individual reflection and remediation¹⁴⁶ but may be perceived as having evaluative consequences. Summative evaluation is a powerful means for driving curricular content and what students learn. Assessment provides information to allow institutions to choose among candidates for advanced training. The public expects greater self-monitoring, communication, and teamwork from health care practitioners.¹⁴⁷ The decline of public trust in medicine may reflect a growing concern that physicians are not achieving these goals.³⁶

For the Trainee
Provide useful feedback about individual strengths and weaknesses that guides future learning
Foster habits of self-reflection and self-remediation
Promote access to advanced training

For the Curriculum
Respond to lack of demonstrated competence (denial of promotion, mandated remediation)
Certify achievement of curricular goals
Foster course or curricular change
Create curricular coherence
Cross-validate other forms of assessment in the curriculum
Establish standards of competence for trainees at different levels

For the Institution
Guide a process of institutional self-reflection and remediation
Discriminate among candidates for further training or promotion
Express institutional values by determining what is assessed and how assessment is conducted
Develop shared educational values among a diverse community of educators
Promote faculty development
Provide data for educational research

For the Public
Certify competence of graduates

Assessment is also a statement of institutional values. Devoting valuable curricular time to peer assessment of professionalism, for example, can promote those values that are assessed while encouraging curricular coherence and faculty development, especially if there are corresponding efforts at the institution toward self-assessment and change.

Whereas performance is directly measurable, competence is an inferred quality.¹⁴⁸ Performance on a multiple-choice test may exceed competence, as in the case of a trainee with a photographic memory but poor clinical judgment. Conversely, competence may exceed test performance, as in the case of a trainee with severe test anxiety. Correlation with National Board scores and feedback on graduates' performance can be useful in validating some assessment instruments but should be done with caution. For example, efficiency is highly valued in residents but less so in medical students.

Medical schools in Canada, the United Kingdom, Australia, Spain, the Netherlands, and the United States have made commitments to developing innovative assessments of professional competence, some of which we describe. These assessments are increasingly multimodal and tailored to the goals and context in which they will be used. Large-scale licensure examinations must use computer-gradable formats, but comprehensive examinations using structured direct observation,¹⁰⁷ OSCE stations, real patient cases,¹⁰⁷ case-based questions,⁷⁹ peer assessments, and essay-type questions¹⁴⁹ are reliable as well. Proponents of the new formats argue that they provide more useful feedback and are more efficient at the medical school or residency level (BOX 1 and BOX 3) than traditional formats.^81,150 They target core knowledge and clinical skills in different contexts and at different levels of assessment. Because of their complexity, a matrix (Figure 1) can be useful to display the domains assessed.

Multimethod assessment
Clinical reasoning in situations that involve clinical uncertainty
Standardized patient exercises linked to postencounter probes of pathophysiology and clinical reasoning
Exercises to assess use of the medical literature
Long-station standardized patient exercises
Simulated continuity
Teamwork exercises
Unannounced standardized patients in clinical settings
Assessments by patients
Peer assessment of professionalism
Portfolios of videotapes
Mentored self-assessment
Remediation based on a learning plan

Comprehensive assessments link content across several formats. Postencounter probes immediately after SP exercises using oral, essay, or multiple-choice questions test pathophysiology and clinical reasoning in context.^151- 152 Triple-jump exercises¹⁵²—consisting of a case presentation, an independent literature search, and then an oral or written postencounter examination—test the use and application of the medical literature. Validated measures of reflective thinking¹⁵³ have been developed that use patient vignettes followed by questions that require clinical judgment. These measures reflect students' capacity to organize and link information; also, they predict clinical reasoning ability 2 years later.¹⁵³ Combining formats appears to have added value with no loss in reliability.^150,154 Ongoing educational outcomes research will show whether composite formats help students learn how to learn more effectively, develop habits of mind that characterize exemplary practice,⁴³ and provide a more multidimensional picture of the examinee than the individual unlinked elements. Two examples of comprehensive assessment formats follow.

A student is instructed to perform a literature search about genetic screening test for Alzheimer disease in anticipation of an SP encounter later that day. Assessment instruments include a structured evaluation of the search strategy and a communication rating scale, completed by an SP, that assesses the clarity of the student's presentation and the student's ability to involve the patient in the decision-making process. Next, the student completes an essay about the ethics of genetic screening and the genetics of Alzheimer disease. This exercise assesses the student's communication skills, clinical reasoning, ability to acquire and use new knowledge, and contextualized use of knowledge of genetics, health economics, and medical ethics.

A rating scale is used to assess a resident on her ability to agree on a plan of action with an SP who portrays an outpatient demanding a computed tomographic scan for headaches without neurological signs. In a postencounter exercise, the resident creates a rank-order differential diagnosis and then answers a series of script concordance^153,155 questions in which the examinee is presented hypothetical additional data (for example, numbness in the left hand) and then asked to judge how her diagnostic hypotheses or therapeutic actions would change. Failure to include a key diagnostic possibility or the overestimation or underestimation of probability are criteria for evaluation. The goal of the exercise is to demonstrate emotional intelligence⁴⁰ and self-awareness in the context of conflict and ambiguity. Similar observations might be made with trainees' video portfolios of real clinical encounters.

Well-functioning health systems are characterized by continuity, partnership between physicians and patients, teamwork between health care practitioners, and communication between health care settings.^156- 157 The use of time in a continuity relationship can be assessed with a series of SP or real-patient exercises. To assess partnership, patient assessment, currently used to assess physicians in practice,¹⁵⁸ is being tested for students and residents.^159- 160 These efforts are guided by data showing that patients' ratings of communication and satisfaction correlate well with biomedical outcomes,^24,29 emotional distress,¹⁶¹ health care use,²⁵ and malpractice litigation.¹⁶² Patient ratings also have the potential to validate other measures of competence.¹⁶³ Several institutions assess teamwork by using peer assessments. Others use sophisticated mannequins to simulate acute cardiovascular physiological derangements found in intensive care settings^164- 169; trainees are graded on teamwork as well as individual problem solving, and statistical adjustments can account for team composition. Communication between health settings could be assessed at the student level, for example, by grading of their written referral letters.¹⁷⁰

Although it could be argued that licensing boards do not have the mandate to remediate examinees who perform poorly or modify educational curricula, medical schools and residency programs do. Tests that demonstrate students' strengths or weaknesses may not provide the student with the opportunity to reflect on actual behaviors and patterns of thought that should be changed. To foster reflection and action, some institutions require a learning plan in which trainees chart their learning needs, the means of achieving them, expected time of completion, and means of verification^146,171 as a required outcome of an assessment.

A strong mentoring system should accompany any comprehensive assessment program. An inadequate system for feedback, mentoring, and remediation will subvert even the most well-conceived and validated examination. Curricular change also can be guided by results of assessments but requires a parallel process of institutional reflection, feedback, and remediation.

These new assessment formats are feasible, and several institutions have invested significant time and resources to develop them. The promise that a more comprehensive assessment of professional competence might improve practice, change medical education, and reduce medical errors should be studied in controlled trials. The public's trust in the medical profession and the ability of medical practitioners to learn from mistakes depends on valid and reliable means of assessment. Medical educators, professional societies, and licensing boards should view professional competence more comprehensively to improve the process of assessment.