Accuracy of Physician Self-assessment Compared With Observed Measures of Competence:  A Systematic Review FREE

Self-assessment and self-directed, lifelong learning have long been mainstays of the medical profession—they are activities presumed to be linked closely to the quality of care provided to patients.¹ Physicians in the United States must demonstrate their engagement in lifelong learning by choosing and participating in continuing medical education (CME) activities² and acquiring CME credit, which is mandated by the majority of state medical boards under the rubric of states' medical practice acts.³ The American Medical Association's Physicians Recognition Award certificate,⁴ which is based on CME participation, meets the CME requirements of the Joint Commission on Accreditation of Healthcare Organizations related to hospital accreditation.

Self-assessment and lifelong learning were adopted by the American Board of Medical Specialties explicitly as 1 of 4 elements in its Maintenance of Certification program.⁵ Furthermore, diplomates of the American Board of Internal Medicine who choose to recertify must complete 10-year cycles of recertification, a process focused on continuous professional development that requires the capacity of physicians to self-assess.⁶ In graduate medical education, the issue of practice-based learning and improvement based on self-assessment is a central tenet of professional development in Canada,⁷ the United States,⁸ and in other countries.⁹

Each of the elements in this chain—the emphasis on self-assessment, self-directed lifelong learning, the acquisition of CME credits and their use for medical relicensure, accreditation, and ongoing certification—is heavily dependent on the ability of physicians to determine their own learning needs and find resources to meet them. However, a recent review of theory-oriented literature of self-assessment (including the health professions) raises serious questions about the failure of professionals to generate summary judgments of their performance in any regular or consistent fashion, a critical requirement for a self-regulating profession.¹⁰

While the term self-assessment is used to describe many types of activities, we were interested in considering the aspects of “self-rating” or “self-audit” in contrast to the use of self-administered examination of knowledge or clinical performance. To our knowledge, no systematic reviews of studies of this type of physician self-assessment compared with external observation as a reference standard exist. We, therefore, reviewed the literature to determine how accurately physicians self-assess compared with external observations of their competence.

The databases of MEDLINE (1966-July 2006), EMBASE (1980-July 2006), CINAHL (1982-July 2006), PsycINFO (1967-July 2006), the Research and Development Resource Base in CME (1978-July 2006),¹¹ and proprietary search engines were searched using the terms self-directed learning, self-assessment, self-reflection, self-rating, reflective practice, multisource feedback, and related terms. Indexing of this topic in the literature databases is limited and few relevant subject headings were available, thus searching relied on identifying key words through seminal articles and expert consensus. Hand-searching was performed by reviewing references of retrieved articles. The complete search strategies are available on request.

Studies that focused on a comparison between physicians' self-assessments as determined by self-ratings and 1 or more external measures of related competencies were included. Studies were selected that used such self-assessments (ie, physician perceptions or predictions of knowledge, skill, or performance) compared with an external, well-described measure (eg, objective structured clinical examinations [OSCEs] in the same domain), or compared observed performance ratings. In addition, included studies had to: use quantifiable and replicable measures; have a study population of at least 50% practicing physicians, residents, or similar health professionals such as nurse practitioners and physician assistants to be able to generalize to these groups; and be conducted in the United Kingdom, Canada, United States, Australia, or New Zealand, which have similar training requirements, maintenance of competence programs, languages, and CME requirements. We excluded studies that were comparisons of self-reports; evaluated medical students or primarily examined other health professionals; described the development or testing of self-assessment measures; assessed physician beliefs about patient status; or were self-assessment programs of specialty societies that provide feedback to physicians on tests of competence. Studies conducted in the context of an educational or quality improvement intervention in the subject area of the assessment were included only if comparative data were obtained before the intervention.

The following information was extracted from each article: study population; content area and self-assessment domain of the study; methods used to measure the self-assessment of study participants; methods used to observe or measure participants' competence or performance; existence and use of quantifiable measures; and study outcomes. When available, data within studies that might explain the association between the self- and external assessments were sought. None of the data extraction was performed in a blinded fashion. A meta-analysis was not performed because the conceptual constructs within the domains of self-assessment were varied and assessed different skills using varying measures.

The methodological quality of the articles was assessed by determining (1) whether there was sufficient description to permit replication of the study population, (2) the content domain of the self-assessment, and (3) the explicit indication of blinding of the external observation to the self-assessment. In addition, the sampling frame on which the study population was based, the methods used in the self- and external assessments, the identification of pilot testing or use of previously validated methods by the authors of the studies, and the presence and appropriateness of statistical tests were determined.

The literature search was performed by one of the authors (L.P.) and duplicated by an independent information specialist. One of the authors (D.A.D.) determined the inclusion criteria. The inclusion criteria were applied to the abstracts of all articles by 2 of the authors (D.A.D., P.E.M.) and corroborated by another author (L.P.). Data extraction methods were developed by one of the authors (D.A.D.) and were applied by 2 of the authors (D.A.D., P.E.M.) to those articles that met the inclusion criteria. Disagreements about search criteria, data extraction, and classification of study results were resolved by consensus.

The search strategies yielded 725 articles after removal of duplicate and irrelevant studies by title search (Figure). After applying inclusion criteria to the abstracts of these articles and excluding studies that were based on self-reports, evaluated medical students or other health professionals who did not fit the inclusion criteria, were patient-focused, or were reports on the development of self-assessment tools and constructs, posteducation assessments, or self-assessments of specialty society programs, there were 30 articles published between 1988 and 2005. After review of the full text of these articles, another 13 were excluded based on these criteria or because they focused on physician characteristics such as the decision-making process or were conducted in excluded countries, leaving 17 articles^12- 28 that met all of the inclusion criteria (Figure). Three studies used 2 external comparisons each,^12,15,23 resulting in 20 comparisons between self- and external assessment.

The majority of the studies reported findings related to clinical medicine, such as procedural skills, palliative care, and general medical knowledge.^{12- 17,21- 25,27- 28} One study focused on teaching abilities,¹⁸ another focused on cultural competencies,¹⁹ and 2 studies focused on evidence-based medicine^20,26 (Table 1). Six studies^{13,18- 20,24,27} examined the self-assessment abilities of practicing physicians and one study¹⁶ focused on practicing physicians, physician assistants, and nurse practitioners. The study by Leopold et al¹⁶ was conducted in the context of a specific, focused educational intervention. The remainder of the studies focused on graduate physician-trainees^{12,14- 15,17,21- 23,26} or both trainees and practicing physicians^25,28 (Table 2 and Table 3).

The 20 comparisons between self- and external assessment were divided into 3 constructs of self-assessment. Six studies^{14- 15,17,20,26,28} focused on predictive self-assessment, which is the ability of the physician to predict his/her performance on a future competency-based assessment.^{12,16,18- 19,21- 25}

Nine comparisons were in the construct of summative and retrospective self-assessment. In 2 studies,^12,16 participants were asked to rate their performance in a recently completed simulation exercise, which was later compared with the ratings given by external observers. Seven studies asked participants to provide mental representations of themselves over time (ie, competent in clinical medicine,^21- 25 competent as surgical teachers,¹⁸ and cultural-linguistic competence¹⁹) compared with the perceptions of performance in these areas by residents,¹⁸ patients,¹⁹ faculty supervisors and staff members,^22- 23 objective tests,^21,23- 24 or chart audit.²⁵

The final construct was concurrent self-assessment. Two studies^13,27 were in this category, asking physicians to self-identify current learning needs. In each, the process involved included a reflection on performance, knowledge, or skills in familiar situations.

Self-ratings were conducted in the studies by questionnaire, checklists, or survey and focused on learning needs,^13,27 confidence in performing procedures,^14- 16 general clinical skills,^{12,21- 25,27- 28} medical and critical appraisal knowledge,^17,20,26 and nonclinical competencies (eg, teaching skills¹⁸ and cultural-linguistic competencies¹⁹).

Studies compared physician self-ratings with stable external objective measures such as OSCEs,^14- 15 standardized patients,^12,21 simulations,¹⁶ performance on in-training or other examinations,^{17,23- 24,26- 28} chart audit,²⁵ or the ability to explain concepts of evidence-based medicine to a blinded interviewer.²⁰ Studies also compared physician self-ratings to data derived from structured interviews aided by physician diaries¹³ or ratings by stakeholders including residents,¹⁸ patients,¹⁹ or faculty supervisors.^{12,15,22- 23} In 3 of the studies,^12,18,23 the instruments used in the physician self-ratings closely matched the instruments used for comparative purposes.

The studies demonstrated heterogeneity in their choice of comparisons and use of statistical methods. The studies reported either use of descriptive statistics^{13,20- 21,24} or inferential statistics.^{12,14- 19,22- 23,25- 28} Two studies^12,22 did not identify the statistical tests applied. One study¹² used the same instrument for self- and external measurements, thus permitting precise tests of agreement.

Of the 20 comparisons between self- and external assessment, 13 demonstrated little, no, or an inverse relationship between self-assessment measures and other indicators.^{12- 18,20- 21,23,26- 27} Six studies used ratings of global confidence compared with performance of procedures, using behavioral checklists for rating performance in the assessment of dementia,¹² procedural skills,^14- 16 health promotion counseling,²¹ and critical care skills.²³ Two studies used structured interviews as an external measure, identifying a lack of congruence between self-assessment and external observation in detecting learning needs in palliative care physicians¹³ and in physicians' abilities to explain evidence-based medicine terms.²⁰ Three studies used tests such as multiple choice examinations in critical care,²³ standardized articles to test critical appraisal skills,²⁶ and true-false tests in 3 areas of primary care.²⁷ One study demonstrated a failure of surgeons to judge the perceptions of residents relative to the surgeons' teaching skills.¹⁸ Overall, the proportion of studies reporting little, no, or inverse relationships did not appear to vary by level of training or experience or by year of study.

In contrast, 7 comparisons^{12,15,19,22,24- 25,28} demonstrated positive associations between self-assessment and most external observations. Three found consistency (and little variability) between postperformance self-ratings by residents and observer ratings on global or general levels of performance in assessing dementia,¹² basic clinical skills,¹⁵ and in competency in managing the psychological aspects of family practice.²² Two studies demonstrated positive associations between self-rated expertise and the diagnosis of childhood sexual abuse by physical exmination,²⁴ or “comfort” in recognizing the diagnostic features of smallpox.²⁸ Both studies used case vignettes and photographs of related physical findings. One study¹⁹ displayed a strong association between physician self-rating of language and cultural competency and patient reports of the interpersonal process of care using a standardized instrument. Another study²⁵ demonstrated an association between self-rated sensitivity to emotional and psychological issues in patients and the diagnosis of these issues in practice by chart audit.

However, both this study²⁵ and another²⁴ displayed variability among a subset of participants rating themselves as skilled. In the study by Robbins et al,²⁵ physicians who believed themselves to be better at detecting hidden emotions were in fact less accurate than their colleagues. In the study by Paradise et al,²⁴ there was wide variability among respondents and disagreement with expert consensus about the diagnosis of sexual abuse among up to 20% of self-styled experts.²⁴Among those studies identified as demonstrating little or no relationship between self- and other-identified observations, there were 3 related findings regarding the misperceptions of abilities. First, Leopold et al¹⁶ found an inverse relationship between confidence and competence in simulated joint injections. Second, Fox et al¹⁵ demonstrated less concordance in an OSCE setting between self- and other perceptions in poorly performed basic clinical skills than in those instances in which participants performed well. Third, Parker et al¹⁷ found that residents scoring in the lowest quartile of a knowledge-based family practice examination recognized their learning needs less well than those in higher-achieving quartiles.

Regarding variables that might explain differences in accuracy of self-assessment, 2 studies reported the age^16,25 and 1 study reported the experience²⁸ of participants, linking this to self-assessment abilities. Age or experience did not correlate with the clinician's ability to judge performance in joint injections.¹⁶ In contrast, age did correlate with a decreased propensity to diagnose emotional aspects of illness in one study²⁵ and with more accuracy in diagnosing smallpox in another.²⁸

The studies demonstrated variability in methodological quality. The majority of populations studied were well described. Only 1 study²⁰ used the vague phrase “general practitioners.” Additionally, the sampling frame was addressed in most studies, although the total number from which the sample was drawn was not identified in 3 studies^12,17,20 and a useful description of nonparticipants was not supplied in 9 studies.^{13,15- 18,23,25- 26,28} The content domain was well characterized in all studies either in text or a tabular form. Two studies^13,20 used standard qualitative research methods such as tape-recorded and transcribed interviews. Only 1 study²⁰ referred to the blinding of an external observer.

Quantitatively, 9 studies^{12,14- 16,19,21,23,25,27} used pretested or validated measures such as OSCEs, standardized patients, and standardized instruments, and 2 of these studies^14,16 described objective criteria for performance observation. Two studies^14- 15 used medical students for the pilot testing of the methods for comparison with self-assessment measures. In contrast, other studies^{13,15- 21,24,26- 27} applied self- or other assessment instruments that were not described as having been pilot tested or validated. Regarding the use and reporting of statistical tests, we found examples of flawed methods. For example, 2 studies^12,22 did not report which tests were used. Other problems were insufficient or confusing data precluding confirmation of odds ratios,¹⁹ dichotomizing variables,^19,28 confusion of and incomplete use of parameters in scales with inadequate statistical application,²⁰ and inadequate justification for use of statistics with mixed scales (eg, confidence comparisons with OSCE scores).^14- 15 When multiple comparisons were performed, no description of efforts to control for inflation of type I error was provided.^12,18 No trends in improvement of methodological rigor were detected over the time span represented by these studies.

This systematic review found that in a majority of the relevant studies, physicians do not appear to accurately self-assess. Weak or no associations between physicians' self-rated assessments and external assessments were observed. While some studies found a reasonable association between physicians' demonstrated self-assessment abilities and external assessments in the area of cultural and linguistic sensitivity,¹⁸ between self- and supervisor ratings at a general level,^12,15,22 between self- and external tests,^24,28 and between self-assessment and chart audit, wide variability and some errors in judgment are demonstrated in other studies.^24- 25 In the studies indicating poor or limited accuracy of self-assessment, this finding was independent of level of training, specialty, the domain of self-assessment, or manner of comparison.

These findings are not new. Sibley et al²⁹ reported similar findings more than 2 decades ago, as did subsequent studies by Gordon³⁰ and Dunning et al.³¹The findings are consistent with studies in other disciplines. For example, in a meta-analysis of quantitative self-assessment studies in law, engineering, guidance counseling, behavioral science, psychology, and medicine, Falchikov and Boud³² noted correlations between self- and external assessments of student performance ranging from 0.05 to 0.82, with a mean of 0.39. Within the health profession, Gordon³⁰ found that correlations for self-assessments of knowledge ranged from 0.02 to 0.65. Furthermore, despite our finding in 2 studies^24,28 that specific self-assessment may be reliable predictors of performance, Eva et al³³ found that poor correlations persist even when domains are well-defined. Finally, perhaps of greatest concern are the findings that those who perform the least well by external assessment also self-assess less well. These results have been demonstrated by others^34- 35 and require further understanding.

Taken together, these conclusions prompt reflection on the use of self-rated assessment and its role in lifelong learning and value in regulation and patient care.

These studies highlight several considerations for the study of self-assessment as an important domain of physician competence. First, the construct of self-rated assessment itself is not easily studied, in large part because its nature is neither fully developed nor tested.¹⁰ We defined 3 discrete types of self-assessment: predictive, summative, and concurrent. The most value may come from attention to conceptual clarity and coherence for the field of self-assessment^36- 37; a more thorough understanding of continuous self-assessment; and a more precise definition of self-assessment to include an increased understanding of physicians' abilities to reflect^38- 39 and of the nature of insight.⁴⁰ Furthermore, given that these studies shed only limited light on the process of self-assessment, further research in this area might move beyond the boundaries of social and behavioral psychology to include cognitive, simulation, or other promising approaches such as the appraisal of perceived self-efficacy or examining the role that age and experience might play in the process.⁴¹

Second, if such studies of self-assessment are undertaken, researchers should augment study rigor and reportability by better describing their populations, sampling frame, and methods; more clearly differentiating between types of self-assessments; attempting to resolve questions of volunteer bias; and articulating best-practice templates for studying and reporting self-assessment compared with external assessment. In their review of self-assessment methods, Ward et al³⁶ also call for increased methodological rigor by improving the validity and reliability of the external reference standard, increasing the description of anchors in questionnaires used in self-assessment tools, and focusing more on reporting results for the individual and less on the group.

Several limitations in this review should be considered. First, while literature searches were conducted by one of the authors and an independent information specialist to provide an exhaustive coverage of the literature, the lack of extensive Medical Subject Headings in the literature databases could have contributed to not retrieving some studies. We attempted to overcome this limitation by using proprietary search engines that used full-text search strategies. Second, some studies lacked full descriptions of methods, outcomes, and use of statistical tests, limiting our ability to describe the studies more fully, to develop explanatory hypotheses, or to generalize. Third, the domains of competence or performance in these studies, the tools used to measure them, and the assumptions (such as predictions of future performance on a test, or self-ratings in past performances as a teacher) were variable, precluding a meta-analytic approach.

Finally, it can be argued that the relatively small number of studies found in this review—with their mixture of methods, differing levels of physician training and experience, volunteer physician participation, and conceptual variability—provides an inadequate evidence base for understanding the ability of physicians to perform self-rated assessments. However, we believe that the selected literature offers fairly consistent evidence of the limited ability of physicians to independently assess their performance. These findings can help inform both further research in this area and the structure and practice of self-directed learning and self-assessment in graduate and continuing education.

If it is true that physicians perform poorly in this domain, new initiatives and formats are needed to assist the self-assessment process and to more accurately promote and assess broader domains of competence such as professionalism and lifelong learning.³⁸ The positive findings were in global performance, with the potential for feedback on broad dimensions of care by faculty supervisors, given over time,^12,15,19,22 or in highly specialized areas such as child sexual abuse or bioterrorism,^24,28 in which the practitioner might be expected to accurately self-assess. Ultimately, a more useful approach may be to focus on externally determined self-assessments to guide the clinician in the use of educational and other activities designed to improve performance.

First, such measures might include the development of a more holistic continuing professional development process involving learning portfolios, documenting practice-based learning and improvement activities, creating less general and more detailed learning and practice objectives, and addressing the general competencies espoused by the Accreditation Council for Graduate Medical Education.⁸

Second, training may reduce the variation between self- and external assessments by encouraging the internalization of objective measurements or benchmarks of performance.¹⁰ Although one study¹⁶ in this review demonstrated only marginally improved correlation between confidence and skill performance following training and feedback, another study⁴² reported that training increases the relationship between observed and self-reported hand-washing techniques following a hospital-wide quality improvement initiative. Similarly, physician trainees may be able to self-rate more accurately when they compare their ratings with those of others.³⁰ Attention to this effect of training and the comparative feedback phenomenon in undergraduate medical education as well as graduate and CME appears both appropriate and timely.^43- 44

Third, given that some improvement needs (eg, those in the psychosocial realm)^13,45- 46 may be more difficult to self-assess, methods such as multisource feedback (360°) evaluations may be a necessary next step, particularly when interpersonal skills, communication skills, or professionalism needs to be evaluated.⁴⁷ Fourth, objective measures of competence and performance deserve serious consideration, especially when issues of medical licensure, recertification, quality, and patient safety are paramount. In this regard, the National Health Service in the United Kingdom has provided an example of externally informed self-assessment in formulating the concept of appraisal (the structured process of “facilitated self-reflection”⁴⁸) in which an external appraiser guides and directs the process of self-assessment. Finally, specialty societies and others can increase their role in providing current evidence-based learning objectives on a regular basis to members of their discipline, giving external markers of competence.