Raising the Passing Grade for Studies of Medical Education

Physicians spend much of their time listening and responding to patients' concerns. Studies have found, however, that clinicians' interpersonal skills are not always as good as their patients might wish.^{1 - 2} In response, several medical organizations have called for improved training and competence in communication skills. The Association of American Medical Colleges, for instance, has included "communication in medicine" as a central aspect of its Medical Schools Outcomes Project, which is intended to guide curricula in all US medical schools.³ Beginning in 2004, the National Board of Medical Examiners will require all US medical students to travel to a testing center for an evaluation of their clinical skills, including communication.⁴ The Accreditation Council for Graduate Medical Education now requires all US residency programs to provide instruction in "interpersonal and communication skills."⁵ By the time this year's class of entering medical students will have completed their residencies, they may find that their interpersonal skills will be subject to lifelong examination. In a recent address to the American Board of Medical Specialties (ABMS), Baird⁶ stated that "an expanded assessment of interpersonal and communication skills would be a useful new endeavor for ABMS."

Given this broad consensus on the need to improve communications skills, the overall quality of the evidence for how to teach these and other aspects of professionalism is surprisingly poor.^{7 - 9} Much of this educational literature is not quantitative at all but, rather, comprises opinion pieces, anecdotal reports, and position articles. In their overview of professionalism curricula, Stephenson et al¹⁰ concluded that "Medical education is not short of excellent ideas about how to improve courses and create the professionals needed by society. What is in much shorter supply is evidence about the effectiveness of such teaching." Goldie¹¹ described a similar lack of suitable studies to evaluate the outcomes of ethics curricula for medical students. Thus, despite a broad consensus on the need for high-quality studies, little evidence exists to guide educators in how to design the best possible programs, or how to evaluate and improve them.

There are many inherent difficulties in conducting high-quality educational research.⁹ Perhaps the most immediate of these is a lack of money and time. Although society has an interest in training physicians who can communicate effectively, such curricula have limited market potential and thus may not compete favorably for corporate support. Grant monies are also limited. Even assuming a more favorable funding environment, a range of methodological obstacles would remain. Educational interventions and outcome measures are expensive and time consuming, and the best of these are always personalized to specific environments and learners. The resulting small sample sizes limit generalizability and statistical power.

Another problem is the difficulty in measuring outcomes that relate to the intent of education. Although better communication would ideally lead to improved patient outcomes, such clinical variables are quite problematic as measures of clinicians' competence. In this issue of THE JOURNAL, Landon and colleagues¹² describe the many limitations of clinical outcomes to assess physicians' quality of care. The difficulties of clinical outcomes to assess educational interventions would add another layer of confounding, including problems with randomization, blinding, and contamination of the intervention. Taken together, these challenges appear so vexing that Prideaux¹³ recently declared that randomized trials in education "are doomed to fail."

Difficult as these problems are, they do not entirely account for the field's incomplete accommodation to scientific methods. A more fundamental problem may be the intuitive and abstract nature of concepts such as "communications," which allows for any number of interpretations. Indeed, the most important educational constructs are probably the most difficult to define in a way that would permit valid quantification. Campbell and Johnson,¹⁴ for instance, asked, "How can the concept of multiprofessional learning become robust if we don't know what it means, cannot agree its goals [sic], and do not seem able to report the weaknesses and problems encountered and lessons learnt?" Other reviewers have remarked on the wide diversity of definitions, across medical schools, for similar educational constructs.^{15 - 16}

In this issue of THE JOURNAL, Yedidia and colleagues¹⁷ report a trial of a communication curriculum at 3 medical schools. Their study demonstrates that much can be accomplished within these inherent limitations of educational research. Although Yedidia et al began with a general interest in improving students' communication skills, they used a telescoping strategy to move from 5 abstract communication tasks to a set of 21 rigorously standardized criteria. These measures were assessed with an objective structured clinical examination (OSCE), a tool that has come into general use for measuring communication skills and other complex clinical behaviors. The specific tasks for the 10 OSCE stations in the study were defined with input of individuals within each of the 3 institutions. The authors then went to great lengths to standardize the OSCEs at each of these geographically disparate sites. In contrast, most curricula do not have the resources available to Yedidia et al and, thus, generally develop OSCEs that cannot reliably be reproduced elsewhere.

The multicenter study by Yedidia et al sheds light on the limits of what the OSCE can be expected to accomplish as a standardized measure of communications skills. Although many well-designed OSCEs have been found to have acceptable test-retest or interobserver reliability,¹⁸ questions remain about both the content and construct validity of these examinations. The first of these criteria, content reliability, reflects the degree to which a test measures a representative sample of performance. Written tests, for instance, can probe a large range of knowledge in a rapid and economical way. Within their methodological limitations, they can produce a reasonably complete picture of examinees' abilities. To achieve a similar content validity, however, oral examinations may require substantially more than the 10 OSCE stations used in the study by Yedidia et al.¹⁹ Standardization of an even larger number of OSCEs across institutions might have been beyond the authors' financial and logistical resources. The costs of administering a fair, standardized, and universal OSCE might be prohibitive under almost any budget.

Construct validity is a more subtle concept and relates to the degree to which a test assesses some hypothesized underlying ability. The problem is that constructs such as "communications skills" are products of cultural and historical imagination. As such, they are inescapably imaginary. They do not lead an independent existence in any identifiable volume of space or expanse of time and are not waiting there to be discovered. Because "communications skills" as generally understood do not have any underlying physical dimensions, they are of an inherently different nature than any conceivable objective criterion.

Although an abstract construct like communication cannot be measured directly, its presence might nonetheless be inferred by the kinds of shadows it casts on various surfaces. To put it another way, measurement of constructs requires operationalization, which means that their measurement is inseparable from the operations by which they are measured. Thus, the standard psychometric approach to assessing a theoretical construct is to triangulate on it with several different methods, as originally described by Campbell and Fiske.²⁰ Although such rigorous evaluation of OSCE methods has not been reported to my knowledge, a number of studies have found that OSCEs have only moderate to poor correlations with other methods of measuring competence, including faculty evaluations, in-service examinations, and standardized testing.^{21 - 24}

From such data, it is tempting to conclude that OSCEs must measure some unique components of clinical skills that are not assessed by other methods.²⁵ It is likely, however, that a portion of these low correlations are attributable to method variance, which is related to the common observation that some individuals do better at some kinds of tests than others, irrespective of apparent underlying ability. While the OSCE does have a certain amount of compelling face validity, its degree of construct validity remains to be determined. At present, all that can be said is that the intervention by Yedidia et al improved communications skills as measured by their particular OSCE. Indeed, given the logistic barriers to standardizing OSCEs across large groups of individuals, such overarching constructs may be forever fractured into countless such smaller operationalizations.

Such a fragmentation could be salutary to the extent that it would move the focus away from abstract discussions of theoretical educational concepts and toward a goal of all students demonstrating measurable competencies. Such a pragmatic approach need not be intellectually sterile. The study by Yedidia et al demonstrates that complex variables such as "communication skills" can be usefully conceived at several levels of specificity. Although all 3 schools committed to a number of basic curricular principles, they were each free to define their own particular timing and curricular content with them. Thus, students at the 3 schools may have had quite different experiences of the "intervention." While this lack of standardization would be a critical failing in a study of a drug or medical device, such variability reflects a realistic understanding of how such curricula are likely to evolve in different institutional cultures. Beyond this reflection of actual practice, the results can also provide valuable formative feedback to curriculum planners at each institution.

Given these caveats, the method used by Yedidia et al permitted a quantification of the effect of the intervention, both within and across schools. The authors found that, overall, the curriculum produced a 5% increase in communication skills. Could their massive intervention thereby be called "successful," an adjective that is often attached to purely narrative descriptions of new curricula? The faculty at these 3 schools certainly seem to have worked very hard to achieve this 5% improvement. Perhaps, in fact, this would be found to be an above-average outcome, were all such curricula assessed as rigorously. Nonetheless, the authors acknowledge that it is difficult to know how to judge this difference in absolute terms, given the lack of similar data for other medical-education interventions.

It would thus be especially helpful to develop common metrics and benchmarks for educational interventions, but this can happen only if future studies were to adopt a similar commitment to quantification. Such an approach has proven fruitful in other areas of clinical research. For instance, the effects of clinical trials can be compared on the common metric of number needed to treat, which allows for meaningful comparisons between the effects of unrelated interventions. Similarly, the results of cost-effectiveness analyses of unrelated interventions can be compared on the basis of quality-adjusted life-years. Such a common metric (perhaps the "number needed to teach") could allow curriculum planners to assess the potential benefits of one curriculum against another. With information about costs and accurate effect sizes, medical schools could also attempt to estimate the true costs of achieving their stated educational objectives.²⁶

Surprisingly, Yedidia et al do not report whether students liked the intervention or how much they thought it improved their communication skills. Given that self-reported satisfaction is a common outcome variable in educational research,²⁷ the lack of data on students' perceptions is certainly atypical of this literature. Self-reported satisfaction can be a useful adjunct to objective measurement of competence. This variable may also be of interest to the extent that students are conceived as consumers of educational services. Exclusive reliance on self-report data, however, is of dubious value in assessing the effect of a curriculum. Self-reported competence may have virtually no connection with objectively measured skills,²⁸ and it has long been known that respondents do not have perfect access to their own internal mental processes in any event.²⁹ Satisfaction data are also likely subject to publication bias, as it seems rare to read accounts of programs that participants did not like.

Overall, the study by Yedidia et al is an important step in setting a higher standard for the quality of research in medical education. The field already has a rich surplus of narrative descriptions and opinion pieces about these issues, but there is scant evidence that this publication enterprise has led to measurable improvements in the quality of medical instruction. Although the funding for high-quality studies remains a challenge, some authors¹⁶ have suggested that funders of education are increasingly demanding rigorous measures of effectiveness.

The public is intensely interested in improving the bedside manner of physicians, and that desire that has now been heard at several levels of organized medicine. Without better evidence for best teaching practices, however, all parties may continue to wonder why physicians cannot do better at understanding and responding to their patients' concerns.