Can Physician Profiles Be Trusted?

More than 25 years ago, Wennberg and Gittelsohn¹ observed that communities in Vermont varied a great deal in the amount of health care they used. Many others have since duplicated these findings of variation in health care utilization across communities (ie, small-area variation). Wennberg and colleagues² concluded that small-area variation in health care utilization could not be attributed to how sick the population was who lived in various communities, nor to differences in their health outcomes, but rather to irrational physician decision making in the setting of uncertainty.

With time, policymakers in the United States who were struggling with rapidly escalating health care costs embraced these findings and interpreted them to mean that physicians were in part to blame. Health care policymakers and health plan administrators became concerned that they could no longer trust physicians to do the right thing and focused on developing tools that would make physicians accountable for their decisions. Several strategies were developed to steer physicians to more rational behavior or, at least, to guide them to less expensive behavior. To that end, insurance companies and managed care plans have used primary care gatekeepers, utilization review, practice guidelines, and financial incentives to change the way physicians were conducting their business. Physician profiling has emerged as a report card on how physicians are behaving.

Physician profiling involves the displaying of a physicians' processes and outcomes of care from information that can be extracted from clinical and administrative databases and from patient surveys. An individual physician's performance is typically compared against normative standards or with other physicians in the medical group or community. The statistical comparisons are often of limited sophistication and rarely involve much more than age and sex adjustment. While the methodology of physician profiling remains crude, some have suggested that it may offer some benefits over other ways of evaluating physicians.³ For example, compared with time-consuming case-by-case utilization review, a physician profile may provide an efficient mechanism to see a physician's pattern of care.

The article by Hofer et al⁴ in this issue of THE JOURNAL examines the reliability of physician profiling. The authors found that physician profiles were not a reliable measure of individual physician's quality of care at least for treating patients with diabetes. The investigators found a great deal of variation in the profiles across physicians. However, after applying a sophisticated case-mix adjustment method to account for differences in the level of illness among patients with diabetes in each physician's practice, only a tiny fraction of the remaining variation was attributable to physicians. This finding stands in stark contrast to assertions made by Wennberg and others that practice variation is driving inappropriate utilization differences across small areas. In fairness to those who ascribe to the hypothesis that physicians are responsible for much of the variation in utilization patterns, the physicians in the present study were drawn from only 1 of 3 settings in which they may have been receiving either formal or informal messages about how to make their practices conform to a uniform style. This factor could have resulted in an underestimate of the physician contribution to variation in practice in settings in which physicians are more isolated.

The finding that physicians only contribute a small amount toward inappropriate utilization should not be all that surprising. Earlier studies that compared communities with high and low rates of using medical procedures found a substantial amount of inappropriate care in those communities, but the proportion of inappropriate care did not explain the varying use rates.⁵ If inappropriate physician practices were responsible for higher use rates, then they should have found a greater proportion of unnecessary care in the high use-rate communities. Similarly, Komaromy et al⁶ reported that physicians vary substantially in response to scenarios designed to determine their admitting practices among patients with particular chronic medical conditions, but this variation did not explain small-area variation in hospitalization rates for these same conditions.

If it is so difficult to demonstrate that physician practice variation drives inappropriate health care utilization, why do the techniques designed to eliminate practice variation work at all? It may boil down to the medical practice equivalent of the Hawthorne effect—physicians who know that they are being watched start to act differently. Physicians randomized and blinded to whether they were receiving true utilization review vs sham review, in which no requests were denied, exhibited a similar level of reduction of service use.⁷ Apparently, physicians only need to think they are being observed to have their care patterns change. Whether this implies quality improvement is a separate question altogether.

Proponents of physician profiles and other forms of report cards acknowledge the limitations of these tools but suggest that inaccurate data are better than none at all and that with time the reliability of the information in these reports will improve.⁸ The results of the study by Hofer et al suggest that this may not be the case, at least for primary care physician profiles. A substantial part of why physician profiles are unreliable is because of the small number of cases with specific diagnoses that each primary care physician sees. It would require a dramatic redistribution of patients with particular diseases to a limited number of physicians to have enough observations per physician to be able to create an accurate profile. Even then, the potential for gaming the system by avoiding the sickest patients remains a very real risk with unacceptable consequences for access to care.

While the health care community awaits clarification on the benefits of reliable physician profiling, investigators should carefully scrutinize the costs of establishing physician profiles. Not only are there financial costs associated with collecting these data, but also there are risks that these tools, which have been used to limit physicians' admitting privileges and contracting opportunities with managed care organizations, may likely contribute to physicians' increasing dissatisfaction with the health care system.⁹ Thus, physician profiling, which was designed to enhance quality, may result in the exact opposite effect. Angry and defensive physicians who feel scrutinized rather than trusted may be less motivated to contribute their best efforts to improving health care quality. Patients are also likely to be disenchanted when learning that the physician profiles are unreliable and only create the pretense of informing them about health care quality.

Two challenges remain for physician profiles: the creation of a reliable instrument and a method of dissemination that fosters true quality improvement. The results reported by Hofer et al should make it easier for responsible participants in the health care system to agree that physician profiles should not be used as a part of contract negotiations between physicians and health plans and that physician profiles are not and may never be ready for public consumption. In the meantime, the health care community should proceed cautiously with using these measures within organizations to facilitate communication while investigators evaluate their roles in contributing to a climate of (mis)trust and quality improvement.¹⁰

The small-area analysis techniques, established by Wennberg and Gittelsohn, have revealed important information on how health care resources are distributed in the population. However, rather than promulgating a possible fiction about why this occurs, it is time that the health care community more carefully evaluates the contribution of physician practice to variation in service use.