The Unreliability of Individual Physician "Report Cards" for Assessing the Costs and Quality of Care of a Chronic Disease FREE

Provider profiling is now widely practiced or attempted in many health care systems.^1- 4 Those who are paying for or "managing" health care are seeking ways to make health care providers more accountable for both the cost and quality of the care that they supply. Reports comparing hospital mortality rates across hospitals and the Health Plan Employer Data and Information Set performance measures across health plans have been widely implemented.^3,5 The goal of the more recent attempts at physician profiling is to hold a single individual (the physician) accountable for what happens to a specific group of patients.^3,6- 8 Developing and disclosing profiles to consumers is a crucial element of the proposed Consumer Bill of Rights and Responsibilities⁹ and is considered an important part of creating efficient health care markets.⁹ Surveys have shown that up to 80% of group practices with capitated patients profile the resource utilization of their physicians.¹ Furthermore, for many hospitals and managed care organizations individual physician profiling has become an integral part of medical staff appointment and issuing clinical privileges.⁸

Although increasing professional accountability is a laudable goal, profiling ventures can be quite expensive (adding as much as $0.59 to $2.17 per member per month),⁹ may have very serious consequences for physicians and could potentially be harmful to patients through distortion of physician incentives.^4,6,8 In their most benign use, profiles are used simply to educate physicians about their practices relative to their peers. However, the most severe consequence of poor performance on a profile is the potential loss of a managed care contract or hospital admitting privileges.⁸

The usefulness of physician profiles depends on their reliability and accuracy. Even so, little published evidence on the reliability of any of the commonly used profiling systems exists.¹⁰ In part, this lack of attention to reliability comes from a lack of data. Few profilers have data with adequate risk adjustment measures, enough observations per provider, and enough providers to try to estimate the true reliability of their measures. The Diabetes Patient Outcomes Research Team study, which examined the type of health care provided to patients with type 2 diabetes living in 3 geographically different locations and enrolled organizationally diverse sites, allows for the opportunity to evaluate the reliability of provider profiling for one of the most common chronic medical conditions. We examined the utility of several potential profile measures, including hospitalization rates, visit rates, total laboratory resource use, and glycemic control. We evaluated (1) the reliability of physician profiles, (2) the importance of detailed case-mix adjustment, (3) the clinical and economic magnitude of variations in physicians' practices, and (4) the potential for physicians to improve their profile measures by gaming the system (ie, through patient selection rather than changing their practice).

The study sample included 3642 patients with diabetes cared for at 3 geographically and organizationally distinct sites: (1) a staff-model health maintenance organization (HMO) on the West Coast, (2) an urban university teaching clinic in the Midwest, and (3) a group of private practice physicians in a New England urban area.¹¹ Patients were eligible if they were older than 30 years and were prescribed either insulin or sulfonylureas, or if they met 1 of the following laboratory criteria: (1) a fasting plasma glucose level greater than 7.8 mmol/L (>140 mg/dL), (2) a random plasma glucose level greater than 11.1 mmol/L (>200 mg/dL), or (3) a glycosylated hemoglobin (HbA_1c) level greater than 3 SDs from the mean.

At 2 sites (HMO and private practice), physicians were randomly selected (within age and sex strata), but a universal sample of primary care clinic physicians were selected at the third site. Physicians reviewed a list of eligible patients with diabetes under their care who were identified by means of pharmacy and laboratory databases (at the HMO and teaching clinic sites) or from their records (private practice site). The physicians excluded patients who did not have type 2 diabetes, were too ill for follow-up, or who either did not speak English or did not have a family interpreter (9% of the patients were excluded). Of the patients who were contacted by telephone and requested to participate in the study, 18% declined. An additional 15% failed to return their baseline surveys. The final cohort consisted of 3642 patients, 1730 from the HMO site, 787 from the university teaching clinic site, and 1125 from the private practice site, who met the eligibility criteria described above. The 232 physicians across all 3 sites had an average of 16 patients who responded to the survey, with physicians averaging 21 patients at the HMO site, 9 patients at the urban teaching hospital, and 18 patients at the private practice site. Less than 5% of the physicians were endocrinologists. Patients were asked in the baseline survey to confirm that we had correctly identified the physician primarily responsible for their diabetes care.

At the HMO site, extensive information was available from medical information systems including hospital discharge and visit records and clinical laboratory and pharmacy systems.¹² Analyses that used medical information systems–based measures such as total laboratory utilization and the results of HbA_1c tests are thus limited to this site.

Dependent Variables. Resource utilization measures were collected from a self-administered survey and included total hospitalizations and total number of office visits in the previous 6-month period. Laboratory utilization measures were constructed at the HMO site where we had access to all of the laboratory records. Each test was mapped to a relative value unit constructed to reflect actual laboratory costs (in 1992 dollars) for performing each test.^12- 13 Diabetes control in 1991 and 1992 was based on the average HbA_1c level each year at the HMO site.¹²

Independent Variables. The variables used to predict utilization and diabetes control included demographic variables (patient sex and age), physician and site, socioeconomic status (including income, education, and employment status), duration of diabetes, and health status measures. We measured generic health status using the 36-item Medical Outcomes Study Short-Form Health Survey and comorbidity using a previously described and validated diabetes-specific instrument, the Total Illness Burden Index.^12,14 This measure uses what patients have reported about all of their diseases and symptom intensity to characterize the total disease burden, including the presence and severity of diabetic complications.

Our overall analytic strategy was to construct case-mix–adjusted regression models for the resource utilization and glucose control–dependent variables. The case-mix–adjusted residuals from these models were then examined for how much they varied systematically by physician. This 2-step approach represents current profiling practice.^3,15 However, we confirmed our results using hierarchical regression for general linear models, an analytic technique less commonly used but more appropriate for this instance, for which patient observations are not independent but are clustered by physician.¹⁶

In the first set of analyses, case-mix adjustment included only age and sex, which are often the only patient characteristics that can be easily obtained from insurance company databases. Next we developed a full case-mix model, which included all of the independent variables described above. We used linear regression for the continuous dependent variables (laboratory relative value units and HbA_1c levels) and negative binomial regression for counts (hospitalizations and visits).¹⁷ Variables for each site were included to remove site effects. In the 2-step approach, we looked for evidence of physician practice effects by analyzing whether residuals from these case-mix–adjusted models were associated with the physician identifier. We used random effects analysis of variance to quantify the variance between and within physician panels. In the hierarchical model approach, the physician level variance (along with its SD) is estimated as a separate parameter in the full case-mix model.

Quantifying the Amount of Physician Variation. The usual assessment of the amount of variation in physician practice patterns simply tabulates the average patient residual after case-mix adjustment by physician.¹⁵ However, even with the most sophisticated and expensive case-mix adjustment, the above analysis will overstate (substantially in some cases) the true range of a physician practice effect. This is particularly true when the patient panel size is small or the physician effect on the profile is relatively small.^16,18- 21 To account for a small signal (in this case, physician effect) to noise (variation due to unmeasured patient factors) ratio, a "shrinkage factor" must be applied to each physician's estimated profile. (A mathematical model describing this concept in the setting of constructing a physician profile measure may be obtained from the authors.)

The shrinkage factor can be thought of as adjusting the performance measure for the level of reliability. The lower the reliability, the more the physician's measure should be shrunk toward the mean of all the physicians. A statistic that describes the proportion of overall variation attributable to physician practice, after accounting for the shrinkage described above, is the intraclass correlation coefficient (ICC).^16,22 The ICC can be thought of as the maximum proportion of variance that can be explained by physician practice patterns²³ or as a reliability-adjusted R². It was calculated from the estimates of variance between and within physician panels.

Quantifying the Reliability of a Physician Measure. Reliability is the extent to which a measure gives the same result on repeated trials. Practically, reliability is often calculated as the correlation between 2 equivalent (but not identical) measures, such as items on a test scale.²³ For a report card measure of physician effect on glucose control, for example, each patient's level of glucose control, after case-mix adjustment, is considered an equivalent measure of the physician's effect on glucose control. The correlation between patient measurements within physician (the intra-class correlation coefficient or ICC) is thus the reliability of an estimate of the physician effect on glucose control based on a single patient measurement. The reliability of a physician profile, composed of a mean of n patient measurements is calculated as a function of n and the ICC using the Spearman-Brown prophecy formula.^23- 24 A reliability of 0.80 suggests that 80% of the variation of an individual physician's profile is due to practice differences and 20% is due to chance variation and is often considered the minimum level necessary for making decisions about individual physicians based on a profile.²⁵ The reliability of a profile increases as the physician panel size increases and as the difference in practice patterns between physicians becomes larger.

Manipulating Profiles. Proponents of profiles argue that good case-mix adjustment can eliminate the advantage of caring for less sick panels of patients. For profiles of glycemic control at the staff-model HMO, we tested the effect of deliberate patient selection. We collected average HbA_1c levels of each patient for the years 1991 and 1992. After excluding physicians with fewer than 4 study patients, we calculated physician profiles for 1991 data and identified as outliers the 10% of physicians with the worst level of glycemic patient control after case-mix adjustment. (While there is no standard cutoff, those generating profiles frequently exclude profiles for providers with particularly small panel sizes.) Before estimating the 1992 profiles, we dropped from the panels of the physicians who were outliers in 1991 the few patients with the worst glycemic control in 1991 (above the 95th percentile overall) replacing them with patients who had average control in 1991. This simulates the effect of a physician eliminating the few (1-3) patients with the worst glycemic control as a strategy to improve their profile.

All of the above analyses were performed using the Stata Statistical Software Package^22,26 with the exception of the hierarchical models, which were estimated using MLwiN multilevel modeling software.²⁷

Table 1 shows the demographic characteristics of the patients with diabetes at each of the 3 sites. While age distributions were similar, there was a much larger percentage of black patients at the urban teaching clinic (64% vs 4%), as well as a lower percentage of married patients. Patients at the urban teaching clinic also had lower income, less education, and lower employment rates than those at the other 2 sites.

As shown in Table 2, the urban university teaching clinic site had almost twice the average number of hospitalizations per patient per year as the HMO site after full case-mix adjustment. The private practice site had about 3 patient visits per year more than the other 2 sites, a difference that remained significant after full case-mix adjustment (Table 2).

The profiling approach most commonly used by payers and administrators is to calculate simple age- and sex-adjusted measures that are averaged by physician to generate a physician profile. Using this approach, it would appear that 13% of the variation in outpatient visit rates and 8% of the variation for hospitalization rates are attributable to the physician (by an analysis of variance of the age- and sex-adjusted residuals, Table 3). However, accounting for reliability, particularly the low signal (physician practice effect) to noise (patient variability) ratio, and more extensive case-mix adjustment, a better estimate of the maximum possible amount of variation due to differences among physicians is closer to 4% for outpatient visits and 1% for hospitalizations (as represented by the ICC, Table 3). This implies that after adjusting for socioeconomic status, comorbidity, and health status, 99% of the variation in hospitalization rates and 96% of the variation in outpatient visit rates is due to unmeasured patient factors or chance. Eliminating all physician variation would have a relatively small effect on overall patient visit rates or hospitalization rates.

Using hierarchical analyses, we can test in a more appropriate statistical model whether this remaining physician effect is statistically significant. After accounting for both the low signal-to-noise ratio of the physician effect and full case-mix adjustment, the physician level variation in visit rates is still statistically significant (χ² 6.6; P=.01). In contrast to the case of outpatient visits, the physician effect for hospitalization rates is not statistically significant (P=.13).

Although the overall physician practice effect for visit rates is statistically significant, the reliability of an individual physician visit rate (based on a panel with 16 patients) is only 0.40 Table 3). Thus, more than 60% of the variation in the median physician's patient-visit–rate profile (1 − reliability) is due to error from chance variation. Figure 1 illustrates that this level of physician variation and panel sizes result in physician profile measures that are inadequately precise to distinguish physicians from one another. The vast majority of visit rates are within 10% (or 1 visit per year) of the mean. Even at the extremes, the confidence limits are wide enough that it would be difficult to say that any physician has a rate significantly different from any of the average physicians. At this level of physician effect (ICC=4%), physicians would need to have more than 100 patients with diabetes for the profiles to have a reliability of 0.80 (Figure 2). At the HMO site, we could identify the total number of patients with diabetes per physician via medical information systems, even if the patients were not enrolled in the study. There, the median primary care physician had 29 total patients with diabetes, 90% had fewer than 60, and no physician (out of more than 250 at the HMO) had more than 85.

Physician visit and hospitalization rates may be influenced by many factors and perhaps are less determined by physician practice than profiles based on measures more closely tied to specific physician interventions. Thus, we next examined whether profiles of total laboratory utilization and the average level of glycemic control, constructed from computerized laboratory data available only at the HMO site, would be more reliable measures of physician practice differences.

For age- and sex-adjusted total laboratory costs (laboratory relative value units), physician practice style appears to account for 7% of the variation in laboratory costs. However, after full case-mix and reliability adjustment, the ICC was only 2.6%, in the middle of the 1% to 4% ICC range found for hospitalizations and visits, respectively. The reliability of the median physician's profile (21 study patients per physician at the HMO) was 0.38 after full case-mix adjustment.

The percentage of variation accounted for by physician practice differences or ICC can be difficult to interpret at a practical or intuitive level. A more direct illustration of the magnitude of the physician effect on total laboratory cost and how the usual physician profiles shrink dramatically after adjusting for reliability is shown in Figure 3. The usual physician profiles (shown on the left side of Figure 3) would suggest that if the physicians in the 90th percentile of utilization were able to change their practice habits to be similar to the practice of physicians in the 10th percentile, the HMO could realize an annual savings of $113 for each of its patients with diabetes. However, after adjusting the estimates for panel size and reliability, the potential annual cost savings resulting from changing the highest to the lowest decile is closer to $40 per patient.

Similarly, for profiles that measure the level of patients' glycemic control, when applying a simple age- and sex-adjusted model, the physician identifier appears to account for 8% of the total variance in level of control (or a range of 2 percentage points between the average HbA_1c level achieved by patients of physicians in the 10th percentile vs those in the 90th percentile). However, the reliability and case-mix adjusted physician effect is only 3.3%. Thus, the better estimate of the true range of physician practice between the 10th and the 90th percentiles is 0.7 percentage points and the reliability of an individual physician's HbA_1c profile is 0.38 (for physicians with 21 patients).

Does Full Case-Mix Adjustment Prevent Gaming the System? Ideally, full case-mix models would eliminate or reduce the perverse incentive for physicians to manipulate profiles by electing not to care for sick patients. However, in Figure 4, we see if those physicians with the worst profiles (patients with higher than expected HbA_1c levels) for 1991 managed to discourage the patients with the top 5% of HbA_1c levels (representing only 1-3 patients per physician) from returning to their panel, they would in most cases achieve a panel HbA_1c profile in 1992 that would be substantially improved than average. About half of this improvement occurs through regression toward the mean (determined by examining 1991 and 1992 profiles without any patient selection for the 1991 outlier physicians) but the other half was due to patient selection. Thus, the patient's HbA_1c levels from the previous year proved a far better predictor of what a patient's HbA_1c level would be in the current year, better than physician practice or our case-mix adjusters. Manipulating their patient pool, based on a patient's prior year HbA_1c level, is the easiest way for physicians to have a substantial improvement in their profile.

If profiling is to be successful at providing consumers and purchasers of health care services with a way to monitor the quality of health care, then profiles must cover more than the delivery of preventive health services to healthy people, such as vaccinations and cancer screening rates, which were the early focus of such profiling efforts as the Health Plan Employer Data and Information Set.^3,5 Diabetes would seem to be an ideal chronic condition to monitor because the disease causes major morbidity, is quite common, and is expensive to treat. Furthermore, in contrast to some areas in medicine, there is good evidence that diabetes care interventions can substantially affect patient outcomes and complications.²⁸

Although we found some differences between our 3 sites that were not attributable to case-mix differences, relatively little of the variation in any of the resource utilization or glycemic control measures evaluated was due to individual physician practice style variation. The usual approach to generating physician profiles (averaging case-mix–adjusted patient measures within each physician's panel) exaggerates both the magnitude of physician practice differences and the savings that could be achieved by correcting the practice of the outlier physicians to that of the average physician. Even if this amount of variation at the physician level is considered to be clinically important, the profiles of individual physicians are not very reliable. This lack of reliability is due to a combination of the small physician effect (relative to the substantial patient variations only a small portion of which is captured by even detailed case-mix measures) and the size of individual physician patient panels.

It is possible that at other sites physician practice effects might be more dramatic. We sampled patients from only 3 sites. However, these sites represent the different types of organizations providing health care now, and the patients in our study have a wide range of socioeconomic backgrounds. We did not study retinal or foot examination rates, which are considered to be important measures of quality of diabetes care. As such examinations reflect a fairly specific process of care, it is possible that they may have a larger component of physician variation.²⁹ Our results relate to only 1 disease. Again, it is possible that, for other diseases, differences among physicians may be larger. However, diabetes is one of the most common diseases in the United States. Apart from hypertension, it is difficult to imagine that there would be enough cases per primary care physician to construct disease-specific profiles for almost any other chronic condition. At the HMO site, none of the more than 250 primary care physicians had more than 85 patients with diabetes. For any of the measures that we examined, at least 100 patients would be needed to reach 80% reliability (often considered the minimum for making decisions about individuals).²⁵ Finally, our hospital and visit-rate data are by patient self-report, but the findings are similar for our other measures of use and laboratory costs, which are based on computerized records.

How do our findings for diabetes compare with the findings of other diseases, which have been examined specifically for physician-level variation?

For inpatient resource utilization at 1 academic center, attending and resident physicians accounted for only about 2% of the variation in total resource utilization for hospitalized medical patients³⁰ and 1% of the variation in total resource utilization for vascular surgery.³¹ Feinglass et al¹⁵ reported what appears to be a large physician practice effect at an urban hospital. Indeed, they suggested that annual patient charges could be reduced by $250,000 if the 10 high resource-use physicians practiced at the average resource-use level for all physicians. However, the proportion of variance accounted for by physician appears from their tables to be at most 3.0%, and these potential savings very likely are overestimated when using the usual profiling approach. Adjusting for the low reliability of the physician effect, an estimate of the savings that could be achieved at that hospital is closer to $65,000 (and this is for patient charges, actual savings would be less). Miller et al³² found similarly little difference in how physicians practice for several specific measures of outpatient resource utilization in the general medical clinics at a university teaching hospital.

Only for a few of the most specific clinical indicators measuring processes of care did we find good evidence in the literature of larger amounts of physician-level variation as a proportion of the total variation in a profile measure. Orav et al,³³ found that the practitioner accounted for a maximum of about 24% of the variance in a process of care score related to the management of digoxin and a minimum of 3% in process scores related to cancer screening.³³ Although there was more evidence for variation at the physician level when examining a few of these very specific processes of care, they also noted that there was essentially no correlation between provider performance on 1 guideline and their performance on any of the others. Thus, any more global quality score would average out differences between physicians and consequently would have low reliability. Unfortunately, very specific process measures will usually apply to only a small fraction of a physician's patients, which makes the process of profiling even more difficult.

In summary, most of the published evidence suggests that the individual physicians rarely account for more than 4% of the variation in common profile measures after case-mix adjustment. It might be useful to profile and control this relatively small amount of physician variation, but only if the costs of intervening are worth the expected gains. For a utilization measure, 4% of the variance may represent a great deal of money. However, given this relatively small physician-specific effect, there might be much more value in seeking factors that affect utilization, satisfaction, and clinical efficacy among all patients more substantially than the practices of their particular physician. Perhaps system and cohort effects minimize differences between providers within sites or groups of physicians, and larger differences may be found between sites or regions of the country.

Given the small amount of variation attributable to the physician and the larger amount attributable to the patient's prior utilization or experience,³⁴ it will generally be much easier for physicians to change their profile results by manipulating their patient populations than by improving their efficiency or quality. Uncertainty about the consequences of managed care deselecting decisions (decisions to terminate a contract with a physician) and the role of profiles in these decisions has created an environment in which the "the soundest strategic advice for physicians has been simply to make every effort to avoid deselection."⁶ Unfortunately, an easy way to ensure a good profile is to refuse to care for sick patients, those who have failed therapy, or those who do not adhere to treatment plans. Our results suggest that, at least for patients with diabetes, even the most sophisticated case-mix adjustment will not eliminate this strong perverse incentive, similar to the incentives that may encourage HMOs to attempt to select healthy populations when enrolling up capitated patients.^34- 37

What conclusions should we draw from the results of our study? Those who produce physician profiles should first make a realistic assessment of the reliability of those profiles. The reliability depends on both panel size and how much physicians vary in their practice. Some specialists may have very large panel sizes, which may allow for reliable profiling, but it is then important to ask if the differences between physicians are worth profiling. Profiling should be considered only if the physician-level variation is deemed important relative to other potential sources of variation. At that point, more complete case-mix measures will be necessary. Age- and sex-adjustment, while inexpensive, is little better than no case-mix adjustment at all. For profiling, more detailed case-mix adjustment, such as the Total Illness Burden Index and the Medical Outcomes Study Short-Form Health Survey, will be needed and adjustments for patients' past use or values may also be necessary.³⁸ Finally, profilers must consider that the application of profiles may foster an environment in which deselection of patients is the easiest way for physicians to avoid becoming deselected themselves. In our opinion, those who implement the profiling system would be as responsible for this result as those physicians who have succumbed to these pressures by denying care to the sickest and most vulnerable people needing medical care.