Relationship Between Low Quality-of-Care Scores and HMOs' Subsequent Public Disclosure of Quality-of-Care Scores FREE

Employers,¹ government purchasers of health insurance,² individual consumers,³ and lawmakers^4- 5 are seeking more information on the quality of health care. Recently, the President's Commission on Consumer Protection and Quality in the Health Care Industry called for widespread public disclosure of quality data by all health care provider organizations including health plans.⁶ Public disclosure is seen as a way to enhance informed consumer decision making,⁷ promote quality improvement,^8- 10 and increase health plans' accountability for health care delivery.^6,11- 14

Public disclosure of data on quality by health maintenance organizations (HMOs), except those enrolling Medicare patients, is voluntary. In 1998, only 32.5% of all HMOs disclosed their scores on the National Committee for Quality Assurance (NCQA) Health Plan Employer Data and Information Set (HEDIS) measures,¹⁵ the most widely used set of quality indicators. If health plans that refuse to disclose quality data provide inferior care, publicly available data would overstate the average quality of HMO care nationally and result in a distorted picture of how a given plan that discloses quality data compares with that average. Selective nondisclosure could also undermine public accountability and quality improvement efforts by weakening the impetus to improve quality.

Despite the importance of this issue, no peer-reviewed studies have examined the relationship of HMO quality to willingness to disclose quality scores. We linked data for multiple years from the NCQA's annual Quality Compass databases to determine if withdrawal from public disclosure of HEDIS scores was related to an HMO's HEDIS performance 1 year earlier.

The NCQA currently uses HEDIS measures, a standardized set of clinical quality indicators, as the principal clinical criteria for its HMO accreditation program. Health plans voluntarily submit these data to the NCQA. The NCQA lists HEDIS scores of individual HMOs in its annual Quality Compass database, designed for use by health insurance purchasers and consumers. Until recently, the NCQA allowed plans to decline public disclosure of their HEDIS scores, yet remain fully eligible for NCQA accreditation. Plans may also disclose data privately (eg, to large purchasers) but refuse public disclosure.

To determine which HMOs that disclosed HEDIS scores in 1997 (the "1997 cohort," n = 329) or 1998 (the "1998 cohort," n = 292) withdrew from public disclosure in the subsequent year, we linked the 1997¹⁶ (the first year of use of HEDIS version 3.0), 1998,¹⁷ and 1999¹⁸ Quality Compass databases (which reflect plan characteristics and performance in 1996, 1997, and 1998, respectively). We used a "link-file" database provided by the NCQA to assist in tracking plans in the Quality Compass databases from year to year, since name changes were common. In addition to identifying plans that withdrew from public disclosure, we identified plans that merged or closed from one year to the next. We also identified HMOs that newly began public disclosure in 1998 or 1999. Last, we telephoned each HMO that we identified as having withdrawn from public disclosure to confirm the plan's identity and whether it had changed its name, merged with another plan, or closed. For the single plan that had closed, we obtained the date of closure from its parent company.

The NCQA requires HMOs to follow a detailed guide that defines each HEDIS measure and specifies standards for data collection. Plans may garner data from administrative records (administrative method) or supplement the administrative method with chart reviews (hybrid method).

For each quality indicator, the plan first draws a sample from the target population (eg, for mammography, women aged 52-69 years continuously enrolled in the HMO for at least 1 year). The HMO then searches administrative records (eg, payment or radiology files) to determine if the intervention occurred within a set time frame (eg, 2 years for a mammogram). If no evidence of the intervention is found, the HMO may choose to search for exclusions (eg, a history of bilateral mastectomy). For the hybrid method, when administrative records fail to give evidence either of the intervention or an exclusion, the plan reviews sampled patients' charts for such evidence. The HEDIS score is calculated as the number of patients who received the intervention divided by the number of eligible patients. The hybrid method, used by most plans for most measures, usually results in higher quality scores.

The NCQA's Quality Compass databases contain information on other health plan characteristics including ownership status (ie, investor-owned vs not-for-profit). When data on ownership status was missing, we consulted InterStudy's HMO Directory^19- 20 and/or telephoned the plan to determine ownership status.

We assessed HMO quality using all HEDIS measures listed under the NCQA's rubric, "effectiveness of care." This rubric encompasses 13 distinct measures for the 1997 cohort, 4 of which are rates for individual childhood immunizations (measles-mumps-rubella, hepatitis B, diphtheria-pertussis-tetanus, and oral polio virus), and one of which is a rate for completion of all of these 4 childhood immunizations. For the 1998 cohort, vaccination for varicella and Haemophilus influenzae type B were added as measures and are included in the rate for completion of all recommended childhood immunizations that year. To avoid giving undue weight to childhood immunizations, we analyzed only the combined immunization rate, yielding 9 HEDIS scores for each plan. We ranked HMOs by quality in 2 ways. First, we ranked HMOs according to their score on each of the 9 HEDIS measures separately. Second, we ranked HMOs based on the average of ranks for all individual HEDIS measures for which the plan submitted data. For this latter analysis, we included only plans reporting scores on at least 5 of the 9 HEDIS measures. When more than 1 plan reported the same score, we assigned these plans the same rank. We then divided the plans into tertiles on the basis of their quality ranks. All analyses were performed using SAS.²¹

Our primary outcome was withdrawal from public disclosure of HEDIS scores 1 year after a previous public disclosure. We defined withdrawal as either (1) a failure to submit any HEDIS scores to NCQA or (2) submission of HEDIS scores but refusal to allow public disclosure. Plans that disclosed even a single HEDIS score or that merged and disclosed pooled HEDIS scores were not considered to have withdrawn. We excluded from our analysis the single plan that closed.

For each of the 9 separate HEDIS measures we classified plans by whether their scores fell in the highest, middle, or lowest tertile of the 329 plans publicly disclosing data in 1997. We then calculated for each quality tertile the proportion of plans that withdrew from disclosure 1 year later and used the χ² test to compare the proportions withdrawing in the highest and lowest tertiles. We report 2-tailed P values for all tests.

We repeated this analysis using the 1998 cohort (the 292 plans disclosing data in 1998), classifying plans according to their quality ranks in 1998 and comparing withdrawal rates 1 year later among plans in the highest- and lowest-quality tertiles.

Thus, each analysis examined whether the quality rank in a given year predicted the likelihood of publicly disclosing quality scores in the subsequent year.

We also used multiple logistic regression to estimate the adjusted odds ratio (OR) for withdrawal from public disclosure for HMOs in the lowest vs highest tertile of average plan rank for all 9 measures combined. We considered plan characteristics (model type, geographic location, and method of data collection) as potential covariates. The final multivariate models included only those variables that showed a significant univariate association (P<.05) with the outcome in both cohort years. In addition, because collecting data by the hybrid method produces higher HEDIS scores than the administrative method,²² we controlled for the method of data collection in all multivariate models.

Because our previous research had shown that investor-owned plans achieve lower HEDIS scores than not-for-profit plans, we also explored the interrelationships among ownership status, tertile of average rank on HEDIS scores, and the likelihood of withdrawal from public disclosure with 2 × 2 contingency tables and χ² tests of significance for both the 1997 and 1998 cohorts. Specifically, we compared investor-owned and not-for-profit plans with regard to the percentage of plans in the lowest tertile of average HEDIS rank and the percentage of plans that withdrew from public disclosure. Last, we compared plans in the upper and lower tertiles of average HEDIS rank with regard to the percentage of plans that withdrew from public disclosure among investor-owned and not-for-profit plans separately.

Finally, to quantify the clinical significance of differences in quality between the highest- and lowest-quality plans, we calculated the mean (SD) rates for each indicator for the highest- and lowest-quality tertile.

The majority of HMOs in both the 1997 and 1998 cohorts were investor-owned and were independent practice associations or mixed model type plans (Table 1). Plans in both cohorts were most commonly located in the South Atlantic, Mid Atlantic, and East North Central regions.

A total of 329 HMOs allowed public disclosure of their HEDIS scores in 1997 (Figure 1). Of these plans, 161 (49%) withdrew from public disclosure the following year. In 1998, 292 plans allowed public disclosure of their HEDIS scores. This cohort consisted of 162 plans (after mergers) that allowed public disclosure in 1997, plus 130 newly participating plans. Of these 292 plans, 67 (23%) withdrew from public disclosure in 1999. For both the 1997 and 1998 cohorts, just over half of all plans that withdrew from public disclosure continued to submit HEDIS scores to NCQA (Figure 1).

HEDIS scores among HMOs that allowed public release of their quality data varied widely in both 1997 and 1998 (Table 2). Absolute differences in mean HEDIS scores of plans in the lowest and highest tertiles ranged from 15.6 to 42.3 percentage points in the 1997 cohort and from 14.6 to 37.5 percentage points in the 1998 cohort. For example, the mean immunization completion rate for 13-year-olds in the 1997 cohort was 74.7% for plans in the highest-quality tertile, but only 32.4% for plans in the lowest-quality tertile.

Health maintenance organizations in the lowest tertile were significantly more likely to withdraw from public disclosure than plans in the highest tertile for 7 of the 9 measures in the 1997 cohort and for 6 of 9 measures in the 1998 cohort (Table 3). Plans in the lowest tertile were 1.6 to 2.7 times more likely to withdraw from public disclosure than plans in the highest tertile in the 1997 cohort and 2.2 to 7.0 times more likely to withdraw in the 1998 cohort. For 7 of the 9 indicators in the 1997 cohort, more than half of plans in the lowest-quality tertile withdrew from public disclosure of HEDIS scores the subsequent year. Withdrawal rates for the 1998 cohort were somewhat lower; nonetheless, for 8 of 9 indicators, at least 25% of plans in the lowest-quality tertile withdrew the subsequent year.

Health maintenance organizations in the lowest tertile of overall quality (average rank for all 9 HEDIS measures) were more likely to withdraw from public disclosure than plans in the highest tertile in both the 1997 (OR, 3.6; 95% confidence interval [CI], 2.1-7.0) and 1998 (OR, 5.7; 95% CI, 2.7-17.7) cohorts after adjustment for the method of data collection and plan model type (the only plan characteristic consistently correlated with plan withdrawal in univariate analyses).

In our analyses according to plan ownership status, investor-owned plans were more likely than not-for-profit plans to be in the lowest-quality tertile in both the 1997 (RR = 3.4; 95% CI, 1.9-5.8) and 1998 (RR = 1.9; 95% CI, 1.3-2.7) cohorts and to withdraw from public disclosure in both the 1997 (RR = 5.7; 95% CI, 2.6-12.2) and 1998 (RR = 1.3; 95% CI, 0.7-2.5) cohorts, although this difference was not statistically significant for the latter cohort. It appears, however, that poor quality rather than profit status per se was the primary determinant of withdrawal from public disclosure. The poorest-quality plans were more likely to withdraw from disclosure than the best-quality plans among both investor-owned plans (RR = 1.5; 95% CI, 1.1-2.1), and not-for-profit plans (RR = 2.2; 95% CI, 0.5-10.4) in the 1997 cohort. Similar results were obtained in the 1998 cohort for both investor-owned (RR = 2.7; 95% CI, 1.1-6.6) and not-for-profit plans (RR = 20.0; 95% CI, 2.8-149.8).

While the total number of HMOs that publicly disclosed HEDIS quality scores changed little each year from 1997 to 1999, the composition of this group changed substantially. Forty-eight percent of plans in the 1997 cohort and 23% of plans in the 1998 cohort withdrew from public disclosure 1 year later. Quality scores varied substantially among HMOs, and lower-scoring plans were much more likely to withdraw.

No previous peer-reviewed studies have examined the relationship between HMO quality scores and withdrawal from participation in public disclosure of scores in the HEDIS program. Previous yearly NCQA reports have documented that nondisclosing plans score poorly. However, our longitudinal analyses provide a quite different view than these reports based on cross-sectional data. Our approach encompasses plans that drop out of the HEDIS program entirely, in addition to those that refuse disclosure. For example, a 1998 NCQA report that provided data on nondisclosing plans included only 88 of the 161 (nondisclosing and drop-out) plans we analyzed.

We also delineate, for the first time, the shifting cohort of HEDIS participants and disclosers. Many plans are disclosers one year and nondisclosers the next (or vice versa). Hence the manipulation of the HEDIS monitoring system is more pervasive than is apparent from the NCQA's cross-sectional comparisons. The differences in analytic approaches also give rise to quite different interpretations. The NCQA suggests that the better cross-sectional performance of disclosing plans is evidence that their quality monitoring system is working. In contrast, our longitudinal data imply that gaming of the system is so extensive as to potentially undermine the quality monitoring process.

Why do HMO executives at lower-scoring plans choose to withdraw from reporting of HEDIS scores? They might believe that their plan's low HEDIS scores result from inadequate data collection methods that could understate true quality. Perhaps some suspect that their plan will suffer from biased comparisons since not all plans' data were audited, especially in the earlier years of the HEDIS program. They may become aware of such issues only after disclosing HEDIS scores at least once. Some executives may regard the costs of data collection as too high,^23- 25 which could explain why some higher-scoring plans withdrew from HEDIS participation. But costs cannot explain most withdrawals; about half of the plans withdrawing from public disclosure still collected and submitted HEDIS scores (Figure 1).

The most likely explanation for our findings is that many plans withdraw because they fear (or know) that they will score low again. Low scores might place such plans at a marketing disadvantage, especially if nondisclosure carries little stigma. Regardless of the explanation, however, our results imply that voluntary disclosure of quality data, the primary national mechanism for HMO quality oversight, is failing to meet its stated goals of informing consumer decision making, providing incentives to improve quality and increasing public accountability.

The NCQA's HEDIS program represents the most comprehensive and influential quality assessment tool²⁶ for HMOs (and any health care sector). HEDIS measures are standardized and subject to external audit to verify the data collection and calculation process.²² Yet, the selective withdrawal by lower-scoring plans means that enrollees, purchasers, and the public often cannot monitor a plan's quality over time. Furthermore, it implies that the average quality of HMO care in the United States is unknowable. Average published HEDIS scores could improve even if the actual average quality were stable or even deteriorating. Hence, HEDIS scores cannot be used as an accurate barometer of HMOs' attainment of specific health goals for the nation.^11,27- 28

The variation in HEDIS scores that we observed between the highest- and lowest-scoring plans has substantial clinical relevance. For example, receiving a β-blocker after a myocardial infarction reduces the risk of cardiovascular death and nonfatal reinfarction by 22% and 27%, respectively.^29- 30 Yet in 1998, a patient surviving a myocardial infarction was only half as likely to receive this medication if enrolled in a health plan in the lowest compared with the highest tertile of quality.

Voluntary disclosure allows HMOs to use the HEDIS program as a marketing tool, sacrificing its value as a quality assessment and improvement tool. When scores are high, plans can disclose them and take advantage of consequent marketing benefits. When scores are low, plans can withdraw from public disclosure. Indeed, until recently, HMOs that refuse to publicly disclose their quality scores were fully eligible for NCQA accreditation; only 4% of HMO applications for accreditation were rejected in 1998.³¹

Investor-owned plans were somewhat more likely to withdraw from public reporting. However, poor quality was associated with withdrawal from public disclosure among both investor-owned and not-for-profit plans. Apparently, the increasingly competitive health care marketplace drives health plans (irrespective of ownership status) to control data release to maximize competitive advantage.

Lack of disclosure is not the only challenge to the HMO quality oversight process. Patients appear to have difficulty understanding quality data^32- 33 and use it infrequently when selecting a health plan.^34- 35 Many employers offer only 1 health insurance option,³⁶ foreclosing patient choice. Even large employers make only limited use of quality data,^1,8 instead, selecting health plans primarily on the basis of cost.^1,37 Indeed, in the current health care market, evidence that public disclosure of quality data improves quality is equivocal.^{9- 10,38- 39} Therefore, improving accountability and encouraging quality improvement would require, at the very least, that quality data be presented in a patient-friendly format, that patients be offered a choice of health plans, and that both patients and large purchasers make purchasing decisions based on quality rather than price. Without publicly available data on quality, however, achieving these goals would accomplish little.

Our findings should also be viewed in the context of the broader debate on public disclosure of quality data by all types of health care provider organizations. Like HMOs, physicians and hospitals have opposed mandatory disclosure of performance data. Improvement of quality and accountability may ultimately depend on forthright disclosure of quality data at all levels of the health care system.

Our study has several limitations. First, since data on nondisclosing plans were, by definition, unavailable to us, we used a plan's performance 1 year earlier as a proxy for current performance. Plans know their current HEDIS scores before having to decide whether or not to disclose them. Thus, it seems likely that low-scoring plans that improved would choose to continue disclosing, while those that did not would be more likely to withdraw from disclosure. Hence, our study may underestimate the relationship between low scores and withdrawal from disclosure. Second, we cannot exclude the possibility that unmeasured plan characteristics such as geographic dispersal of medical provider sites or differences in data systems could systematically influence our results. Third, although our data show that lower-scoring plans are more likely to withdraw from disclosure, we have no direct data on HMO executives' reasoning regarding this decision. Last, only HMOs are currently eligible to submit HEDIS scores and receive NCQA accreditation. Whether the selective reporting of quality data we observed would apply to fee-for-service insurance or other facets of the health care system is unknown.

Few industries whose impact on health rivals HMOs' are as free of public oversight. Airlines and car manufacturers are required to disclose standardized data on the safety of their products. Our findings suggest that voluntary quality reporting by HMOs will not create the preconditions for effective quality oversight. Reporting and public disclosure of HEDIS and other meaningful quality data by HMOs should be mandatory.