Comparison of Change in Quality of Care Between Safety-Net and Non–Safety-Net Hospitals FREE

Numerous studies have documented that health care quality in the United States is often inadequate.^1- 2 In an effort to improve quality, policy makers have created incentive programs designed to stimulate quality improvement, such as public reporting of quality measures and pay for performance.

Prior work has documented that hospitals that serve low-income patients and racial and ethnic minorities often have lower quality of care.^3- 5 These studies provide an important insight into a source of disparities across racial, ethnic, and socioeconomic groups (ie, that disparities stem in part from location of care) and suggest that improving quality at low-performing hospitals could reduce disparities. Nonetheless, some policy makers and clinicians have expressed concern that public reporting and pay for performance may have unintended and negative consequences, including the potential to worsen disparities.^6- 8

In theory, quality improvement incentives could exacerbate existing disparities and reinforce a 2-tier system of health care delivery in several ways. First, quality improvement may require investing resources, for example, to adopt information systems or increase hospital staffing.^9- 10 Hospitals that predominantly treat poor and underserved patients—ie, safety-net hospitals—may be disadvantaged if they lack the resources necessary to improve or even to ensure accurate data collection for performance measurement. Second, safety-net hospitals may be penalized under incentive systems that reward the highest performers because they have lower performance at baseline.^3- 5 Additionally, safety-net hospitals do not have the same payer mix as other hospitals and often are inconveniently located from the perspective of insured patients. Therefore, safety-net hospitals may not get the economic benefits from public reporting and pay for performance that other hospitals do.¹¹

While public reporting and pay for performance have several potential benefits,^8,12 it is generally accepted that these should not be sought at the expense of worsening disparities.^7,13 To help inform these quality improvement efforts, we examined changes in disparities of quality of care between safety-net and non–safety-net hospitals. Using data that included almost all US hospitals, we estimated whether disparities in care have changed over time. We also estimated how a hospital pay-for-performance system currently being piloted by the Centers for Medicare & Medicaid Services (CMS) might differentially affect the financial performance of safety-net hospitals.

The CMS publicly reports hospital performance on its Web site, Hospital Compare (http://www.hospitalcompare.hhs.gov/). Hospitals with performance among the top 10% of hospitals nationally are so designated. Participation in Hospital Compare is voluntary. However, hospitals that choose not to participate forfeit a percentage of the annual Medicare fee schedule update. Therefore, 4464 of 4677 US hospitals (95%) participated between January 1, 2004, and December 31, 2006. Compared with hospitals that did participate, the 213 hospitals that did not were more likely to be for-profit (52.6% vs 15.8%) and less likely to be critical-access hospitals (8.0% vs 18.6%), to not have emergency services (54.0% vs 5.6%), and to not be accredited (53.1% vs 22.6%) (P <.001 for all comparisons).

We evaluated hospital performance using the first 3 years of Hospital Compare data (2004 to 2006) and all 10 measures that have been included throughout this period. These measures cover 3 clinical conditions: acute myocardial infarction (AMI), heart failure, and pneumonia. Five measures relate to AMI: use of aspirin within 24 hours and prescribed at discharge, use of β-blockers within 24 hours and prescribed at discharge, and use of angiotensin-converting enzyme inhibitors or angiotensin receptor blockers for left ventricular dysfunction. Two measures relate to heart failure: assessment of left ventricular function and the use of angiotensin-converting enzyme inhibitors or angiotensin receptor blockers for left ventricular dysfunction. Three measures relate to pneumonia: the time to initial administration of antibiotics, pneumococcal vaccination, and assessment of oxygenation.

For each measure, performance is calculated as the percentage of eligible patients who received the indicated care. Hospital performance was also assessed using a condition-specific opportunity model composite measure, the same methodology the CMS uses in its hospital pay-for-performance demonstration project.¹⁴ This involves aggregating individual measures within conditions using a weighted average of performance across all measures, weighting each individual measure by the number of patients eligible—eg, the number of times indicated AMI care was received at a hospital divided by the number of times patients were eligible for all AMI measures.

In 2004, the CMS initiated a pay-for-performance demonstration project in a subset of US hospitals. Performance at these hospitals is evaluated using a set of 33 performance measures across 5 clinical conditions, including the 10 measures described above. Using a condition-specific composite score constructed as described above, the CMS identified participating hospitals with the highest performance in each clinical area and gave these hospitals a bonus. Hospitals in the top decile received 2% of their Medicare diagnosis related group (DRG) payments for the condition as a bonus, while hospitals in the second decile received 1%. In the third year of the demonstration, hospitals that did not achieve performance above the threshold defining the 9th and 10th deciles in the baseline year faced financial penalties. Hospitals incurred a 1% penalty if their score fell below the 9th decile baseline level and a 2% penalty if their score fell below the 10th decile.¹⁵

Using the methods of this pay-for-performance demonstration, we calculated the payments each US hospital would have received had the CMS implemented this pay for performance nationally.¹⁴ Our calculation of composite performance differs from that used by the CMS demonstration project, because we used only the subset of 10 measures available for the entire period that apply to 3 of the 5 clinical conditions in the demonstration.

We calculated performance incentives based on condition-specific DRG payments made to every US hospital available in the 100% Medicare Provider Analysis and Review file. We estimated 2006 payments based on 2005 DRG payments, because the 2006 Medicare files had not yet been released. Because we compare payments across years, we used the medical consumer price index to adjust payments to 2004 dollars.

The Institute of Medicine recently defined safety-net hospitals as those that “deliver a significant level of health care . . . to uninsured, Medicaid, and other vulnerable patients.”¹⁶ We explored several definitions of safety-net hospitals for this study: the percentage of patients insured by Medicaid and designation as a public hospital (both from the American Hospital Association Annual Survey), the percentage of uncompensated care (from Medicare Cost Reports), and the percentage of Medicare patients receiving Supplemental Security Income plus the total number of patients enrolled in Medicaid (from the CMS). We chose to use the percentage of patients insured by Medicaid because it is available for all US hospitals and Medicaid is increasingly recognized as the main source to finance health care for low-income populations.¹⁷ For all analyses, we measured the percentage of patients insured by Medicaid in 2003 because hospital patient-mix might change in response to the publicly reported information in Hospital Compare, biasing estimates of the relationship between hospital performance and concurrent percentage of patients insured by Medicaid. Because this definition of safety-net hospitals is imperfect, we tested the sensitivity of our results to alternative definitions.

We obtained data on profit status, number of beds, and teaching status (major [member of the Council of Teaching Hospitals], minor [approved by the Accreditation Council for Graduate Medical Education], none) from the 2004 American Hospital Association Annual Survey. These characteristics were chosen because they are often used as implicit measures of hospital quality and are associated with hospital quality.^18- 21

We assessed the relationship between categories of percentages of Medicaid patients at hospitals at baseline and hospital characteristics and 2004 hospital performance, using χ² tests for categorical variables and analysis of variance for continuous variables. Categories of percentages of Medicaid patients were defined as low (lowest quartile of percentage of Medicaid patients in 2003), middle (middle 2 quartiles), and high (highest quartile).

We used linear regression to estimate the change in performance at each hospital over the study period as a function of the percentage of Medicaid patients in that hospital in 2003, controlling for hospital characteristics. For each hospital, change in performance was calculated at 2 periods: from 2004 to 2005 and from 2005 to 2006. We controlled for the expected ceiling effect among hospitals with very high performance (using baseline hospital performance) and regional differences in performance (using state fixed effects). For ease of interpretation, we used these regression results to simulate differences in the predicted change in performance at hospitals with high vs low percentages of Medicaid patients rather than presenting regression coefficients. Furthermore, based on the longitudinal assessment of changes in performance between 2004 and 2006, we calculated the percentage of hospitals with performance scores in the 1st and 10th deciles at the beginning of the study period (2004) and at the end (2006), for each category of percentage of Medicaid patients.

To evaluate whether longitudinal differences in performance improvement affect hospital finances, we conducted a simulation to estimate the bonus (or penalty) each hospital would receive under a pay-for-performance system modeled after the current CMS demonstration. We compared both the bonuses (or penalties) each hospital received expressed as a percentage of total DRG payments and the total bonus payments received by all hospitals within each decile of percentage of Medicaid patients, expressed in absolute dollars.

We tested the sensitivity of our results to alternative definitions of safety-net hospitals. In addition to our main analyses, we defined safety-net hospitals as public hospitals (nonfederal government hospitals), by the percentage of Medicare patients receiving Supplemental Security Income plus the percentage of the total number of patients enrolled in Medicaid, or by the percentage of total expenses attributable to uncompensated care.

Additionally, while we included all hospital performance scores in the main analyses regardless of the size of the hospital, performance scores at small hospitals may be statistically unstable due to small sample size. Therefore, we repeated all analyses after excluding performance scores based on fewer than 25 patients.

The study protocol was approved by the University of Pennsylvania institutional review board. All analyses were performed using Stata version 10.0 (StataCorp, College Station, Texas). We used 2-sided tests for significance; P <.05 was considered statistically significant.

For the 10 measures used in this study, a total of 4464 hospitals participated in Hospital Compare between 2004 and 2006. Of these, 381 with performance scores in only 1 of the 3 years were excluded because we could not calculate the change in performance at hospitals with only 1 observation; 418 hospitals in Hospital Compare were not in the American Hospital Association data and were also excluded. Therefore, 3665 hospitals (82%) were included in the final analyses. The 799 excluded hospitals were more likely to be government owned (32.0% vs 21.8%) or to be critical-access hospitals (52.7% vs 11.1%) and less likely to have emergency services (86.5% vs 95.5%) or to be accredited (48.4% vs 83.8%), compared with the included hospitals (P < .001 for all comparisons).

Hospitals with a high percentage of Medicaid patients were more likely to be large, government, and major teaching hospitals. In addition, they had lower 2004 performance scores than other hospitals across most measures (Table 1).

Changes in hospital performance between 2004 and 2006 were significantly associated with the percentage of Medicaid patients at each hospital, with smaller performance gains at those with high percentages. Table 2 shows adjusted estimated performance improvements at hospitals with low (5%) and high (40%) percentages of Medicaid patients. Hospitals with low percentages of Medicaid patients improved their performance significantly more than those with high percentages. For example, hospitals with low percentages of Medicaid patients improved composite AMI performance by 3.8 percentage points, vs 2.3 percentage points at those with high percentages of Medicaid patients (P = .03). This resulted in a relative difference in performance gain of 39%. This pattern was repeated across most individual performance measures and all 3 condition-specific composite measures.

Over time, hospitals with high percentages of Medicaid patients were less likely to be ranked as top performers on the Hospital Compare Web site. The Figure shows the probability of a hospital being in the 1st or 10th decile of performance in 2004 and 2006, by category of percentage of Medicaid patients. The percentage of hospitals in the top decile for AMI performance with high percentages of Medicaid patients decreased by more than half, from 10.1% in 2004 to 2.8% in 2006. At the same time, the percentage of hospitals in the top AMI performance decile with low percentages of Medicaid patients increased from 13.6% to 19.7%. These patterns held true across all 3 conditions.

Hospitals with high percentages of Medicaid patients would have received smaller bonus payments and been more likely to incur penalties under pay for performance (Table 3). For example, hospitals in the 10th decile for percentage of Medicaid patients (mean of 54.5% Medicaid patients) would have received bonuses across the 3 clinical conditions worth 0.3% of their total DRG payments in 2004. In 2006, under the CMS pay-for-performance demonstration rules, these hospitals on average would have incurred a performance penalty of 0.1% for AMI and heart failure and a reduction in their pneumonia bonus (from 0.3% to 0.2%).

Bonus payments are also shown in absolute dollar values as total bonus payments received by all hospitals within each decile based on their percentage of Medicaid patients. Hospitals with a higher percentage of patients insured by Medicaid would have received declining bonuses (or even penalties) under pay-for-performance.

Alternative definitions of safety-net hospitals produced similar results. Excluding measures with fewer than 25 eligible patients also did not affect the results.

Hospitals with high percentages of Medicaid patients had smaller improvements in hospital performance from 2004 to 2006 than those with low percentages. As a result, these safety-net hospitals were less likely to be identified as top performers on the Hospital Compare Web site and would have had substantially smaller pay-for-performance payments or incurred financial penalties by 2006. Over time, trends such as these could damage the reputations of safety-net hospitals and worsen their financial status, potentially reducing their ability to further respond to quality-improvement incentives. These results raise concerns that incentive programs such as the current CMS Hospital Compare public reporting program and the CMS pay-for-performance demonstration project may exacerbate existing disparities in health care.

While our findings demonstrate smaller improvements in performance for safety-net hospitals than for non–safety-net hospitals, these differences may not be attributable to the CMS public reporting program. Instead, these differences may be a continuation of trends in performance improvement that started prior to public reporting. Without data on hospital performance from prior to the release of Hospital Compare, we were unable to directly test whether the launch of Hospital Compare caused a change in the rate of performance improvement. However, this does not diminish the importance of our results. Whether the slower rate of improvement at safety-net hospitals is the result of public reporting or underlying trends, the implication is the same: safety-net hospitals may suffer from relative comparisons under public reporting or pay-for-performance incentives.

This could be important for vulnerable populations. Care for poor and underserved patients in the United States is currently concentrated at a small number of hospitals,⁴ and the quality of care at these hospitals is lower than that delivered at other hospitals.^3- 4 Furthermore, safety-net hospitals, on average, are already in worse financial condition than other hospitals.^22- 23 In 2003, safety-net hospitals had a median patient revenue margin of −3.0%, significantly lower than the median (−1.1%) at non–safety-net hospitals. Even after adding subsidies and government budget allocations to safety-net hospitals, the median total income margin for safety-net hospitals remained lower, with a higher percentage of safety-net hospitals having a negative total income margin.²²

Over the past decade, poor financial performance at safety-net hospitals has been a chronic problem, with margins averaging less than 2%, leaving inadequate financing for working capital or reinvestment in infrastructure and technology.²³ In addition, because safety-net hospitals serve a large proportion of Medicaid and uninsured patients, they are unable to shift costs onto other payers, increasing the difficulty of maintaining financial health. These characteristics of safety-net hospitals put them at risk for having declining financial performance and sluggish improvements in clinical performance in the face of public reporting and pay-for-performance incentive programs.

We found that projected differences in performance and pay-for-performance incentives are small. However, these small differences still raise concerns. In the hospital industry, where margins are low and sometimes negative, these financial differences can translate into meaningful differences in a hospital's resources. In addition, the small differences in payments over the short term may translate into much larger differences over the long term, because poor financial performance could lead to smaller investments in quality improvement that, in turn, further worsen both clinical and financial performance. With lower baseline performance and few resources to invest in improvement, incentives designed to stimulate quality improvement may exacerbate existing disparities, with rich hospitals getting richer and poor hospitals becoming poorer.

In prior studies that have focused on patients with commercial insurance, health plans with the lowest baseline performance—and hence the most opportunity to improve—have had the largest gains over time.²⁴ In the present study, safety-net hospitals were not able to achieve the above-average improvements seen in other settings among hospitals with low baseline performance. We do not present direct evidence of whether limited resources prevented safety-net hospitals from achieving greater improvements. However, the combination of lower baseline performance and smaller gains in performance over time does suggest that disparities in the quality of care between safety-net and non–safety-net hospitals are increasing.

Whatever their cause, it is imperative to address these widening disparities between safety-net and non–safety-net hospitals, and several strategies are possible. First, and perhaps most importantly, it may be necessary to restructure the financing of safety-net hospitals and provide subsidies addressing the production costs of quality improvement. Improving the financing of safety-net hospitals may require not only reforming Medicaid payment but also direct investments in safety-net hospitals that could fund quality-improvement initiatives. This approach has been used in the outpatient setting with the Health Disparities Collaborative of the Health Resources and Service Administration, in which learning collaboratives improved quality at federally qualified health centers.^25- 26

In addition, pay for performance could be redesigned to minimize unintended consequences. For example, providing improvement-based bonuses or paying each time appropriate care is delivered, rather than basing payment on achieving target thresholds (such as the top 10%), may reduce safety-net hospitals' disadvantages.¹³ Our study documents that safety-net hospitals achieve performance gains despite obstacles. Nonetheless, safety-net hospitals in 2006 would have been more likely to be identified as bottom-decile hospitals under systems that pay based on relative performance. Alternatively, a payment system that rewards hospitals each time appropriate care is provided would offer safety-net hospitals increasing financial returns from 2004 to 2006. Such systems may increase the likelihood that improvements at safety-net hospitals are sustainable and that care is equitable across hospitals over time. The CMS recently proposed a nationwide pay-for-performance system¹⁵ that would reward hospitals for not only achieving high performance but also for significantly improving performance.²⁷ While rewarding hospital improvement likely represents an important step toward minimizing unintended consequences of pay for performance, safety-net hospitals may nonetheless continue to be identified as poorly performing hospitals under public reporting systems, regardless of whether they are improving.

Our study has several limitations. First, we used observational data subject to unidentified confounding. It is possible that the demonstrated relationship between hospital characteristics and performance improvement is confounded by other hospital factors. We controlled for several hospital characteristics to reduce the likelihood of this risk. Additionally, because the percentages of patients insured by Medicaid may change with performance improvement, we used lagged values of these percentages. Nonetheless, it is possible that some of the differences in performance gains among hospitals are attributable to unobserved hospital characteristics.

Second, these results reflect changes in performance over 3 years, a relatively short time frame. Further analyses will be necessary to determine whether these trends continue.

Third, our results on differences in payment under pay for performance are based on a simulation that is modeled after the current CMS demonstration. Our calculation of the composite measure directly mirrors those methods used by the CMS, but we used only a subset of the CMS measures. Therefore, if the subset of measures we used are not highly correlated with the measures not included in our calculations, our estimates of financial incentives may be incorrect.

Fourth, these results do not reflect differences in payments under all potential payment systems. Our predictions are specifically modeled after the CMS demonstration. Other payment strategies would lead to different bonuses, although it will still be the case that safety-net hospitals are less likely to be high performers and thus will receive lower financial incentives under most current programs.

Improving quality of care at US hospitals is a high priority, and improving quality of care for vulnerable populations is particularly important. Incentive programs such as public reporting and pay for performance may improve quality of care at many hospitals.^28- 29 However, these incentives may have unintended consequences, including exacerbating existing disparities in quality of care across hospitals. Our study suggests that safety-net hospitals may be unable to compete for performance bonuses. This has the potential to have deleterious effects on existing financial and clinical disparities in performance. As the CMS and others proceed with the implementation of incentives for quality improvement, it is imperative that steps be taken to ensure that disparities are not worsened.