Do Quality Improvement Organizations Improve the Quality of Hospital Care for Medicare Beneficiaries? FREE

Since its inception in 1965, the Medicare program has been concerned that Medicare beneficiaries receive appropriate and efficiently provided medical care. Initially, professional standards review organizations and, later, peer review organizations focused on identifying quality problems in the treatment of Medicare beneficiaries.¹ Evaluations criticized these organizations for overly emphasizing cost containment, creating adversarial relationships with providers, and conducting quality reviews of unclear validity and effectiveness.^1- 4 A 1990 Institute of Medicine (IOM) report noted these and other problems with the peer review organizations, including their unproven effect on improving the quality of care for Medicare beneficiaries.⁵

In 1992, using the IOM report for guidance, the Medicare program shifted its quality efforts to partnering with hospitals to improve the quality of care overall in addition to the regulatory case reviews.^6- 7 Medicare currently contracts with quality improvement organizations (QIOs) and allocates approximately $200 million annually for quality improvement.⁸ Quality improvement organizations work with hospitals on quality improvement in a variety of ways, including providing educational materials, using data collection and feedback to track performance on quality indicators, and assisting hospitals in implementing systems changes (eg, standing orders, clinical pathways). In dollar terms, the QIOs are the federal government’s largest initiative for improving the quality of care.⁹

Despite this significant financial investment and the questionable impact of previous quality assurance efforts, the QIOs’ effectiveness at improving the quality of care has not been rigorously evaluated. Jencks et al^8,10 have published 2 articles to track improvements in the quality of care for Medicare beneficiaries using quality process indicators adopted by Medicare and the QIOs. Their studies suggested widespread improvement on the indicators; however, they provide little evidence that the improvement can be specifically attributed to the QIOs because no concurrent control group was included in the analyses. Many other quality initiatives (eg, the National Committee for Quality Assurance, the Joint Commission on Accreditation of Healthcare Organizations, and internal hospital quality improvement efforts) were operating simultaneously. Analysis using only a historical control makes it difficult to attribute improvements specifically to the QIOs’ activities, particularly since the QIOs are focusing on many of the same indicators as these other organizations.

This study addresses this methodological weakness by using a concurrent control group. Because hospitals participate with the QIOs on quality improvement on a voluntary basis, it is possible to compare the improvement in hospitals that voluntarily partnered with the QIOs with the improvement in hospitals that did not voluntarily partner with the QIOs.

Our study had 2 objectives: (1) to explore characteristics of hospitals that voluntarily participate with the QIOs vs those that do not and (2) to determine whether hospitals voluntarily participating with the QIOs improve the quality of care for Medicare beneficiaries more than nonparticipating hospitals.

This was a retrospective comparative study that assessed performance of hospitals on 15 quality of care indicators using data from a cross-sectional sample of Medicare beneficiary medical records abstracted in 1998 (baseline) and a separate sample of medical records abstracted in 2000-2001 (follow-up).^8,10 These 15 indicators were the focus of the QIOs’ improvement efforts in the inpatient setting during the study period. The improvement in the performance on these 15 quality indicators by hospitals that actively participated with the QIOs was compared with the improvement by hospitals that did not participate with the QIOs.

Medical records were assigned to the “participating” and “nonparticipating” groups based on information obtained from the QIOs. Using records kept by the QIOs and reported to Medicare during the study period regarding the QIOs’ collaborations with individual hospitals, the QIOs classified the hospitals using 4 nonmutually exclusive categories: (1) hospital did not participate with the QIO at all; (2) hospital expressed an interest in participating with the QIO but did not track performance using data or implement systems changes; (3) hospital used data collected by itself or by the QIO for quality performance tracking as a result of working with the QIO; and (4) hospital implemented systems changes as a result of working with the QIO.

While there is no single definition of “active” participation, based on consultations with internal experts at the QIOs, a hospital was considered as “actively” participating with the QIOs in the primary analysis if it either (1) used data collected by itself or by the QIO for quality performance tracking as a result of working with the QIO or (2) implemented systems changes (eg, standing orders, critical pathways, chart reminders, and the like) as a result of working with the QIO. A hospital was classified as (1) “participating” if it performed either function at any point during the study period and (2) “not actively participating” if it performed neither of these activities. To test the sensitivity of the study findings to alternative definitions of “active” participation, 4 different definitions of hospital participation were analyzed: (1) participation defined as hospitals that only used data to track quality performance; (2) participation defined as hospitals that only implemented systems changes; (3) participation defined as hospitals that both used data to track quality performance and implemented systems changes; and (4) nonparticipation defined as hospitals that did not participate with the QIO at all (ie, excludes hospitals that expressed an interest in participating from the control group).

Hospital participation was classified for each of the 5 clinical areas separately because a hospital may choose to work with the QIO on improving care in one clinical area but not another. Because the atrial fibrillation and stroke indicators were combined together in previous analyses,^8,10 hospital participation for these 2 clinical areas was assigned jointly. Data on hospital bed size using 3 categories (1-100, 101-250, ≥251 beds) and profit status using 2 categories (not-for-profit/government, for-profit) were also provided by the QIOs.

Quality of care was assessed using 15 dichotomous process indicators of quality care associated with improved outcomes for treatment and prevention in 5 clinical areas prevalent among Medicare beneficiaries: atrial fibrillation, acute myocardial infarction, heart failure, pneumonia, and stroke (Table 1). Two other inpatient indicators (time to thrombolytic therapy and time to angioplasty in acute myocardial infarction) are not included in this analysis or in the analyses by Jencks et al^8,10 because they did not have sufficient sample sizes due to the limited number of eligible cases. Also, the heart failure indicator is a combination of 2 separate indicators (evaluation of left ventricular ejection fraction [LVEF]; angiotensin-converting enzyme inhibitor prescribed at discharge for patients with LVEF <40%). This combined indicator has also been used in other analyses.¹¹ (For more information on the development and testing of these indicators, see Jencks et al.⁸)

All QIOs were invited to participate in this study. Four QIOs with responsibility for 5 states (Maryland, New York, Nevada, Utah, and Washington) and the District of Columbia were both willing to participate in the study and able to provide the data required to conduct the analyses. The QIOs provided exactly the same data used by Jencks et al.^8,10 Specifically, approximately 700 to 800 medical records per state per clinical area were abstracted at baseline and follow-up as part of the Jencks et al studies^8,10 to provide state-level measures of performance on the quality indicators. The medical records selected for abstraction were systematically sampled from a random starting point after sorting by age, race, sex, and hospital.⁸ The baseline sample of records was collected in 1998 (prior to the start of the QIOs’ contract cycle that began in 1999), and the follow-up sample of records was collected in 2000-2001 (toward the end of the QIOs’ contract cycle that ended in 2002) for the studies by Jencks et al.^8,10 These medical records were abstracted by 2 Medicare contractors.

Using data from each record, algorithms were used by Jencks et al^8,10 to determine whether patients were eligible for a given quality indicator based on guidelines and contraindications and whether eligible patients received the care outlined by the quality indicator. Table 1 reports both the total number of records abstracted for each clinical area and the number of patients eligible for each of the 15 quality indicators. The data set includes only Medicare beneficiaries enrolled in the traditional fee-for-service plans (approximately 85% of beneficiaries during this time period) because the quality of Medicare managed care is tracked separately.

Data on patient age (continuous), sex, and race (white vs nonwhite) were provided by the QIOs. These variables were included because of their association with the quality of care delivered. The race variable was derived from Social Security Administration files for which persons self-identify their race using predefined categories. For the purposes of this study, race was collapsed into 2 categories to maintain patient confidentiality. Overall, 82% of the sample was classified as white and 18% was classified as nonwhite.

The analysis of the first question, to determine whether hospital characteristics (bed size, profit status) are associated with participation with the QIOs, used χ² tests. This analysis was performed at the hospital level for each clinical area separately, with each unique hospital identifier from baseline representing a single observation. The bed size and profit status of participating hospitals were compared with nonparticipating hospitals. Data from all 5 states and the District of Columbia were combined, and then analyses were conducted on a state-by-state basis. Also, the baseline and follow-up performance on the 15 quality indicators of participating and nonparticipating hospitals in the data set combining all states and the District of Columbia were compared using χ² tests in unadjusted analyses and using logistic regression adjusting for both hospital and patient characteristics.

The analysis of the second question, whether hospitals participating with the QIOs improved the quality of care more than hospitals not participating with the QIOs, used logistic regression models. The analyses were conducted aggregating all medical records from participating hospitals into one group and all medical records from nonparticipating hospitals into the comparison group. Sample sizes were insufficient for reliable estimation of individual hospital performance. Separate models were constructed for each of the 15 dichotomous quality indicators using patient-level data. The outcome of performance of the quality indicator among eligible patients (yes/no) was modeled as a function of participation with the QIOs (yes/no), period (baseline/follow-up), and an interaction between participation and period. The coefficient for the interaction term in this model indicates whether participating hospitals improved more or less than nonparticipating hospitals from baseline to follow-up. These models were analyzed adjusting for patient age, sex, and race, and hospital bed size and profit status.

Generalized estimating equations¹² were also tested because logistic regression assumes independence of observations, but these data sets could include more than 1 patient per hospital, leading to potential clustering. However, the generalized estimating equation analyses produced similar results and are, therefore, not reported.

The initial analysis was conducted combining the data from all 5 states and the District of Columbia; the same analyses were also conducted on a state-by-state basis. State-level analyses were considered exploratory due to small sample sizes.

Because this study used separate cross-sectional samples of medical records for baseline and follow-up, there may be some differences in which hospitals were sampled at the 2 time points (ie, a hospital may appear in the baseline sample but not the follow-up sample and vice versa). A sensitivity analysis was conducted using only hospitals included in both the baseline and follow-up data sets. Analysis of this subset reached similar study conclusions, probably because 98% of patient records were sampled from hospitals included in both the baseline and follow-up data sets. As a result, only the full data set results are presented here.

All tests were conducted at the P<.05 level of significance. This is a liberal definition of significance for this study because using a .05 significance level would be expected to produce a statistically significant result in approximately 1 of 20 tests by chance alone, and 15 indicators are being tested in this study. SAS software, version 9.00 (SAS Institute Inc, Cary, NC) was used for all analyses.

The initial data collection did not require informed consent because the data were collected by the Medicare program for administrative purposes.⁸ Prior to these data being shared with the study authors, all identifying information was removed. The Committee on Human Research of the Johns Hopkins Bloomberg School of Public Health reviewed this study and determined that it qualified as exempt because the data sets used included no identifiable information.

Across the 5 clinical areas, between 56% and 69% of hospitals participated with the QIOs using the primary definition of active participation. There were statistically significant differences in the characteristics of hospitals that participated with the QIOs vs those that did not (Table 2). Across all clinical areas, nonparticipating hospitals were more likely to be smaller and for-profit. The findings related to differences in bed size persisted in state-by-state analyses, but the findings related to differences in profit status varied from state to state.

There were also statistically significant differences on the baseline performance of 5 of 15 quality indicators, with the participating hospitals performing better at baseline on 3 of 5 indicators. At follow-up, there were statistically significant differences on 4 of 15 quality indicators, with participating hospitals performing better at follow-up on 4 of 4 indicators. In unadjusted analyses, participating hospitals improved on 13 of 15 indicators and nonparticipating hospitals improved on 12 of 15 indicators (Table 3).

Participating hospitals showed statistically significantly greater improvement than nonparticipating hospitals controlling for hospital and patient characteristics on only 1 of 15 indicators (Table 4). The one indicator showing greater improvement in participating hospitals was patient screened for or given pneumococcal vaccine (P = .005). For the remaining nonstatistically significant 14 indicators, the participating hospitals improved more on 8 indicators and the nonparticipating hospitals improved more on 6 indicators. There was no trend regarding which indicators the participating hospitals improved more on based on either clinical area or baseline performance.

The sensitivity analyses that used alternative definitions of hospital participation had little impact on the findings. There were never more than 2 indicators statistically significantly different between groups. Some of these statistically significant differences showed greater improvement among participating hospitals and some showed greater improvement among nonparticipating hospitals.

The same trend was found in the exploratory analyses conducted at the state level. In each of the 5 states and the District of Columbia, there were never any more than 2 statistically significant differences by participation and there was no consistent trend of one group improving more than the other. Participating hospitals improved more than nonparticipating hospitals on between 5 and 8 indicators but improved less on between 6 and 9 indicators (some indicators were not analyzable due to small sample sizes). The one exception is the state of Washington, where participating hospitals improved more than nonparticipating hospitals on 11 of 13 indicators, although none were statistically significant. Whether the differences here represent a true effect of the specific QIO should be further explored. This is not possible with the existing data.

The IOM is currently evaluating the QIO program, including operations, program evaluations, and whether other entities could perform the QIOs’ functions.¹³ The findings from this study do not support the hypothesis that the QIO program improves the quality of care for Medicare beneficiaries in the inpatient setting.

The QIO program was initiated because evaluations of previous Medicare quality efforts did not find clear evidence of their effectiveness.^{1- 4,14- 15} Evaluations of the QIO program suggest that it leads to improvement in the quality of care, but these evaluations have significant methodological flaws. One study found that providing participating hospitals with performance data was associated with improved care processes.¹⁶ However, this study focused on only a small percentage (4%) of quality improvement projects and had no control group. The Cooperative Cardiovascular Project assessed improvements in care for acute myocardial infarction following implementation of a demonstration project in 4 pilot states.^17- 18 Significant improvements were found in the pilot states, but evidence from the study suggests the nation as a whole also improved during this time. The one national study examined change across all states and all clinical areas and found improvement over time.^8,10 It is important to note that this statewide improvement over time was what the QIOs were contracted to accomplish. However, because this national study used only the historical controls to calculate improvement, it is difficult—if not impossible—to determine whether the improvement found was directly related to the QIOs.

The current study was undertaken to incorporate a control group but has some of its own limitations. This study focuses on inpatient care only and does not evaluate the QIOs’ efforts in the outpatient setting. Also, the baseline and follow-up medical records were not necessarily sampled from the same hospitals. However, 98% of patient records sampled were from hospitals included in both the baseline and follow-up data sets, and analysis of the subset of overlapping hospitals did not alter the study’s overall conclusions.

Medicare was unable to provide the national-level data, so individual QIOs had to be recruited for participation. Only 4 QIOs representing 5 states and the District of Columbia participated in this study. While these QIOs are from different regions of the country and the 4 QIOs represent a range of average performance on the quality measures based on the analyses by Jencks et al,¹⁰ care should be taken in generalizing these findings across all QIOs, especially given the high proportion of positive but statistically insignificant results from Washington State.

Defining hospital participation was one of the greatest challenges in this study; however, sensitivity analyses that varied the definition of participation did not significantly affect the findings. Also, the classification of hospital participation by the QIOs is subject to bias. While records were kept by the QIOs and reported to Medicare during the study period to track hospital participation, the validity of these data is difficult to evaluate retrospectively. Further, at that time, QIOs had the incentive to attribute hospital activities as being related to the QIO. This potential misclassification could lead to biased estimation of the QIOs’ impact. Spillover effects of QIO interventions to nonparticipating hospitals and the conduct of some QIO interventions statewide could also contribute to biased estimates.

This study has a relatively short period between baseline (1998) and follow-up (2000-2001) for demonstrating change. The analysis by Jencks et al¹⁰ showed that change did occur over this time period, but some hospitals classified as participating with the QIOs may not have fully implemented their QIO-related quality improvement interventions at the time of follow-up data collection. Because the QIO contract period did not end until 2002, the follow-up data may not reflect the full extent of the QIO interventions.

While these data are several years old, they are from the most recently completed QIO contract cycle. The seventh contract cycle will end in 2005, and evaluations of the QIOs’ impact in the nursing home and outpatient settings using prospectively identified participants may be available.

Because hospitals participate with the QIOs on a voluntary basis and because QIOs specifically target certain hospitals for intervention, there may be important differences between participating and nonparticipating hospitals other than their participation that could affect their ability to improve care, thus leading to selection bias. Whether the differences in hospital characteristics and baseline performance found here reflect other important differences in participating vs nonparticipating hospitals that may have affected their ability to improve the quality of care requires further research. Hospital bed size and profit status, 2 important factors associated with the quality of care,^19- 20 were controlled for. Because hospitals had to be nonidentifiable in the data set, additional information on hospital characteristics could not be obtained.

While this study does not definitively answer the question of whether the QIOs improve the quality of care for Medicare beneficiaries, the findings suggest that the improvement demonstrated over time by Jencks et al^8,10 using these inpatient quality indicators cannot be attributed to the QIOs. Additional efforts to assess and improve the QIOs’ effectiveness may be needed. The current IOM assessment of the QIO program provides an opportunity to evaluate the program further and recommend program modifications as needed.¹³