Comparison of 30-Day Mortality Models for Profiling Hospital Performance in Acute Ischemic Stroke With vs Without Adjustment for Stroke Severity FREE

Increasing attention has been given to defining the quality and value of health care through reporting of process and outcome measures.^1- 2 National quality profiling efforts have begun to report hospital-level performance for Medicare beneficiaries, including 30-day mortality rates, for common medical conditions, including acute myocardial infarction, heart failure, and community-acquired pneumonia.^3- 5 These outcome measures have been adopted by accreditation organizations, the Centers for Medicare & Medicaid Services (CMS), and other payers, and rewards based on risk-adjusted outcomes have been included in health care reform legislation.^6- 7 Because stroke is among the leading causes of death, disability, hospitalizations, and health care expenditures in the United States,⁸ there is increasing interest in also reporting outcomes for Medicare beneficiaries hospitalized with acute ischemic stroke.^9- 10

Risk adjustment for case mix is considered essential for accurately assessing and reporting hospital-level outcomes.^3- 5,11 The risk-adjustment models currently used by CMS incorporate data exclusively from administrative claims.^3- 5 Although claims data risk models for acute myocardial infarction, heart failure, and community-acquired pneumonia were validated against clinical data,^3- 5 adequate case-mix adjustment for acute ischemic stroke by claims data may be particularly difficult, and current models do not include adjustment for stroke severity. It has been previously shown in patient-level analyses that stroke severity as indexed by the National Institutes of Health Stroke Scale (NIHSS) is an important predictor of mortality in acute ischemic stroke.^12- 15 However, whether adjustment for stroke severity is necessary for hospital-level risk profiling has not been well studied.

Using data from hospitals participating in the Get With The Guidelines–Stroke (GWTG-Stroke) program on acute ischemic stroke admissions linked to Medicare data, this study was designed to (1) evaluate the change in model performance by adding or not adding NIHSS score for predicting hospital-level 30-day all-cause mortality for Medicare beneficiaries with acute ischemic stroke and (2) determine whether there are meaningful differences in hospital ranking with the use of models with and without adjustment for NIHSS score.

Clinical data including stroke severity were obtained from GWTG-Stroke, and administrative claims data were obtained from CMS. Data from the GWTG-Stroke registry were linked with enrollment files and inpatient claims from CMS for the period April 1, 2003, through December 31, 2009. Follow-up continued through 2010. The design, inclusion criteria, and data collection methods for GWTG-Stroke have been described previously.^16- 17 Patients were eligible for inclusion in the GWTG-Stroke registry if they were admitted for acute stroke. Trained hospital personnel ascertained acute ischemic stroke admissions by either prospective clinical identification, retrospective identification using discharge codes from the International Classification of Diseases, Ninth Revision, Clinical Modification (ICD-9-CM), or a combination. Patient data abstracted by trained hospital personnel included demographics, medical history, in-hospital treatment and events, discharge treatment and counseling, mortality, and discharge destination. Admission staff, medical staff, or both recorded race/ethnicity, usually as the patient was registered. Prior studies have suggested differences in outcomes based on race/ethnicity.

All patient data were deidentified before submission. All states and regions of the United States were represented, and a variety of centers participated, from community hospitals to large tertiary centers. Data on hospital-level characteristics were obtained from the American Hospital Association.¹⁸ All participating institutions were required to comply with local regulatory and privacy guidelines and, as determined by each participating institution, to obtain institutional review board approval. Outcome Sciences served as the registry coordinating center. The Duke Clinical Research Institute served as the data analysis center and has an agreement to analyze the aggregate deidentified data for research purposes. The institutional review board of the Duke University Health System approved the study.

The CMS files (100% Medicare Research Identifiable Files) included data for all fee-for-service Medicare beneficiaries aged 65 years or older who were hospitalized with a diagnosis of acute stroke (ICD-9-CM codes 430.x, 431.x, 433.x, 434.x, and 436.x). We linked patient data in the GWTG-Stroke registry with Medicare Part A inpatient claims, matching by admission and discharge dates, hospital, date of birth, and sex using methods previously described.^{15,17,19- 20} Patients in Medicare managed care plans (15%-25% of the population depending on the region of the country) or other types of insurance are not included in fee-for-service Medicare claims files and therefore cannot be matched.^{15,17,19- 20} Patients from centers with fewer than 25 ischemic stroke patients with NIHSS score documented during the study period were excluded to minimize the likelihood of sampling error.

The NIHSS is a 15-item neurologic examination stroke scale used to provide a quantitative measure of stroke-related neurologic deficit by evaluating the effect of acute ischemic stroke on the levels of consciousness, language, neglect, visual-field loss, extraocular movement, motor strength, ataxia, dysarthria, and sensory loss.^15,21 The NIHSS is designed to be a simple, valid, and reliable tool that can be administered at the bedside consistently by physicians, nurses, or therapists. Each item is scored with 3 to 5 grades, with 0 as normal and the final total score having a potential range of 0 to 42, with higher scores indicating greater stroke severity. The first recorded NIHSS score, as close to admission time as possible, was collected.

The outcome of interest was hospital-level all-cause mortality within 30 days from time of admission. Deaths and dates of death were obtained, with complete ascertainment, from the CMS vital status files.^{15,17,19- 20}

The approach to risk model development without the NIHSS score was to closely follow the previously described approach used by the Yale New Haven Health System/Center for Outcomes Research and Evaluation and CMS for other publicly reported 30-day mortality^3- 6 and described in developing a model for 30-day risk-standardized mortality for acute ischemic stroke.¹⁰ This approach to risk adjustment generally follows the principles stated in the American Heart Association Scientific Statement “Standards for statistical models used for public reporting of health outcomes.”¹¹

The candidate variables for the risk model are patient-level risk adjustors that are expected to be predictive of mortality, based on empirical analysis, prior literature, and clinical judgment, including demographic factors (age, sex) and indicators of comorbidity.¹¹ For each patient, the candidate variables considered for this model were derived from the Medicare claims files and included secondary diagnosis and procedure codes from the index hospitalization and from the principal and secondary diagnosis codes from hospitalizations, institutional outpatient visits, and physician encounters in the 12 months before the index hospitalization.^3- 6 The model is intended to adjust for case differences based on the diverse aspects of the clinical status of acute ischemic stroke patients at time of admission. Condition categories are drawn from more than 15 000 ICD-9-CM diagnosis codes.¹¹

The final set of risk-adjustment variables for the claims-based model was selected to be aligned with those included in the proposed CMS acute ischemic stroke 30-day mortality measure. The final 87 variables are shown in eTable 1. The approach to model development with the NIHSS score included was methodologically identical, except for the addition of a measure of disease severity (NIHSS). The NIHSS score was treated as a continuous parameter. Hierarchical generalized logistic regression models were used to model the binary outcome of mortality within 30 days of admission as a function of patient demographic and clinical characteristics and a random hospital-specific effect. This strategy accounts for within-hospital correlation of the observed outcomes. Additional details are provided in the eMethods.

Discrimination of the base claims model without NIHSS score was assessed by determining the C statistic and was compared with the discrimination of the claims model with NIHSS score. We used the integrated discrimination improvement (IDI) index to measure how the model that included NIHSS score reclassified patients compared with the model without NIHSS score.^22- 23 A higher IDI index indicates a greater improvement in risk discrimination and improved reclassification. The net reclassification improvement (NRI) index, which compares the shifts in reclassified categories by observed outcome, resulting from the addition of NIHSS score to the model was also determined. A higher NRI index indicates a greater improvement in risk discrimination and improved reclassification.

We also ranked hospitals by their 30-day adjusted mortality from each model and plotted the agreement between these rankings. We ranked these intercepts to group hospitals into 3 categories, top 20%, middle 60%, and bottom 20%, and compared the results across models. We chose these categories because they reflect categories that may be relevant to pay-for-performance programs, in which the top 20% of hospitals are eligible for bonus payments and the bottom 20% of hospitals may be subject to a payment penalty.⁷ We also grouped hospitals into the top 5%, middle 90%, and bottom 5% and compared the results across models. As an additional approach, hospitals with the 95% credible intervals of the estimated random intercepts not covering the null point are considered to have performance that is significantly better or worse than the average hospital. These categories are analogous to the portions of hospitals identified in Hospital Compare as having better than, no different than, or worse than expected 30-day risk-standardized mortality rates.⁶ All P values are 2-sided, with P < .05 considered statistically significant. SAS software version 9.2 (SAS Institute) was used for all analyses.

There were 693 458 hospitalizations of patients with acute ischemic stroke enrolled in GWTG-Stroke between April 2003 and December 2009. From 472 443 hospitalizations of patients aged 65 years or older, we matched 318 393 patients (67.4%) to fee-for-service Medicare claims from 1428 hospitals. Of these acute ischemic stroke patients, there were 143 481 (45.1%) with NIHSS score documented. We further confined the population to patients' first index stroke admission during the study period (n = 138 314), nontransferred patients with acute ischemic stroke (n = 131 404), and patients from centers with fewer than 25 ischemic stroke patients with NIHSS score documented. This resulted in a final study population of 127 950 acute ischemic stroke patients from 782 GWTG-Stroke hospitals. Among these hospitals, there were 124 428 nontransferred acute ischemic stroke patients without NIHSS score documented during the study period. The demographics, clinical characteristics, geographic distribution, and hospital characteristics of the study patients with and without NIHSS score recorded from the 782 hospitals were similar, with few exceptions (eTable 2).

The characteristics of the 127 950 acute ischemic stroke patients with NIHSS score documented are shown in Table 1. The median age was 80 years, 57.3% were women, and 86.2% were white. Prior stroke or transient ischemic attack was present in 32.4% of patients. Comorbidities were common with hypertension in 82.6%, diabetes in 29.2%, coronary artery disease or prior myocardial infarction in 33.7%, and history of atrial fibrillation or flutter in 26.9%. The NIHSS median score in this overall population was 5 (interquartile range [IQR], 2-12). The median hospital-level NIHSS score was 5 (IQR, 4-7). Of the 782 GWTG-Stroke hospitals included in this study, median bed size was 377, all regions of the United States were represented, and teaching hospitals accounted for 21.3% of all hospitals and 28.1% of admissions (Table 1). There were 18 186 deaths (14.5%) within the first 30 days, including 7430 deaths during the index hospitalization (in-hospital mortality, 5.8%). The median hospital-level 30-day mortality rate was 14.5% (IQR, 11.3%-17.9%).

Table 2 reports the performance of the claims models without NIHSS score vs the claims model with NIHSS score for 30-day mortality among acute ischemic stroke patients at all registry hospitals in the analysis. Discrimination, calibration, and explained variance were substantially improved with the addition of NIHSS score. The hospital claims mortality model without NIHSS score had a C statistic of 0.772 (95% CI, 0.769-0.776), whereas the NIHSS score alone had a C statistic of 0.822 (95% CI, 0.819-0.825) and the claims model with the NIHSS score included had a C statistic of 0.864 (95% CI, 0.861-0.867; absolute difference for claims model with vs without NIHSS score, +0.091; 95% CI, 0.088-0.094; P < .001). Explained variance improved over the model without NIHSS score (Table 2).

Additional tests for model discrimination were improved with the addition of NIHSS score. The NRI index (93.1%; 95% CI, 91.6%-94.6%; P < .001) and IDI index scores (discrimination slope for model with vs without NIHSS score, 27.7% vs 12.8%; difference, +15.0%; 95% CI, 14.6%-15.3%; P < .001; relative IDI, 1.17) all demonstrated substantially more accurate classification of hospital 30-day mortality after the addition of NIHSS score to the claims model. Although both claims models resulted in wide variance of predicted risk between the most extreme deciles, the model with NIHSS exhibited better agreement between observed and predicted mortality rates (Hosmer-Lemeshow goodness-of-fit test χ², 352.9 vs 173.8; P < .001).

Of the 782 hospitals in the analysis, the median absolute change in rank position was 79 places (IQR, 35-155) when hospitals were ranked with risk models without and with NIHSS score. The numbers of hospitals in which the ranking categories changed based on the 30-day mortality models with and without NIHSS score are shown in Table 3. Compared with the 20%/60%/20% rankings generated by the 2 models, the weighted κ was 0.585 (unweighted κ, 0.530), and the model without NIHSS score differed in category for 206 of 782 hospitals (26.3%) (Table 3).

There was considerable disagreement between the models with and without NIHSS score regarding which hospitals were in the top 5 percentile with respect to the lowest risk-adjusted mortality with weighted κ of only 0.533 (unweighted κ, 0.521). Of the 39 hospitals identified as top performing (top 5 percentile) by the model without NIHSS score, only 23 were identified by the model with NIHSS score, and 16 other hospitals were newly identified (Table 3).

There was even greater disagreement about the bottom-performing hospitals. Of the 40 bottom-performing hospitals according to the claims model without NIHSS score, 19 of these hospitals (47.5%) were no longer identified as being in the worst fifth percentile for mortality after applying the model that adjusted for NIHSS score. For the analysis of hospitals ranked as better than, no different than, and worse than expected for 30-day risk-standardized mortality, 9 of 22 hospitals (40.9%) identified as having better than expected mortality rates in the model without NIHSS score were reclassified to as-expected mortality rates in the model with NIHSS score, and 15 of 26 hospitals (57.7%) classified as worse than expected outcomes were reclassified as to as expected (Table 3). The weighted κ was 0.502 and unweighted κ, 0.494.

To evaluate whether there was selection bias introduced by analyzing the population of acute ischemic stroke patients with NIHSS score documented, the discrimination of the claims model in the overall population (n = 252 379) with or without NIHSS score documented was evaluated. The C statistic (0.772; 95% CI, 0.769-0.774) was similar to that demonstrated for the patients with NIHSS score documented.

Using data from the GWTG-Stroke registry combined with administrative claims data, we found that acute ischemic stroke risk-adjustment models varied in their ability to accurately predict hospital 30-day mortality risk depending on whether or not they adjusted for initial stroke severity. Beyond substantial differences in model discrimination and calibration, a hospital's variance from its expected, risk-standardized 30-day mortality outcomes, relative to its peers, frequently changed based on which risk-adjustment model was applied. More than 40% of hospitals identified in the top or bottom 5% of hospital risk-adjusted mortality would have been reclassified into the middle mortality range using a model adjusting for NIHSS score compared with a model without NIHSS score adjustment. Similarly, when considering the top 20% and bottom 20% ranked hospitals, close to one-third of hospitals would have been reclassified. These findings highlight the importance of including a valid specific measure of stroke severity in hospital risk models for mortality after acute ischemic stroke for Medicare beneficiaries. Furthermore, this study suggests that inclusion of admission stroke severity may be essential for optimal ranking of hospital with respect to 30-day mortality.

The increasing use of 30-day mortality rates to assess hospital quality has intensified their importance. This outcome measure has the potential to give both patients and clinicians important feedback concerning a hospital's quality of care.¹ Because hospitals may treat patients with differing case mix and illness severity, adequate risk adjustment is essential.¹¹ Outcome measures that do not adequately risk-adjust may systematically favor hospitals that care for patients with less severe illness, regardless of whether these hospitals' approaches to patient management contributed to better or worse patient outcomes.¹¹ Whether or not clinicians agree with the basic tenets of hospital comparisons, nearly all agree that, if outcomes are to be compared, such comparisons should be done only after appropriately risk-adjusting the results.^11,24- 25 The primary concern with outcome measures that do not adequately discriminate mortality risk is that hospital rankings based on these models may distort hospital profiling and quality assessment.^24- 25 Thus, a key question confronting clinicians, hospitals, payers, and policy makers is whether current and emerging measures that assess 30-day mortality are adequate for public reporting and use for rewarding and penalizing hospitals in value-based purchasing.⁷

It has been previously reported that risk-standardization models for nonstroke conditions that adjust for demographics and comorbid conditions based on administrative claims data are sufficient for public reporting, despite not adjusting for indicators of disease severity, laboratory test results, and diagnostic studies at time of presentation.^3- 6 The CMS is now considering an outcome measure of 30-day mortality for acute ischemic stroke.²⁶

In this GWTG-Stroke analysis, we demonstrate that a hospital risk model based on claims data alone without adjustment for stroke severity has substantially worse discrimination compared with a model that adjusts for stroke severity using the NIHSS score. This study also suggests that the ranking of hospitals could be confounded if the risk-adjustment models do not take into account the severity of the acute ischemic stroke, demographics, and other factors present at the time of acute ischemic stroke presentation. For conditions such as heart failure, acute myocardial infarction, and pneumonia, it is believed that claims-only models for 30-day mortality can adequately discriminate mortality risk at the hospital level for Medicare patients.^3- 5 However, in contrast to these conditions, these findings suggest that this is not the case for hospital risk models for acute ischemic stroke.

It logically follows that a measure of stroke severity would be essential for optimal discrimination of mortality risk, as NIHSS score is well documented to be a key risk determinant in acute ischemic stroke.^12- 15 Prior patient-level analyses have shown that NIHSS score was the strongest predictive variable for in-hospital and 30-day mortality and substantially improved the performance of a model based on clinical variables without stroke severity.^14- 15 Our findings are also consistent with, and substantially extend, prior smaller and regionally restricted analyses. An analysis of 2 administrative data prediction models used to assess New York hospitals found that, in the absence of a measure of index stroke severity, the mortality prediction models were noncongruent and yielded hospital rankings that agreed only slightly more often than expected by chance.²⁴

As public reporting and value-based purchasing policies increase for outcome measures, it is important to recognize the effect that using models with less than ideal discrimination and calibration has on the ranking of hospitals and the lack of correlation among ranking by models that do and do not adjust for critical risk determinants.⁷ To truly identify highest- and lowest-performing hospitals for acute ischemic stroke outcomes, models that adjust for stroke severity will likely be needed.

It is also important to carefully consider that rewarding or punishing hospitals on the basis of a risk model that does not account for stroke severity may misalign incentives, as many hospitals identified as performing better than or worse than expected were likely misclassified. As a consequence, hospitals may consider turning away patients with more severe strokes or transferring them to other hospitals after emergency department assessment to avoid being misclassified as having higher risk-standardized mortality. A 30-day mortality model for acute ischemic stroke without adjustment for stroke severity provides lesser discrimination, produces different rankings of hospital performance, and may be biased in favor of hospitals treating less severe strokes than a model with adjustment for stroke severity.

Although the present analysis supports collection and adjustment of stroke severity for hospital 30-day mortality, NIHSS score was recorded for 50.7% of hospitalized acute ischemic stroke patients during the study period. The time and expertise needed to perform even a short standardized stroke severity assessment and ensuring these data are accurately abstracted and entered into the Hospital Compare data collection system are important barriers that will need to be overcome.⁶ The increasing use of quality measures reporting the portion of acute ischemic stroke patients with NIHSS scores recorded may facilitate this process.²⁷ The Hospital Alliance and CMS should consider requiring the collection and reporting of the initial NIHSS score for all hospitalized acute ischemic stroke patients prior to implementation of a 30-day mortality risk-standardized model.

Several limitations in our study should be noted. First, the patient population studied is Medicare fee-for-service beneficiaries enrolled in GWTG-Stroke and may not be representative of all patients hospitalized with acute ischemic stroke. However, the differences between patients with and without NIHSS score recorded were small, and other recent analyses suggest that fee-for-service Medicare patients enrolled in GWTG-Stroke are representative of the entire US population of fee-for-service Medicare patients.²⁰ Second, this study includes only patients in fee-for-service Medicare and thus does not include patients who were enrolled in managed care, uninsured individuals, and patients younger than 65 years of age. Third, variables used for risk adjustment were drawn from claims data and are dependent on their accuracy, as is any model derived from Medicare administrative claims databases. Fourth, these risk models did not adjust for therapies provided, such as tissue plasminogen activator. Fifth, as is the convention for CMS mortality models, the primary outcome was all-cause mortality rather than cause-specific mortality (eg, stroke-specific mortality).^3- 5 We did not assess 30-day rehospitalization, health-related quality of life, functional recovery, patient satisfaction, and other clinical outcomes that may be of interest for hospital outcome measures, and the need for adjustment for stroke severity for these other outcome measures requires further study. Sixth, the hospital ranking methodologies applied did not use shrinkage analysis and may have differed in other ways than those currently applied by CMS.^3- 5 Whether these differences would affect these findings needs additional analysis, but prior studies have suggested selection of risk adjustors had far more influence on ranking profiles than choice of statistical strategies.²⁸

Adding stroke severity assessed with the NIHSS score to a hospital 30-day mortality model based on claims data for Medicare beneficiaries with acute ischemic stroke is associated with substantial improvement in model discrimination and changes in mortality performance ranking for a considerable proportion of hospitals. These findings suggest that it may be critical to collect and include stroke severity for optimal hospital risk adjustment of 30-day mortality for Medicare beneficiaries with acute ischemic stroke.