Expert Panel vs Decision-Analysis Recommendations for Postdischarge Coronary Angiography After Myocardial Infarction FREE

Practice guidelines are systematically developed statements to assist practitioners and patients regarding decisions about appropriate health care under specific clinical circumstances.¹ The recent development and proliferation of clinical practice guidelines have been driven largely by the need to improve the quality of health care, reduce practice variation, and decrease costs and health-care use.^2- 3 Clinical guidelines help clinicians choose appropriate care and serve a role in assessing accountability in health care delivery.^4- 6

Methods of guideline development vary by organization. Methods adopted by the RAND Corporation rely on a formal review of the medical literature for the specific clinical problem to inform the judgments of experts convened to make recommendations in the clinical area.^7- 9 Decision analysis and cost-effectiveness analysis also integrate multiple data sources and information to determine policy.^10- 11 The decision-analytic approach has influenced professional organizations including the American College of Physicians and the American Cancer Society,^12- 14 and it has been recognized by the Agency for Health Care Policy and Research as central to the task of guideline development.¹⁵ For both of these approaches to clinical decision making, the question of interest is often the same (Is a particular technology appropriate in a given clinical setting?) and both rely on similar scientific evidence. However, the process by which the evidence is processed is different in the 2 approaches. Bernstein et al¹⁶ evaluated the relationship between an expert panel process and a decision-analytic method in the context of coronary revascularization for patients with stable angina. They found a low level of agreement for ineffective percutaneous transluminal coronary angioplasty procedures and moderate agreement for ineffective coronary artery bypass grafting procedures.

Routine coronary angiography for certain patient subgroups in the convalescent phase of acute myocardial infarction (MI) remains controversial as evidenced by the geographic and temporal variations in cardiac catheterization rates.^17- 20 Recently, a panel of experts was convened to update the appropriateness ratings for coronary angiography after MI.^21- 23 In the present study, we used a previously published cost-effectiveness model²⁴ to compare the new appropriateness ratings with 2 outcomes of the cost-effectiveness model—gain in quality-adjusted life-years (QALYs) and the incremental cost-effectiveness ratio of coronary angiography compared with no angiography—for patients who have been discharged from the hospital and are within 12 weeks of an acute MI. Because the reasons for performing coronary angiography after a patient is discharged from the hospital can be quite different from those prior to discharge and because the decision-analytic model was not designed to apply to the decision before discharge, for this analysis we consider the comparison only for the decision after discharge.

Appropriateness ratings for coronary angiography after acute MI were updated in October 1995, using methods adopted by the RAND Corporation.^7,23 The RAND method determines appropriateness based on the extent to which the benefits of the intervention can be expected to outweigh the risks. Using a modified Delphi approach, ratings are provided by 9 medical experts from various disciplines. Each expert assigns an integer between 1 (indicating that the risks of the intervention greatly exceed the benefit) and 9 (indicating that the benefits greatly exceed the risks), where 5 represents scenarios in which the risks are equal to the benefits. For the updated ratings, a set of 890 distinct clinical scenarios (indications) of patients after acute MI were determined; 798 of these indications were specific to the period after discharge and for the following 12 weeks, which was the time frame of our analysis.

For the updated ratings, Bayesian analysis of the ordinal categorical rater data was used to estimate 3 parameters for all 890 scenarios: (1) the mean appropriateness score (range, −3.0 to 2.4, higher values indicate a greater degree of appropriateness), (2) the probability that the underlying appropriateness level is 7 or greater (clinical scenario was categorized as appropriate if the probability was >80%), and (3) the probability that the underlying appropriateness level is 3 or lower (clinical scenario was categorized as inappropriate if the probability was >80%).²³ Scenarios that were neither appropriate or inappropriate based on the latter 2 parameters were deemed equivocal. These estimates account for the heterogeneity of both the rater and the indication. For our analysis, each indication was assigned both an appropriateness score and an appropriateness category (ie, appropriate, equivocal, or inappropriate). Clinical scenarios that were classified as appropriate were further classified as either necessary (which means it would be improper care not to recommend angiography for these patients) or appropriate.

Previously, we developed a decision-analytic model to compare the costs and quality-adjusted life expectancy from treatment guided by routine coronary angiography during the convalescent phase of acute MI vs initial medical therapy without angiography.²⁴ Our model quantified angiographic results in terms of number of stenosed vessels (0-3), with a separate category for left-main disease. The model considers 2 postcatheterization strategies: (1) coronary artery bypass grafting for 3-vessel or left-main coronary artery disease and medical therapy otherwise, and (2) coronary artery bypass grafting for 3-vessel or left-main disease, percutaneous transluminal coronary angioplasty for 1- or 2-vessel disease, and medical therapy otherwise.

Since we expected the cost-effectiveness for coronary angiography to vary with clinical characteristics, we grouped patients by relevant variables available at the time of the decision. Probabilities for the number of stenosed vessels, procedure-related mortality, long-term survival, and subsequent acute MI and revascularization were modeled conditionally on the following patient characteristics: age (35-84 years, by decade); sex; and clinical factors including postinfarction angina (none vs mild vs severe), exercise tolerance test (ETT) result (negative vs positive vs strongly positive vs not done), left ventricular ejection fraction (0.20-0.49 vs ≥0.50), congestive heart failure (present vs absent), and prior acute MI (yes vs no). Based on the patient variables in the decision-analytic model (and that all persons who have severe postinfarction angina would not undergo ETT), there were 720 possible patient subgroups.

Survival probabilities in the model were based on postinfarction cohort studies, clinical trials of coronary revascularization, and age-specific and sex-specific life tables for the United States for noncoronary death rates.²⁵ Costs were based on Medicare claims data for hospitalizations and resource-based relative values for professional services. Quality-of-life adjustments were based on time-tradeoff utilities obtained from postinfarction Medicare recipients.^19,24 The only modifications from the original model²⁴ were changing the annual discount rate for future costs and consequences to 3% (from 5%) and adjusting costs to 1997 US dollars using the medical component of the consumer price index.

From the decision analysis model, each of the patient subgroups was assigned (1) a discounted QALY gain (coronary angiography vs no angiography), (2) an incremental discounted lifetime cost (coronary angiography vs no coronary angiography), and (3) an incremental cost-effectiveness ratio, all with coronary angiography compared with no angiography.

First, we assessed the importance of each clinical variable as a determinant of the appropriateness score. We did this by calculating the differences between appropriateness scores for indications stratified by the variable of interest, keeping all other variables the same. For example, to determine the effect of age, we compared the appropriateness scores of all the indications for persons younger than 75 years with the scores of the corresponding indications for persons who were 75 years or older. In a similar manner, we evaluated the relative importance of the variables included in the decision-analytic model on QALY gains and incremental cost-effectiveness ratios. For each variable we calculated the differences in these outcomes between patient subgroups stratified by the variable of interest, keeping all other variables the same.

We then compared the expert panel clinical scenarios that were common between the expert ratings and the decision model. For each of the 798 clinical scenarios available from the expert ratings, we determined whether we could simulate that particular scenario in the decision model. For example, 1 indication from the expert ratings was for patients younger than 75 years with no postinfarction angina on maximal medical managment and a positive ETT result. In our decision model, this scenario could be represented by 64 patient cohorts (4 age groups, 2 sex groups, 2 left ventricular ejection fraction groups, 2 previous prior acute MI groups, and 2 congestive heart failure groups). Thirty-six common scenarios were identified based on age (<75 or ≥75 years), angina level (none, mild, or severe), ETT result (negative, positive, strongly positive, or not done), and left ventricular ejection fraction (abnormal or normal). Abnormal left ventricular ejection fraction was defined as 0.35 to 0.49 in the expert ratings and 0.20 to 0.49 in the decision model. We assumed that all patients were treated maximally with medications and that no patients had a stress imaging test performed (variables included in the expert ratings but not in the decision analysis model). We calculated an average gain in QALYs for each of the 36 indications by taking a weighted average of the gains calculated for each patient subgroup in the decision model that fit a particular indication. We also calculated an average incremental cost-effectiveness ratio for each indication by taking a weighted average of the differences in lifetime cost divided by the weighted average of the QALY gains.

Weighting was based on the expected proportion of each subgroup that qualified for a particular indication in the population (eg, all patient subgroups younger than 75 years, with severe angina, and with an abnormal left ventricular ejection fraction), as obtained from the Cooperative Cardiovascular Project data, a national cohort of Medicare beneficiaries, and literature estimates.^26- 28 We assumed that the odds ratios applicable to patient variables from the Cooperative Cardiovascular Project data were applicable to the younger age groups but that the marginal proportions were different, and we used iterative proportional fitting techniques to estimate the proportion of each patient subgroup defined in the decision model that would be expected in patients with acute MI who were discharged from the hospital without undergoing coronary angiography.

Using the outcomes assigned to each of the 36 clinical scenarios common to both the expert panel and decision-analytic approaches, Spearman rank correlation coefficients were calculated for the appropriateness score vs both the gain in QALYs and the incremental cost-effectiveness ratio of coronary angiography compared with no angiography.

Table 1 shows the median values of the appropriateness scores for the categorizations of each variable incorporated by the expert panel guidelines, the median differences between categories, and a relative degree of impact for the variable. For example, the median appropriateness score for patients who are younger than 75 years is 0.8 greater than for patients older than 75 years (all other factors being equal), indicating that coronary angiography was rated as more appropriate for patients younger than 75 years, and that the relative degree of impact for age (<75 vs ≥75 years) is similar to that of intensity of medical therapy (maximal medical therapy vs none or less than maximal medical therapy) among patients with mild angina. Variables that showed relatively minimal effects were left ventricular ejection fraction (<0.35 vs 0.35-0.49 and 0.35-0.49 vs ≥0.50 had median differences of 0.1) and maximal medical therapy for patients with no angina (maximal medical therapy vs none or less than maximal medical therapy had a median difference of 0.1). The ETT result exerted substantial leverage on the appropriateness scores, with angiography recommended more strongly for patients with a positive ETT result than those with a negative result, by a margin of 1.3. The variable that showed the greatest effect was a stress imaging result among patients for whom an ETT was not done or equivocal (positive vs negative had a median difference of 1.8).

Table 2 shows the median values of the QALY gains and incremental cost-effectiveness ratios (both for angiography compared with no angiography) for the categorizations of each variable evaluated by the decision-analytic model, and the median differences between categories. The difference in QALY gains for strongly positive vs positive ETT results was greater than the difference for positive vs negative ETT results (0.37 and 0.16 QALYs, respectively). However, the difference in incremental cost-effectiveness ratios for strongly positive vs positive ETT results was less than that for positive vs negative ETT results ($9000 and $23,000 per QALY, respectively).

Of the 36 indications that could be mapped between the expert panel and the decision-analytic model, 15 were classified as appropriate, 14 as equivocal, and 7 as inappropriate. For indications that were ranked as appropriate, the median QALY benefit was 0.59 years (interquartile range, 0.41-0.76 years) and the median incremental cost-effectiveness ratio was $27,000 per QALY gained (interquartile range, $23,000-$35,000 per QALY gained). For indications that were ranked as inappropriate, the median QALY benefit was 0.24 years (interquartile range, 0.19-0.34) and the median incremental cost-effectiveness ratio was $54,000 per QALY gained (interquartile range, $36,000-$58,000 per QALY gained). Four of the 15 scenarios classified as appropriate were considered necessary with a median QALY benefit of 0.64 years and a median incremental cost-effectiveness ratio of $24,000 per QALY gained. Results for indications that were ranked equivocal were intermediate to those ranked appropriate and inappropriate.

Figure 1 (top) shows a plot of the estimated appropriateness score vs the QALY gain for the 36 indications. The Spearman rank correlation coefficient between these 2 variables was 0.58 (P<.001). There were 3 indications that were deemed inappropriate by the expert panel process that had QALY gains of greater than 3 quality-adjusted (discounted) months. These indications were for persons (1) younger than 75 years, without angina, without an ETT, and with an abnormal ejection fraction (0.34 QALYs), (2) 75 years or older, without angina, without an ETT, and with a normal ejection fraction (0.41 QALYs), and (3) 75 years or older, without angina, without an ETT, and with abnormal ejection fraction (0.34 QALYs). There was 1 indication that was deemed appropriate that had a QALY gain of fewer than 3 quality-adjusted (discounted) months. This indication was for persons younger than 75 years, without angina, with a positive ETT result, and with a normal ejection fraction (0.23 QALYs).

Figure 1 (bottom) shows a plot of the estimated appropriateness score vs the incremental cost per QALY gained for the 36 indications. These correlations are expected to be negative because a higher cost-effectiveness ratio corresponds to a less desirable indication. The Spearman rank correlation coefficient between these 2 variables was −0.66 (P<.001), which is higher in absolute value than the correlation shown with QALY gain. There were 3 indications that were deemed inappropriate that had an incremental cost-effectiveness of coronary angiography of less than $50,000 per QALY gained. These indications were for persons (1) younger than 75 years, without angina, without an ETT, and with a normal ejection fraction ($41,000/QALY), (2) younger than 75 years, without angina, without an ETT, and with an abnormal ejection fraction ($32,000/QALY), and (3) 75 years or older, without angina, without an ETT, and with a normal ejection fraction ($36,000/QALY). There were no indications that were deemed appropriate by the expert panel process that had an incremental cost-effectiveness of coronary angiography greater than $50,000 per QALY gained.

We compared appropriateness scores obtained from an expert panel for use of coronary angiography after MI (after discharge and before 12 weeks) with the health gains and cost-effectiveness predicted by a decision-analytic model. Although the patient characteristics that were used to define the individual clinical scenarios varied somewhat between the expert ratings and the decision model, we were able to compare 36 clinical scenarios. Among those indications that were evaluated in both the expert ratings and the decision-analytic model, we found moderate to good correlations between the appropriateness scores and (1) the QALY gains (Spearman rank correlation of 0.58) and (2) the estimated incremental cost-effectiveness ratios (Spearman rank correlation of −0.66) for coronary angiography compared with no angiography. Although the expert panel's task was to consider health effects and not costs, we actually found comparable correlations with the health gains and the cost-effectiveness ratios.

Both the appropriateness ratings and the decision-analytic model considered a large number of clinical indications (798 in the former, 720 in the latter). Common variables used in both approaches were age, postinfarction angina level, ETT result, and left ventricular ejection fraction, although there were differences in how these variables were categorized. Two variables that were included in the appropriateness ratings but were not considered in the decision-analytic model were the intensity of medical therapy and stress imaging, both of which were important factors in the expert ratings. Three variables were considered in the decision-analytic model but were not considered in the expert ratings: sex, prior acute MI, and congestive heart failure. Sex and congestive heart failure demonstrated only modest effects on the incremental cost-effectiveness ratio of coronary angiography, whereas prior MI showed a more substantial effect with a difference of 0.35 QALYs gained and $16,000 per QALY gained for patients with a prior MI compared with patients without a prior MI.

Although there was good correlation between the appropriateness scores and the results from the decision model, there were a number of scenarios in which disagreement existed. All 4 scenarios that involved patients without postinfarction angina and without an ETT were rated as inappropriate by the expert panel. They all had either a QALY gain of greater than 3 quality-adjusted (discounted) months, an incremental cost-effectiveness ratio of coronary angiography below $50,000 per QALY gained, or both. Possible reasons for these disparities are likely due to the underlying assumptions of each approach. In the decision-analytic model, the assumption for patients who did not undergo an ETT was that they had the same underlying coronary anatomy as those patients who did undergo an ETT (given that other clinical factors were equal). Although differences in the prevalence of other clinical factors between patients with and without an ETT were accounted for in our analysis of the Cooperative Cardiovascular Project data, the experts may have assumed that patients who did not get an ETT had a more favorable underlying coronary anatomy or had other comorbid conditions that made angiography less appealing.

Although the correlation between the expert ratings and the cost-effectiveness ratio of angiography was moderately high (r = −0.66), the calibration of the experts among the categories inappropriate, equivocal, and appropriate reflected more reluctance to recommend angiography than suggested by the cost-effectiveness ratios. The median gain in QALYs among the indications rated as inappropriate was approximately 3 months, which is actually a considerable gain compared with other diagnostic tests applied to high-risk patient populations with chronic disease.²⁹ In addition, several of the inappropriate scenarios compare favorably with many generally accepted medical practices, including annual mammography for women older than 50 years,³⁰ and hospital dialysis for end-stage renal disease.³¹

What might explain the fact that the raters seemed less willing to recommend coronary angiography in scenarios for which the expected health gain is positive and the cost-effectiveness ratio is reasonable? Perhaps the ratings reflected a desire for more substantial proof of efficacy, rather than the indirect evidence that underlie the decision model. While sensitivity analyses that bias the results against coronary angiography result in substantial increases in cost-effectiveness ratios for a number of patient subgroups, the health gains remain positive.

This study supports the notion that holistic expert judgments can at times differ substantially from the results of formal decision analysis, and that neither approach is superior to the other. However, the moderate concordance between the 2 methods supports the validity of both methods. If the results were totally uncorrelated, one or both methods would be suspect. Each is a unique approach to the systematic development of statements to assist practitioners and patients in making decisions about appropriate health care. Further probing of the value judgments that underlie expert ratings of appropriateness may reveal reasons for disparities from data-based decision models and, in this case, reasons for more conservative recommendations than would seem to be warranted by expected health gains and cost-effectiveness considerations. Formal synthesis of expert judgment and decision modeling is warranted in future efforts at guideline development.