Challenges to Improve Coronary Heart Disease Risk Assessment

Coronary heart disease (CHD) is common and includes the diagnoses angina pectoris, myocardial infarction, and coronary disease death. Predicting the first occurrence of these events is important and may affect clinical decisions and care. It can be expected that more than a third of adult Americans will develop CHD during their lifetime,¹ and improvements in diagnosis and care may provide great benefits.

The article by Tzoulaki and colleagues² assesses the scientific literature concerning efforts to improve the prediction of CHD over and above the Framingham risk score.³ The authors report that many articles have provided information on new risk factors, but the study designs and data analyses in those reports raise concerns about the usefulness of the new information and about how improvement over the Framingham risk score was assessed. Researchers and clinicians should understand how CHD risk estimation is undertaken and evaluated, as these methods are frequently used to assess risk of developing disease across a variety of health disciplines.⁴

Risk estimates for initial CHD events should be derived from studies based on complete information for carefully measured risk factors at baseline, adequate follow-up, and reliable outcome data. A common initial step is to use proportional hazards univariate or age-adjusted regression models.³ When possible, the variables of interest are analyzed as continuous measures. Factors that are significant in the univariate analyses are then considered for inclusion in multivariable prediction models. With this approach, a set of variables was developed by Framingham investigators and included age, sex, systolic blood pressure, cholesterol level, high-density lipoprotein cholesterol level, diabetes mellitus, and current smoking.⁵ This approach was further evaluated with external validation sets from cohort studies across the United States.³ Including newer variables is of considerable interest because factors might improve the prediction of CHD and prevention strategies might be more effective.

A variety of performance criteria are used to evaluate the usefulness of CHD risk prediction and a brief summary explains the terms used to interpret these studies, including relative risk, discrimination, calibration, and reclassification.

For each risk factor, proportional hazards modeling yields regression coefficients for a study cohort. The relative risk of a variable is computed by exponentiating the regression coefficient in the multivariable regression models. This measure estimates the difference in risk for an individual with a given risk factor such as cigarette smoking compared with the risk for an individual who does not smoke. An analogous approach can be undertaken to estimate effects for continuous variables by showing effects for a specific number of units for the variable or by identifying differences in risk associated with a difference in the number of units associated with a standard deviation for the factor.

Discrimination is the ability of a statistical model to separate those who experience clinical CHD events from those who do not. The C statistic, analogous to the area under a receiver operating characteristic curve, is the typical performance measure used. This statistic represents an estimate of the probability that a model assigns a higher risk to those who develop CHD within a specified follow-up than to those who do not and represents a composite of the overall sensitivity and specificity of the prediction equation.⁶ Values for the C statistic range from 0.00 to 1.00; 0.50 reflects discrimination by chance. Higher values generally indicate a good level of agreement between observed and predicted risks. The average C statistic for the prediction of CHD is typically in the 0.70 range.⁷ The error associated with C-statistic estimates can be estimated and used to compare differences in risk prediction models.⁷

Calibration measures how closely predicted estimates of absolute risk agree with actual outcomes. To present calibration analyses, the data are often separated into deciles of risk, and observed rates are tested for differences from what was expected from the estimating equation. A version of the Hosmer-Lemeshow χ² statistic can be used to evaluate how well the observed and expected agree; smaller χ² values generally indicate good calibration.³ A CHD prediction model might be recalibrated if it provides relatively good ranking of risk and discrimination, but the model systematically overestimates or underestimates CHD risk in the new population.

Recalibrating a CHD risk prediction equation typically involves using the mean risk factor values and average incidence rate for the new population in the previously published equation. Kaplan-Meier estimates can be used to determine average incidence rates.⁸ This approach was undertaken for Framingham risk equations that were applied to the CHD experience of Japanese-American men in the Honolulu Heart Study and for Chinese men and women.^{6 ,8} Without calibration, the Framingham risk equation provided relatively good discrimination, but it did not provide reliable estimates of absolute risk. After calibration, the estimation fitted the observed experience more closely, and the Hosmer-Lemeshow χ² was lower.

Specialized testing in subgroups has been used for reclassification of risk for vascular disease. This strategy is typically considered when a risk factor appears to be important, but there is little difference in the C statistic when the new factor is added to a risk prediction model. For example, exercise testing information might upgrade, downgrade, or not change estimates of vascular disease risk in patients being evaluated for angina pectoris. Methods developed to assess reclassification can use a multivariable estimation procedure and evaluate the utility of a new test to increase, decrease, or not change risk estimates. Pencina et al⁹ have published an updated method to assess reclassification that takes into account the potential reclassification of both cases and noncases. Strategies to reclassify individuals have become popular to investigate the role of inflammatory and other biomarkers and evaluate the role of subclinical disease detected by imaging.^{8 - 11}

How should researchers proceed concerning future studies of CHD risk assessment? First, investigators should evaluate the effect of the traditional approach in their data and use their own continuous variable measures to predict CHD. Second, they should estimate CHD risk in a prediction model that includes both traditional CHD risk variables (continuous measures) and their new factor of interest. The statistical significance of the new factor may be modest. If the new factor is statistically associated with developing CHD, the investigators should evaluate how much discrimination is improved by the factor, assess internal or external validity, and determine whether calibration or reclassification might affect the results. Another consideration is whether the new factor could potentially replace a factor included in the Framingham risk score. For example, apolipoprotein B might be considered to replace cholesterol (or low-density lipoprotein cholesterol). In such an instance, a slightly different approach is merited, and the investigators should compare the discrimination of the model that includes apolipoproteins to the model that includes the traditional lipid measures.

Clinicians should cautiously interpret results that claim importance of new risk factors for initial CHD events. Risk assessment with traditional variables works relatively well at the present time: the approach is simple, the factors used to make the estimates are familiar, the cost is low, the interpretation is understandable by clinicians and patients, and results help guide lifestyle and medication recommendations. Looking forward, there is some room for improvement in CHD risk assessment, and studies that include careful consideration of discrimination, calibration, validation, and potentially reclassification will provide the most reliable information.