We're unable to sign you in at this time. Please try again in a few minutes.
We were able to sign you in, but your subscription(s) could not be found. Please try again in a few minutes.
There may be a problem with your account. Please contact the AMA Service Center to resolve this issue.
Contact the AMA Service Center:
Telephone: 1 (800) 262-2350 or 1 (312) 670-7827  *   Email: subscriptions@jamanetwork.com
Error Message ......
Review |

Assessment of Claims of Improved Prediction Beyond the Framingham Risk Score

Ioanna Tzoulaki, PhD; George Liberopoulos, MD; John P. A. Ioannidis, MD
JAMA. 2009;302(21):2345-2352. doi:10.1001/jama.2009.1757.
Text Size: A A A
Published online

Context With heightened interest in predictive medicine, many studies try to document information that can improve prediction of major clinical outcomes.

Objective To evaluate the reported design and analysis of studies that examined whether additional predictors improve predictive performance when added to the Framingham risk score (FRS), one of the most widely validated and cited clinical prediction scores.

Study Selection Two independent investigators searched 1908 articles citing the article that described the FRS in 1998 until September 2009 through the ISI Web of Knowledge database. Articles were eligible if they included any analyses comparing the predictive performance of the FRS vs the FRS plus some additional predictor for a prospectively assessed outcome.

Data Analyses We recorded information on FRS calculation, modeling of additional predictors, outcomes assessed, population evaluated, subgroup analysis documentation, and flaws in the methods that may have affected the reported improvements in predictive ability. We also evaluated the correlation of reported design and analysis features with the predictive model discrimination and improvements with the additional predictors.

Results We evaluated 79 eligible articles. Forty-nine studies (62%) did not calculate the FRS as it has been proposed, 15 (19%) modeled the additional predictor in more than 1 way and presented only the best-fit or area-under-the-curve (AUC) results for only 1 model, 41 (52%) did not examine the original outcome that the FRS was developed for, 33 (42%) studied a population different from what the FRS was intended for, and 25 (32%) claimed improved prediction in 1 subgroup but only 7 (9%) formally tested subgroup differences. Evaluation of independence in multivariable regressions, discrimination in AUC, calibration, and reclassification were reported in 77, 36, 7, and 7 studies, respectively, but these methods were adequately documented in only 60, 13, 4, and 2 studies, respectively. Overall, 63 studies (80%) claimed some improved prediction. Increase in AUC was larger when the predictive performance of the FRS was lower (ρ = −0.57, P < .001). Increase in AUC was significantly larger when evaluation of independence in multivariable regression or discrimination in AUC analysis was not adequately documented and when the additional predictor had been modeled in more than 1 way and only 1 model was reported for AUC.

Conclusion The majority of examined studies claimed that they found factors that could offer additional predictive value beyond what the FRS could achieve; however, most had flaws in their design, analyses, and reporting that cast some doubt on the reliability of the claims for improved prediction.

Figures in this Article

Sign in

Purchase Options

• Buy this article
• Subscribe to the journal
• Rent this article ?


Place holder to copy figure label and caption
Figure. Correlation Between Predictive Ability of the Framingham Risk Score and Change in Predictive Ability After Inclusion of Additional Predictors
Graphic Jump Location

Association between area under the receiver operating curve (AUC) of Framingham risk score (FRS) and difference in AUC between AUC of FRS alone and AUC of FRS plus inclusion of additional predictor in 88 analyses. Different markers indicate whether the improvement in AUC was found to be nominally statistically significant (P < .05) (17 analyses) or not statistically significant (30 analyses) or that information on level of significance was missing in the article (41 analyses).



Also Meets CME requirements for:
Browse CME for all U.S. States
Accreditation Information
The American Medical Association is accredited by the Accreditation Council for Continuing Medical Education to provide continuing medical education for physicians. The AMA designates this journal-based CME activity for a maximum of 1 AMA PRA Category 1 CreditTM per course. Physicians should claim only the credit commensurate with the extent of their participation in the activity. Physicians who complete the CME course and score at least 80% correct on the quiz are eligible for AMA PRA Category 1 CreditTM.
Note: You must get at least of the answers correct to pass this quiz.
Please click the checkbox indicating that you have read the full article in order to submit your answers.
Your answers have been saved for later.
You have not filled in all the answers to complete this quiz
The following questions were not answered:
Sorry, you have unsuccessfully completed this CME quiz with a score of
The following questions were not answered correctly:
Commitment to Change (optional):
Indicate what change(s) you will implement in your practice, if any, based on this CME course.
Your quiz results:
The filled radio buttons indicate your responses. The preferred responses are highlighted
For CME Course: A Proposed Model for Initial Assessment and Management of Acute Heart Failure Syndromes
Indicate what changes(s) you will implement in your practice, if any, based on this CME course.


Some tools below are only available to our subscribers or users with an online account.

145 Citations

Sign in

Purchase Options

• Buy this article
• Subscribe to the journal
• Rent this article ?

Related Content

Customize your page view by dragging & repositioning the boxes below.

See Also...
Articles Related By Topic
Related Collections
PubMed Articles