0
ARTICLE |

Clinical Prediction Rules: Title and subTitle BreakA Review and Suggested Modifications of Methodological Standards FREE

Andreas Laupacis, MD; Nandita Sekar, MD; lan G. Stiell, MD
[+] Author Affiliations

Reprints not available from the authors.


JAMA. 1997;277(6):488-494. doi:10.1001/jama.1997.03540300056034
Text Size: A A A
Published online

Background.  —Clinical prediction rules are decision-making tools for clinicians, containing variables from the history, physical examination, or simple diagnostic tests.

Objective.  —To review the quality of recently published clinical prediction rules and to suggest methodological standards for their development and evaluation.

Data Sources.  —Four general medical journals were manually searched for clinical prediction rules published from 1991 through 1994.

Study Selection.  —Four hundred sixty potentially eligible reports were identified, of which 30 were clinical prediction rules eligible for study. Most methodological standards could only be evaluated in 29 studies.

Data Abstraction.  —Two investigators independently evaluated the quality of each report using a standard data sheet. Disagreements were resolved by consensus.

Data Synthesis.  —The mathematical technique was used to develop the rule, and the results of the rule were described in 100% (29/29) of the reports. All the rules but 1 (97% [28/29]) were felt to be clinically sensible. The outcomes and predictive variables were clearly defined in 83% (24/29) and 59% (17/29) of the reports, respectively. Blind assessment of outcomes and predictive variables occurred in 41% (12/29) and 79% (23/29) of the reports, respectively, and the rules were prospectively validated in 79% (11/14). Reproducibility of predictive variables was assessed in only 3% (1/29) of the reports, and the effect of the rule on clinical use was prospectively measured in only 3% (1/30). Forty-one percent (12/29) of the rules were felt to be easy to use.

Conclusions.  —Although clinical prediction rules comply with some methodological criteria, for other criteria, better compliance is needed.

REFERENCES

Mark DB, Shaw L, Harrell FE, et al.  Prognostic value of a treadmill exercise score in outpatients with suspected coronary artery disease . N Engl J Med . 1991;;325:849-853.
Stiell IG, McKnight RD, Greenberg GH, et al.  Implementation of the Ottawa Ankle Rules . JAMA . 1994;;271:827-832.
Wasson JH, Sox HC, Neff RK, Goldman L.  Clinical prediction rules: application and methodological standards . N Engl J Med . 1985;;313:793-799.
Stiell IG, Greenberg GH, McKnight RD, Nair RC, McDowell I, Worthington JR.  A study to develop clinical decision rules for the use of radiography in acute ankle injuries . Ann Emerg Med . 1992;;21:384-390.
Stiell IG, Greenberg GH, McKnight RD, et al.  Decision rules for the use of radiography in acute ankle injuries: refinement and prospective validation . JAMA . 1993;;269:1127-1132.
Stiell I, Wells G, Laupacis A, et al.  Multicentre trial to introduce the Ottawa ankle rules for use of radiography in acute ankle injuries . BMJ . 1995;;311:594-597.
Stiell IG, Greenberg GH, Wells GA, et al.  Prospective validation of a decision rule for the use of radiography in acute knee injuries . JAMA . 1996;;275:611-615.
Sauve JS, Thorpe KE, Sackett DL, et al.  Can bruits distinguish high-grade from moderate symptomatic carotid stenosis? Ann Intern Med . 1994;;120:633-637.
Farr BM, Sloman AJ, Fisch MJ.  Predicting death in patients hospitalized for community-acquired pneumonia . Ann Intern Med . 1991;;115:428-436.
Marottoli RA, Cooney LM, Wagner R, Doucette J, Tinetti ME.  Predictors of automobile crashes and moving violations among elderly drivers . Ann Intern Med . 1994;;121:842-846.
Silver MT, Rose GA, Paul SD, O'Donnell CJ, O'Gara PT, Eagle KA.  A clinical rule to predict preserved left ventricular ejection fraction in patients after myocardial infarction . Ann Intern Med . 1994;;121:750-756.
Pryor DB, Shaw L, McCants CB, et al.  Value of the history and physical in identifying patients at increased risk for coronary artery disease . Ann Intern Med . 1993;;118:81-90.
Williams JW, Simel DL, Roberts L, Samsa GP.  Clinical evaluation of sinusitis: making the diagnosis by history and physical examination . Ann Intern Med . 1992;;117:705-710.
Ashton CM, Petersen NJ, Wray NP, et al.  The incidence of perioperative myocardial infarction in men undergoing noncardiac surgery . Ann Intern Med . 1993;;118:504-510.
The Stroke Prevention in Atrial Fibrillation Investigators.  Predictors of thromboembolism in atrial fibrillation, I: clinical features of patients at risk . Ann Intern Med . 1992;;116:1-5.
The Stroke Prevention in Atrial Fibrillation Investigators.  Predictors of thromboembolism in atrial fibrillation, II: echocardiographic features of patients at risk . Ann Intern Med . 1992;;116:6-12.
Morrow K, Morris CK, Frolicher VF, et al.  Prediction of cardiovascular death in men undergoing noninvasive evaluation for coronary artery disease . Ann Intern Med . 1993;;118:689-695.
Zarchy TM, Ershoff D.  Which clinical variables predict an abnormal double-contrast barium enema result? Ann Intern Med . 1991;;114:137-141.
Inouye SK, Viscoli CM, Horwitz RI, Hurst LD, Tinetti ME.  A predictive model for delirium in hospitalized elderly medical patients based on admission characteristics . Ann Intern Med . 1993;;119:474-481.
Nguyen T, Sambrook P, Kelly P, et al.  Prediction of osteoporotic fractures by postural instability and bone density . BMJ . 1993;;307:1111-1115.
Medical Research Council Antiepileptic Drug Withdrawal Study Group.  Prognostic index for recurrence of seizures after remission of epilepsy . BMJ . 1993;;306:1374-1378.
van Duijn NP, Brouwer HJ, Lamberts H.  Use of symptoms and signs to diagnose maxillary sinusitis in general practice: comparison with ultrasonography . BMJ . 1992;;305:684-687.
Skene AI, Smith JM, Dore CJ, Charlett A, Lewis JD.  Venous leg ulcers: a prognostic index to predict time to healing . BMJ . 1992;;305:1119-1121.
Poungvarin N, Viriyavejakul A, Komontri C.  Siriraj stroke score and validation study to distinguish supratentorial intracerebral hemorrhage from infarction . BMJ . 1991;;302:1565-1567.
Celani MG, Righetti E, Migliacci R, et al.  Comparability and validity of two clinical scores in the early differential diagnosis of acute stroke . BMJ . 1994;;308:1674-1676.
Dramaix M, Hennart P, Brasseur D, et al.  Serum albumin concentration, arm circumference, and oedema and subsequent risk of dying in children in Central Africa . BMJ . 1993;;307:710-713.
Parsons RW, Jamrozik KD, Hobbs MST, Thompson DL.  Early identification of patients at low risk of death after myocardial infarction and potentially suitable for early hospital discharge . BMJ . 1994;;308:1006-1010.
Nouri FM, Lincoln NB.  Predicting driving performance after stroke . BMJ . 1993;;307:482-483.
Bates DW, Lee TH.  Rapid classification of positive blood cultures: prospective validation of a multivariate algorithm . JAMA . 1992;;267:1962-1966.
Marcantonio ER, Goldman L, Mangione CM, et al.  A clinical prediction rule for delirium after elective noncardiac surgery . JAMA . 1994;;271:134-139.
Browner WS, Li J, Mangano DT, for the Study of Perioperative Ischemia Research Group.  In-hospital and long-term mortality in male veterans following noncardiac surgery . JAMA . 1992;;268:228-232.
Chodak GW, Vogelzang NJ, Caplan RJ, Soloway M, Smith JA.  Independent prognostic factors in patients with metastatic (stage D2) prostate cancer . JAMA . 1991;;265:618-621.
Hollenberg M, Mangano DT, Browner WS, et al.  Predictors of postoperative myocardial ischemia in patients undergoing noncardiac surgery . JAMA . 1992;;268:205-209.
Cooper-Patrick L, Crum RM, Ford DE.  Identifying suicidal ideation in general medical patients . JAMA . 1994;;272:1757-1762.
The International Non-Hodgkins Lymphoma Prognostic Factors Project.  A predictive model for aggressive non-Hodgkin's lymphoma . N Engl J Med . 1993;;329:987-994.
Lee TH, Juarez G, Cook EF, et al.  Ruling out acute myocardial infarction: a prospective multicenter validation of a 12-hour strategy for patients at low risk . N Engl J Med . 1991;;324:1239-1246.
Williams M, Warwick D, Bannister GC.  Death and thromboembolic disease after total hip replacement without routine chemical prophylaxis . Thromb Haemost . 1995;;73:953.
Wells PS, Hirsh J, Anderson DR, et al.  Accuracy of clinical assessment of deep-vein thrombosis . Lancet . 1995;;345:1326-1330.
Sackett DL, Haynes RB, Guyatt GH, Tugwell P. Clinical Epidemiology: A Basic Science for Clinical Medicine . 2nd ed. Boston, Mass: Little Brown & Co Inc; 1991;:179-180.
Jutras D.  Clinical practice guidelines as legal norms . Can Med Assoc J . 1993;;148:905-908.
Ciampi A, Hogg SA, McKinney S, Thiffault J.  RECPAM: A computer program for recursive partitioning and amalgamation for censored survival data and other situations frequently occurring in biostatistics, I: methods and program features . Comput Methods Programs Biomed . 1988;;26:239-256.
Ciampi A, Chang CH, Hogg S, McKinney S.  Recursive partitioning: a versatile method for exploratory data analysis in biostatistics . In: MacNeill IB, Umphrey GJ, eds. Time Series and Econometric Modelling/Biostatistics . Boston, Mass: D Reidel Publishing Co; 1987;:23-50.
Concato J, Feinstein AR, Holford TR.  The risk of determining risk with multivariable models . Ann Intern Med . 1993;;118:201-210.
Baxt WG.  Use of an artificial neural network for the diagnosis of myocardial infarction . Ann Intern Med . 1991;;115:843-848.
Shufflebarger CM, Young W.  What is a neural network? Ann Emerg Med . 1992;;21:1461-1462.
The PIOPED Investigators.  Value of the ventilation/perfusion scan in acute pulmonary embolism: results of the prospective investigation of pulmonary embolism diagnosis (PIOPED) . JAMA . 1990;;263:2753-2759.
Rifkin MD, Zerhouni EA, Gatsonis CA, et al.  Comparison of magnetic resonance imaging and ultrasonography in staging early prostate cancer: results of a multi-institutional cooperative trial . N Engl J Med . 1990;;323:621-626.
Hanley JA, McNeil BJ.  The meaning and use of the area under a receiver operating characteristic (ROC) curve . Radiology . 1982;;143:29-36.
Fletcher RH, Fletcher SW, Wagner EH. Clinical Epidemiology: The Essentials . 2nd ed. Baltimore, Md: Williams & Wilkins; 1988;.
Kerr L, Kelly A, Grant J, et al.  Failed validation of a clinical decision rule for the use of radiography in acute ankle injury . N Z Med J . 1994;;107:294-295.
Stiell IG, Greenberg GH, McKnight RD, Wells GA.  Ottawa ankle rules for radiography of acute ankle injuries . N Z Med J . 1995;;108:111.
Fleiss JL. Statistical Methods for Rates and Proportions . 2nd ed. New York, NY: John Wiley & Sons Inc; 1981;.
Feinstein AR. Clinimetrics . New Haven, Conn: Yale University Press; 1987;.
Kirshner B, Guyatt G.  A methodological framework for assessing health indices . J Chronic Dis . 1985;;38:27-36.
The Standards of Reporting Trials Group.  A proposal for structured reporting of randomized controlled trials . JAMA . 1994;;272:1926-1931.
The Asilomar Working Group on Recommendations for Reporting of Clinical Trials in the Biomedical Literature.  Checklist of information for inclusion in reports of clinical trials . Ann Intern Med . 1996;;124:741-743.

Figures

Tables

Interactive Graphics

Video

Country-Specific Mortality and Growth Failure in Infancy and Yound Children and Association With Material Stature

Use interactive graphics and maps to view and sort country-specific infant and early dhildhood mortality and growth failure data and their association with maternal

Mark DB, Shaw L, Harrell FE, et al.  Prognostic value of a treadmill exercise score in outpatients with suspected coronary artery disease . N Engl J Med . 1991;;325:849-853.
Stiell IG, McKnight RD, Greenberg GH, et al.  Implementation of the Ottawa Ankle Rules . JAMA . 1994;;271:827-832.
Wasson JH, Sox HC, Neff RK, Goldman L.  Clinical prediction rules: application and methodological standards . N Engl J Med . 1985;;313:793-799.
Stiell IG, Greenberg GH, McKnight RD, Nair RC, McDowell I, Worthington JR.  A study to develop clinical decision rules for the use of radiography in acute ankle injuries . Ann Emerg Med . 1992;;21:384-390.
Stiell IG, Greenberg GH, McKnight RD, et al.  Decision rules for the use of radiography in acute ankle injuries: refinement and prospective validation . JAMA . 1993;;269:1127-1132.
Stiell I, Wells G, Laupacis A, et al.  Multicentre trial to introduce the Ottawa ankle rules for use of radiography in acute ankle injuries . BMJ . 1995;;311:594-597.
Stiell IG, Greenberg GH, Wells GA, et al.  Prospective validation of a decision rule for the use of radiography in acute knee injuries . JAMA . 1996;;275:611-615.
Sauve JS, Thorpe KE, Sackett DL, et al.  Can bruits distinguish high-grade from moderate symptomatic carotid stenosis? Ann Intern Med . 1994;;120:633-637.
Farr BM, Sloman AJ, Fisch MJ.  Predicting death in patients hospitalized for community-acquired pneumonia . Ann Intern Med . 1991;;115:428-436.
Marottoli RA, Cooney LM, Wagner R, Doucette J, Tinetti ME.  Predictors of automobile crashes and moving violations among elderly drivers . Ann Intern Med . 1994;;121:842-846.
Silver MT, Rose GA, Paul SD, O'Donnell CJ, O'Gara PT, Eagle KA.  A clinical rule to predict preserved left ventricular ejection fraction in patients after myocardial infarction . Ann Intern Med . 1994;;121:750-756.
Pryor DB, Shaw L, McCants CB, et al.  Value of the history and physical in identifying patients at increased risk for coronary artery disease . Ann Intern Med . 1993;;118:81-90.
Williams JW, Simel DL, Roberts L, Samsa GP.  Clinical evaluation of sinusitis: making the diagnosis by history and physical examination . Ann Intern Med . 1992;;117:705-710.
Ashton CM, Petersen NJ, Wray NP, et al.  The incidence of perioperative myocardial infarction in men undergoing noncardiac surgery . Ann Intern Med . 1993;;118:504-510.
The Stroke Prevention in Atrial Fibrillation Investigators.  Predictors of thromboembolism in atrial fibrillation, I: clinical features of patients at risk . Ann Intern Med . 1992;;116:1-5.
The Stroke Prevention in Atrial Fibrillation Investigators.  Predictors of thromboembolism in atrial fibrillation, II: echocardiographic features of patients at risk . Ann Intern Med . 1992;;116:6-12.
Morrow K, Morris CK, Frolicher VF, et al.  Prediction of cardiovascular death in men undergoing noninvasive evaluation for coronary artery disease . Ann Intern Med . 1993;;118:689-695.
Zarchy TM, Ershoff D.  Which clinical variables predict an abnormal double-contrast barium enema result? Ann Intern Med . 1991;;114:137-141.
Inouye SK, Viscoli CM, Horwitz RI, Hurst LD, Tinetti ME.  A predictive model for delirium in hospitalized elderly medical patients based on admission characteristics . Ann Intern Med . 1993;;119:474-481.
Nguyen T, Sambrook P, Kelly P, et al.  Prediction of osteoporotic fractures by postural instability and bone density . BMJ . 1993;;307:1111-1115.
Medical Research Council Antiepileptic Drug Withdrawal Study Group.  Prognostic index for recurrence of seizures after remission of epilepsy . BMJ . 1993;;306:1374-1378.
van Duijn NP, Brouwer HJ, Lamberts H.  Use of symptoms and signs to diagnose maxillary sinusitis in general practice: comparison with ultrasonography . BMJ . 1992;;305:684-687.
Skene AI, Smith JM, Dore CJ, Charlett A, Lewis JD.  Venous leg ulcers: a prognostic index to predict time to healing . BMJ . 1992;;305:1119-1121.
Poungvarin N, Viriyavejakul A, Komontri C.  Siriraj stroke score and validation study to distinguish supratentorial intracerebral hemorrhage from infarction . BMJ . 1991;;302:1565-1567.
Celani MG, Righetti E, Migliacci R, et al.  Comparability and validity of two clinical scores in the early differential diagnosis of acute stroke . BMJ . 1994;;308:1674-1676.
Dramaix M, Hennart P, Brasseur D, et al.  Serum albumin concentration, arm circumference, and oedema and subsequent risk of dying in children in Central Africa . BMJ . 1993;;307:710-713.
Parsons RW, Jamrozik KD, Hobbs MST, Thompson DL.  Early identification of patients at low risk of death after myocardial infarction and potentially suitable for early hospital discharge . BMJ . 1994;;308:1006-1010.
Nouri FM, Lincoln NB.  Predicting driving performance after stroke . BMJ . 1993;;307:482-483.
Bates DW, Lee TH.  Rapid classification of positive blood cultures: prospective validation of a multivariate algorithm . JAMA . 1992;;267:1962-1966.
Marcantonio ER, Goldman L, Mangione CM, et al.  A clinical prediction rule for delirium after elective noncardiac surgery . JAMA . 1994;;271:134-139.
Browner WS, Li J, Mangano DT, for the Study of Perioperative Ischemia Research Group.  In-hospital and long-term mortality in male veterans following noncardiac surgery . JAMA . 1992;;268:228-232.
Chodak GW, Vogelzang NJ, Caplan RJ, Soloway M, Smith JA.  Independent prognostic factors in patients with metastatic (stage D2) prostate cancer . JAMA . 1991;;265:618-621.
Hollenberg M, Mangano DT, Browner WS, et al.  Predictors of postoperative myocardial ischemia in patients undergoing noncardiac surgery . JAMA . 1992;;268:205-209.
Cooper-Patrick L, Crum RM, Ford DE.  Identifying suicidal ideation in general medical patients . JAMA . 1994;;272:1757-1762.
The International Non-Hodgkins Lymphoma Prognostic Factors Project.  A predictive model for aggressive non-Hodgkin's lymphoma . N Engl J Med . 1993;;329:987-994.
Lee TH, Juarez G, Cook EF, et al.  Ruling out acute myocardial infarction: a prospective multicenter validation of a 12-hour strategy for patients at low risk . N Engl J Med . 1991;;324:1239-1246.
Williams M, Warwick D, Bannister GC.  Death and thromboembolic disease after total hip replacement without routine chemical prophylaxis . Thromb Haemost . 1995;;73:953.
Wells PS, Hirsh J, Anderson DR, et al.  Accuracy of clinical assessment of deep-vein thrombosis . Lancet . 1995;;345:1326-1330.
Sackett DL, Haynes RB, Guyatt GH, Tugwell P. Clinical Epidemiology: A Basic Science for Clinical Medicine . 2nd ed. Boston, Mass: Little Brown & Co Inc; 1991;:179-180.
Jutras D.  Clinical practice guidelines as legal norms . Can Med Assoc J . 1993;;148:905-908.
Ciampi A, Hogg SA, McKinney S, Thiffault J.  RECPAM: A computer program for recursive partitioning and amalgamation for censored survival data and other situations frequently occurring in biostatistics, I: methods and program features . Comput Methods Programs Biomed . 1988;;26:239-256.
Ciampi A, Chang CH, Hogg S, McKinney S.  Recursive partitioning: a versatile method for exploratory data analysis in biostatistics . In: MacNeill IB, Umphrey GJ, eds. Time Series and Econometric Modelling/Biostatistics . Boston, Mass: D Reidel Publishing Co; 1987;:23-50.
Concato J, Feinstein AR, Holford TR.  The risk of determining risk with multivariable models . Ann Intern Med . 1993;;118:201-210.
Baxt WG.  Use of an artificial neural network for the diagnosis of myocardial infarction . Ann Intern Med . 1991;;115:843-848.
Shufflebarger CM, Young W.  What is a neural network? Ann Emerg Med . 1992;;21:1461-1462.
The PIOPED Investigators.  Value of the ventilation/perfusion scan in acute pulmonary embolism: results of the prospective investigation of pulmonary embolism diagnosis (PIOPED) . JAMA . 1990;;263:2753-2759.
Rifkin MD, Zerhouni EA, Gatsonis CA, et al.  Comparison of magnetic resonance imaging and ultrasonography in staging early prostate cancer: results of a multi-institutional cooperative trial . N Engl J Med . 1990;;323:621-626.
Hanley JA, McNeil BJ.  The meaning and use of the area under a receiver operating characteristic (ROC) curve . Radiology . 1982;;143:29-36.
Fletcher RH, Fletcher SW, Wagner EH. Clinical Epidemiology: The Essentials . 2nd ed. Baltimore, Md: Williams & Wilkins; 1988;.
Kerr L, Kelly A, Grant J, et al.  Failed validation of a clinical decision rule for the use of radiography in acute ankle injury . N Z Med J . 1994;;107:294-295.
Stiell IG, Greenberg GH, McKnight RD, Wells GA.  Ottawa ankle rules for radiography of acute ankle injuries . N Z Med J . 1995;;108:111.
Fleiss JL. Statistical Methods for Rates and Proportions . 2nd ed. New York, NY: John Wiley & Sons Inc; 1981;.
Feinstein AR. Clinimetrics . New Haven, Conn: Yale University Press; 1987;.
Kirshner B, Guyatt G.  A methodological framework for assessing health indices . J Chronic Dis . 1985;;38:27-36.
The Standards of Reporting Trials Group.  A proposal for structured reporting of randomized controlled trials . JAMA . 1994;;272:1926-1931.
The Asilomar Working Group on Recommendations for Reporting of Clinical Trials in the Biomedical Literature.  Checklist of information for inclusion in reports of clinical trials . Ann Intern Med . 1996;;124:741-743.
CME Course for:


You need to register in order to view this quiz.


To understand the clinical management of acute heart failure syndromes.
Accreditation Information The American Medical Association is accredited by the Accreditation Council for Continuing Medical Education to provide continuing medical education for physicians.
The AMA designates this journal-based CME activity for a maximum of 1 AMA PRA Category 1 CreditTM per course. Physicians should claim only the credit commensurate with the extent of their participation in the activity.
Physicians who complete the CME course and score at least 80% correct on the quiz are eligible for AMA PRA Category 1 CreditTM.
Note: You must get at least of the answers correct to pass this quiz.
Note: You must get at least of the answers correct to pass this quiz.
You have not filled in all the answers to complete this quiz
The following questions were not answered:
Sorry, you have unsuccessfully completed this CME quiz with a score of
The following questions were not answered correctly:
For CME Course: A Proposed Model for Initial Assessment and Management of Acute Heart Failure Syndromes
Indicate what changes(s) you will implement in your practice, if any, based on this CME course.
To view and print your certificate and access a summary of your CME courses go to My CME.
NOTE:
Citing articles are presented as examples only. In non-demo SCM6 implementation, integration with CrossRef’s “Cited By” API will populate this tab (http://www.crossref.org/citedby.html).
Submit a Response

Some tools below are only available to our subscribers or users with an online account.

Related Content

Customize your page view by dragging & repositioning the boxes below.