Users' Guides to the Medical Literature: Title and subTitle BreakXXII: How to Use Articles About Clinical Decision Rules

You are the medical director of a busy inner-city emergency department. Faced with a limited budget and pressure to improve efficiency, you have conducted an audit of radiological procedures ordered for minor trauma and found a high rate of x-rays ordered for ankle and knee trauma. You are aware of the Ottawa ankle rules (Figure 1) that identify patients for whom ankle radiographs can be omitted without adverse consequences. In addition, you are aware that a small number of faculty and residents currently rely on these models to make quick frontline decisions in the emergency department.

You are interested in knowing the accuracy of the rules, whether they are applicable to your patient population, and whether you should be implementing the rules in your own practice. Furthermore, you wonder if implementing the rules can change clinical behavior and reduce costs without compromising quality care. You decide to consult the original medical literature and to assess the evidence for yourself.

Currently, decision rules have no separate medical subject heading (MeSH) in the National Library of Medicine MEDLINE database. You therefore search PubMed under the MeSH heading ankle fractures and add the text words rules and decision rules. This search yields 5 citations, of which 3 deal directly with the Ottawa clinical decision rules for ankle fractures.^{1 - 3}

In reviewing these articles and deciding whether to implement changes in your emergency department, you require criteria for determining the strength of the inference you can make about the accuracy and impact of the Ottawa ankle rules. This article will provide you with the tools to answer those questions.

Establishing patients' diagnosis and prognosis are closely linked activities central to every physician's practice. The diagnoses we make and our assessment of patients' prognosis often determine the recommendations we make to our patients. Clinical experience provides us with an intuitive sense of which findings on history, physical examination, and investigation are critical in making an accurate diagnosis or an accurate assessment of our patients' condition. While often extraordinarily accurate, this intuition may sometimes be misleading.

A clinical decision rule (CDR) can be defined as a clinical tool that quantifies the individual contributions that various components of the history, physical examination, and basic laboratory results make toward the diagnosis, prognosis, or likely response to treatment in an individual patient.⁴ Clinical decision rules attempt to formally test, simplify, and increase the accuracy of clinicians' diagnostic and prognostic assessments and are most likely to be useful in situations where decision making is complex, the clinical stakes are high, or there are opportunities to achieve cost savings without compromising patient care. Available CDRs include guides for whether to treat sore throats⁵ and for establishing a pretest probability of pulmonary embolus.⁶ Other CDRs provide screening tests for common problems that frequently go undetected, including alcoholism⁷ and depression.⁸ Another category of CDRs help estimate risk, such as the risk of developing delirium in hospitalized patients⁹ or the risk of bleeding while receiving anticoagulation therapy.¹⁰

Developing and testing a CDR involves 3 steps: creating or deriving the rule, testing or validating the rule, and assessing the impact of the rule on clinical behavior (impact analysis). The validation process may require several studies to fully test the accuracy of the rule at different clinical sites (Figure 2). Each step in the development of a CDR may be published separately by different authors, or all 3 steps may be included in a single article. Table 1 presents a hierarchy that can guide clinicians in assessing the full range of evidence supporting use of a CDR in their practice.

We note that our hierarchy applies only to CDRs intended for application in clinical practice. Investigators may use identical methodology to generate equations that stratify patients into different risk groups for nonclinical purposes. For example, investigators can use such equations for statistical adjustment in studies involving large databases. These rules, which are not so clinical, do not involve application by front-line practitioners, and thus require a somewhat different hierarchy of strength of evidence.

We will now review the steps in the development and testing of a CDR. We will relate each stage of the process to the hierarchy presented in Table 1. Although we will address issues of interest to investigators engaged in developing CDRs, we do so only for the purpose of equipping our clinician readers with the knowledge and tools they need to evaluate existing CDRs for application to clinical practice.

Our search found 3 articles related to the Ottawa ankle rules, the first of which described the CDR derivation.¹ Investigators who develop a CDR begin by constructing a list of potential predictors of the outcome of interest, in this case, radiological ankle fractures. The list typically includes items from the history, physical examination, and basic laboratory tests. The investigators then examine a group of patients and determine if the candidate clinical predictors are present and the patient's status on the outcome of interest, in this case, the result of the ankle radiograph. Statistical analysis reveals which predictors are most powerful and which predictors can be omitted from the rule without loss of predictive power. Typically, the statistical techniques used in this process are based on logistic regression; readers can find a clinician-friendly description of these methods in another article.¹¹ Other techniques that investigators sometimes use include discriminant analysis,¹² which produces equations similar to regression analysis; recursive partitioning analysis, which builds a tree in which the patient populations are split into smaller and smaller categories based on risk factors¹³ ; and neural networks.¹⁴

Clinical decision rules that investigators have derived, but not validated, should not be considered ready for clinical application (Table 1). Investigators interested in performing the validation of a CDR, however, need criteria to judge whether investigators have conducted a rigorous derivation process and, thus, whether the rule is promising enough to move forward to the validation phase. A list of important criteria for derivation is provided in Table 2. Interested readers can find a complete discussion on the derivation process and these criteria in an article by Laupacis et al.⁴

There are 3 reasons why even rigorously derived CDRs are not ready for application in clinical practice without further validation. First, CDRs may reflect associations between given predictors and outcomes that are due primarily to chance. If that is so, a different set of predictors will emerge in a different group of patients, even if the patients come from the same setting. Second, predictors may be idiosyncratic to the population, to the clinicians using the rule, or to other aspects of the design of individual studies. If that is so, the rule may fail in a new setting. Perhaps most important, clinicians may, because of problems in the feasibility of rule application in the clinical setting, fail to implement a rule comprehensively or accurately. The result would be that a rule succeeds in theory but fails in practice.

Statistical methods can deal with the first of these problems. For instance, investigators may split their population into 2 groups and use one to develop the rule and the other to test it. Alternatively, they may use more sophisticated statistical methods built on the same logic. Conceptually, these approaches involve removing 1 patient from the sample, generating the rule using the remainder of the patients, and testing it on the patient who was removed from the sample. This procedure, sometimes referred to as a bootstrap technique, is repeated in sequence for every patient being studied.

While statistical validation within the same setting or group of subjects reduces the likelihood that the rule reflects the play of chance rather than true associations, it fails to address the other 2 threats to validity. The success of the CDR may be peculiar to the particular populations of patients and clinicians involved in the derivation study. Even if this is not so, clinicians may have difficulties using the rule in practice, difficulties that compromise its predictive power. Thus, to graduate from level 4, studies must involve clinicians actually using the rule in practice.

A CDR developed to predict serious outcomes (eg, heart failure and ventricular arrhythmia) in syncope patients highlights the importance of validation.¹⁵ Investigators derived the rule using data from 252 patients who presented to the emergency department and then attempted to prospectively validate it in a sample of 374 patients. The CDR gave individuals a score from 0 to 4, depending on the number of clinical predictors present. The probability of poor outcomes corresponding to almost every score in the derivation set was approximately twice that of the validation. For example, in the derivation set the risk of a poor outcome in a patient with a score on the CDR of 3 was estimated to be 52%; a patient with the same score in the validation set had a probability of a poor outcome of only 27%. This variation in results may have been caused by a difference in the severity of the syncope cases entered into the 2 studies or to different criteria for generating a score of 3. Because of the risk that it will provide misleading information when applied in a real-world clinical setting, we situate a CDR that has undergone development without validation as level 4 on our hierarchy (Table 1).

Despite this major limitation, clinicians can still extract clinically relevant messages from an article describing the development of a CDR. They may wish to note the most important predictors and consider them more carefully in their own practice. They may also consider giving less importance to variables that failed to show predictive power. For instance, in developing a CDR to predict mortality from pneumonia, the investigators found that white blood cell count had no bearing on subsequent mortality.¹⁶ This being the case, clinicians maywish to put less weight on white blood cell count when making decisions about admitting pneumonia patients to the hospital.

To move up the hierarchy, CDRs must provide additional evidence of validity. The second article found in our search described the refinement and prospective validation of the Ottawa ankle rules.² Validation of a CDR involves demonstrating that its repeated application as part of the process of clinical care leads to the same results. Ideally, a validation entails the investigators applying the rule prospectively in a new population with a different prevalence and spectrum of disease from that of the patients in whom the rule was derived. One key issue is to be sure that the CDR performs similarly in a variety of populations and in the hands of a variety of clinicians working in a variety of institutions. A second issue is to be sure that the CDR works well when clinicians are applying it consciously as a rule, as opposed to a purely statistical validation.

If the setting in which the CDR was originally developed was limited and its validation has been confined to this setting, application by clinicians working in other settings is less secure. Validation in a similar setting can take a number of forms. Most simply, after developing the CDR, the investigators return to their population, draw a new sample of patients, and test the rule's performance. Thus, we classify rules that have been validated in the same, or very similar limited or narrow populations, to the sample used in the development as level 3 on our hierarchy and recommend clinicians use the results cautiously (Table 1).

If investigators draw patients in the derivation phase from a sufficiently heterogeneous population across a variety of institutions, testing the rule in the same population provides strong validation. Validation in a new population provides the clinician with strong inferences about the usefulness of the rule, corresponding to level 2 in our hierarchy (Table 1).

The Ottawa ankle rules were first derived in 2 large university-based emergency departments in Ottawa¹ and were then prospectively validated in a large sample of patients from the same emergency departments.² At this stage, the rules would be classified as level 2 in our hierarchy because of the large number and diversity of patients and physicians involved in the study. Since that initial validation, the rules have been validated in several different clinical sites with relatively consistent results.^{17 - 20} This evidence even further strengthens our inference about their predictive power.

Many CDRs are derived and then validated in a small, narrowly selected group of patients (level 3). One such rule was derived to predict preserved left ventricular function after a myocardial infarction.²¹ The initial derivation relied on data from 314 patients admitted to 1 tertiary care center. The investigators derived the rule using data from 162 patients and then performed a validation in 152 patients in the same setting. Of those whom the CDR identified as having preserved ejection fraction, 99% indeed had preserved left ventricular function. At this stage, we would consider the rule had met criteria for level 3, and its use should be restricted to settings similar to the validation study, ie, similar coronary care unit settings.

Investigators further validated the CDR for preserved left ventricular function, in 2 larger trials, one that enrolled 213 patients²² from a single site and a larger trial that enrolled 1891 patients from several different institutions.²³ In both studies, of those patients predicted to have preserved ventricular function (ejection fraction >40%), 86% actually had preserved ventricular function. This drop in predictive value changes the implications of applying the rule in clinical practice. At this point in development, the rule would be considered level 2, meaning that the rule can used in clinical settings with a high degree of confidence but with the adjusted values. The development of this rule highlights the importance of the validation of a rule in a diverse patient population before broadly applying it in clinical settings.

Whether or not investigators have conducted their validation study in a similar, narrow (level 3) population or a broad, heterogeneous (level 2) population, their results allow stronger inferences if they have adhered to the methodological standards listed in Table 3. First, were the patients chosen in an unbiased fashion, and do they represent a wide spectrum of severity of disease? Second, was there a blinded assessment of the criterion standard for all patients? Third, was there an explicit and accurate interpretation of the predictor variables and actual rule without knowledge of the outcome? If those evaluating predictor status of study patients are aware of the outcome or if those assessing the outcome are aware of patients' status with respect to the predictors, their assessments may be biased. For instance, in a CDR developed to predict the presence of pneumonia in patients presenting with cough,²⁴ the authors make no mention of blinding during either the derivation or the validation process. Knowledge of history or physical examination findings may have influenced the judgements of the unblinded radiologists. Lastly, investigators should achieve close to 100% follow-up of those they enrolled. Interested readers can find a complete discussion of the validation process and these criteria in an article by Laupacis et al.⁴

The investigators testing the Ottawa ankle rules enrolled consecutive patients, obtained radiographs for all of them, and ensured that not only were the clinicians assessing the clinical predictors unaware of the radiographic results but that the radiologists had no knowledge of the clinical data.

Whatever the level of evidence associated with a CDR, its usefulness will depend on its predictive power. Investigators may report their results in a variety of ways. The ankle component of the Ottawa ankle rules states that an ankle x-ray series is only indicated for patients with pain near the malleoli and either inability to bear weight or localized bone tenderness at the posterior edge or tip of either malleolus (Figure 1). The developers calculated the sensitivity and specificity of their rule as a diagnostic test using this criterion. In the development process, all patients with fracture had a positive result (sensitivity of 100%), but only 40% of those without fractures had a negative result (specificity of 40%). These results suggest that if clinicians order radiographs only in those patients with a positive result they will not miss any fractures and will avoid the test in 40% of those without a fracture.

The validation study confirmed these results; in particular, the test maintained a sensitivity of 100%. This is reassuring, and more so because the sample size was sufficiently large to result in a relatively narrow confidence interval (CI) (95% CIs, 93%-100%). Thus, clinicians adopting the rule would miss very few, if any, fractures.

Another way of reporting CDR results is in terms of probability of the target condition being present given a particular CDR result. For example, a recent CDR for pulmonary embolus derived by Wells and colleagues⁶ placed patients into low (3.4%; 95% CI, 2.2%-5%), intermediate (28%; 95% CI, 23.4%-32.2%), or high probability (78%; 95% CI, 69.2%-86.0%) categories. When investigators report CDR results in this fashion, they are implicitly incorporating all clinical information. In doing so, they remove any need for clinicians to consider independent information in deciding on the likelihood of the diagnosis or a patient's prognosis.

Finally, CDRs may also report their results as likelihood ratios (LRs) or as absolute or relative risks. For example the CAGE, a CDR for detecting alcoholism, has been reported as LRs (eg, for CAGE scores of 0/4, LR=0.14; for 1/4, LR=1.5; for 2/4, LR=4.5; for 3/4, LR=13; and for 4/4, LR=100). In this example, the probability of disease, alcoholism, depends on the combination of the prevalence of disease in the community and the score on the CAGE CDR.⁷ When investigators report their results as LRs, they are implicitly suggesting that clinicians should use other, independent information to generate a pretest (or prerule) probability. They can then use the LRs generated by the rule to establish a posttest probability. Clinicians can find approaches to using LRs in clinical practice in a previous Users' Guide.²⁵

Use of a CDR involves remembering predictor variables and often entails making calculations to determine a patient's probability of having the CDR's target outcome. Pocket cards and computer algorithms can facilitate the task of using complex CDRs. Nonetheless, CDRs demand clinician time and energy, and their use is warranted only if they change physician behavior and if that behavior change results in improved patient outcomes or reduced costs while maintaining quality of care. If these conditions are not met, whatever the accuracy of a CDR, attempts to use it systematically will be a waste of time.

There are a number of reasons why an accurate CDR may not produce a change in behavior or an improvement in outcomes. First, clinicians' intuitive estimation of probabilities may be as good as, if not better than, the CDR. If this is so, CDR information will not improve their practice. Second, the calculations involved may be cumbersome, and clinicians may, as a result, not use the rule. Finally, there may be practical barriers to acting on the results of the CDR. For instance, in the case of the Ottawa ankle rules, clinicians may be sufficiently concerned about protecting themselves against litigation that they order radiographs despite a CDR result suggesting a negligible probability of fracture.

These are the considerations that lead us to classify a CDR with evidence of reproducible accuracy in diverse populations as level 2 and insist on a positive result from a study of impact before a CDR graduates to level 1.

Ideally, an impact study would randomize patients, or larger administrative units, to the application or nonapplication of the CDR and follow up patients for all relevant outcomes (including quality of life, morbidity, and resource utilization). Randomization of individual patients is unlikely to be appropriate because one would expect the participating clinicians to incorporate the rule into the care of all their patients. A suitable alternative is to randomize institutions or practice settings and conduct analyses appropriate to these larger units of randomization. Another potential design is to look at a group before and after clinicians began to use the CDR and compare that with a control group in which there has been no intervention.

Investigators examining the impact of the Ottawa ankle rules randomized 6 emergency departments to use or not use their CDR.³ Prior to initiating the study, 1 center dropped out, leaving a total of 5 emergency departments, 2 in the intervention group and 3 in the usual care group. The intervention consisted of introducing the CDR at a general meeting, distributing pocket cards summarizing the rules, posting the rule throughout the emergency department, and applying preprinted data collection forms to each chart. In the control group, the only intervention was the introduction of preprinted data collection forms without the Ottawa ankle rules attached to each chart.

A total of 1911 eligible patients entered the study: 1005 in the control group and 906 in the intervention group. There were 691 radiographs requested in the intervention group and 996 in the control group. In an analysis that focused on the ordering physician, the investigators found that the mean proportion of patients referred for radiography was 99.6% in the control group and 78.9% in the intervention group (P=.03). The investigators noted 3 missed fractures in the intervention group, none of which led to adverse outcomes. Thus, the investigators demonstrated a positive resource utilization impact of the Ottawa ankle rules (decreased test ordering) without increase in adverse outcomes, moving the CDR to level 1 in the hierarchy (Table 1).

You have found level 1 evidence supporting the use of the Ottawa ankle rules in reducing unnecessary ankle radiographs in patients presenting to the emergency department with ankle injuries. You therefore feel confident that you can productively use the rule in your own practice. However, another recent study makes you aware that changing the behavior of your colleagues to realize the possible reductions in cost may be a challenge: Cameron and Naylor²⁶ reported on an initiative in which clinicians expert in the use of the Ottawa ankle rules trained 16 other individuals to teach the use of the rules. These individuals returned to their emergency departments armed with slides, overheads, a 13-minute instructional video, and a mandate to train their colleagues locally and regionally in the use of the rules.

Unfortunately this program led to no change in the use of ankle radiography. The results demonstrate that even the availability of a level 1 CDR may require local implementation strategies with known effectiveness in changing provider behavior to ensure implementation.^{27 - 29} Among the possible strategies, which are most likely to be effective if used as part of a package of interventions, include computer reminders, mobilization of local opinion leaders, one-to-one conversations with a respected information source (academic detailing), and audit and feedback.

Clinical decision rules inform our clinical judgment and have the potential to change clinical behavior and reduce unnecessary costs while maintaining quality of care and patient satisfaction. The challenge for clinicians is to evaluate the strength of the rule and its likely impact and to find ways of efficiently incorporating level 1 rules into their daily practice.

A summary of some frequently used CDRs, evaluated in an evidence-based fashion (ie, highlighting the level of evidence), is currently available on the Internet for clinician use (http://med.mssm.edu/ebm).