Effects of Computerized Clinical Decision Support Systems on Practitioner Performance and Patient Outcomes: Title and subTitle BreakA Systematic Review FREE

Computerized clinical decision support systems (CDSSs) are information systems designed to improve clinical decision making. Characteristics of individual patients are matched to a computerized knowledge base, and software algorithms generate patient-specific recommendations. Practitioners, health care staff, or patients can manually enter patient characteristics into the computer system; alternatively, electronic medical records can be queried for retrieval of patient characteristics. Computer-generated recommendations are delivered to the clinician through the electronic medical record, by pager, or through printouts placed in a patient’s paper chart. Such systems have been developed for a myriad of clinical issues, including diagnosis of chest pain, treatment of infertility, and timely administration of immunizations. These systems provide several modes of decision support, including alerts of critical values, reminders of overdue preventive health tasks, advice for drug prescribing, critiques of existing health care orders, and suggestions for various active care issues.

As with any health care innovation, CDSSs should be rigorously evaluated before widespread dissemination into clinical practice. Various stages in this assessment process have been previously described. Iterative qualitative and quantitative assessment begin early in the software development cycle.^{1 - 2} When preliminary testing suggests that a CDSS improves clinical care or patient outcomes, confirmatory controlled trials are warranted. We previously reviewed controlled trials of computer-aided quality assurance³ and CDSSs published up to 1992⁴ and 1998.⁵ This field is rapidly evolving because of technological advances, increasing access to computer systems in clinical practice, and growing concern about the process and quality of medical care. We therefore updated previous reviews to provide a cumulative summary of controlled trials evaluating the effectiveness of CDSSs on practitioner performance and patient outcomes.

The primary questions of this review were (1) Do CDSSs improve practitioner performance or patient outcomes? and (2) Which CDSS and study-level factors are associated with effective CDSSs? A priori, we hypothesized that studies reporting better outcomes would assess CDSSs that automatically prompted users (vs requiring the user to actively initiate the system), were built into an electronic medical record or computer order entry system (vs a stand-alone system), provided reminders (vs information on disease management, drug dosing, or diagnosis), were tested using less rigorous study methods, were studied by their software developers (vs by evaluators not involved in the CDSS design), described pilot testing, and described user training.

We included English-language randomized and nonrandomized trials with a contemporaneous control group that compared patient care with a CDSS to routine care without a CDSS and evaluated clinical performance (ie, a measure of process of care) or a patient outcome. We stipulated that the CDSS had to provide patient-specific advice that was reviewed by a health care practitioner before any clinical action. Studies were excluded if the system (1) was used solely by medical students, (2) only provided summaries of patient information, (3) provided feedback on groups of patients without individual assessment, (4) only provided computer-aided instruction, or (5) was used for image analysis. Studies assessing CDSS diagnostic performance against a defined gold standard were not included in this review unless clinical use of the diagnostic CDSS was also compared with routine care. Based on these criteria, we reevaluated all studies from our previous reviews for inclusion.

We have previously described our methods for finding relevant studies until March 1998.⁵ For this update, we examined citations from MEDLINE, EMBASE, Evidence-Based Reviews databases (Cochrane Database of Systematic Reviews, ACP Journal Club, Database of Abstracts of Reviews of Effects, and Cochrane Central Register of Controlled Trials), and Inspec bibliographic databases from 1998 through September 2004. All citations were downloaded into Reference Manager, version 10.0 (Thomson ISI ResearchSoft, Philadelphia, Pa). An experienced librarian developed the search strategies using sensitive terms for identifying clinical studies of CDSSs. We pilot-tested search strategies and modified them to ensure that they identified known eligible articles. The final strategies used the terms computer-assisted decision making, computer-assisted diagnosis, computer-assisted therapy, decision support systems, reminder systems, hospital information systems, randomized controlled trial, and cohort studies (complete strategies available from the authors). Pairs of reviewers independently evaluated the eligibility of all studies identified in our search. Disagreements were resolved by a third reviewer or by consensus. Full-text articles were retrieved if any reviewer considered a citation potentially relevant. Supplementary methods of finding studies included a review of article reference lists, articles citing included studies as listed in the Science Citation Index, PubMed related articles feature, informatics conference proceedings, information provided by primary study authors, and other recent reviews.^{6 - 11} Where data from a trial were distributed in more than 1 publication, we cited the principal publication.

Pairs of reviewers independently abstracted the following data from all studies meeting eligibility criteria: study setting, study methods, CDSS characteristics, patient characteristics, and outcomes. Disagreements were resolved by a third reviewer or by consensus. We attempted to contact primary authors of all included studies to confirm data and provide missing data.

All studies were scored for methodological quality on a 10-point scale consisting of 5 potential sources of bias, which we have described elsewhere.⁵ In brief, we considered the method of allocation to study groups (random, 2, vs quasi-random, 1, vs selected concurrent controls, 0), the unit of the allocation (a cluster such as a practice, 2, vs physician, 1, vs patient, 0), the presence of baseline differences between the groups that were potentially linked to study outcomes (of particular importance for observational studies; no baseline differences present or appropriate statistical adjustments made for differences, 2, vs baseline differences present and no statistical adjustments made, 1, vs baseline characteristics not reported, 0), the objectivity of the outcome (objective outcomes or subjective outcomes with blinded assessment, 2, vs subjective outcomes with no blinding but clearly defined assessment criteria, 1, vs subjective outcomes with no blinding and poorly defined, 0), and the completeness of follow-up for the appropriate unit of analysis (>90%, 2, vs 80 to 90%, 1, vs <80% or not described, 0). The unit of allocation was included because of the possibility of group contamination in trials in which interventions were applied to clinicians even though individual patients were allocated to the intervention and control groups.¹² Contamination bias would lead to underestimating the effect of a CDSS.

The studies substantially differed in the type and number of outcomes assessed. In addition, the majority of studies did not define a single outcome for statistical testing. We aimed to efficiently summarize the benefits of CDSSs and to identify CDSS and study characteristics that predicted success. For a given study we abstracted all reported practitioner performance and patient health outcomes. Situations where the CDSS worsened outcomes were rare. Thus, for each study we defined the effects of CDSSs in terms of success, defined as an improvement in at least 50% of outcomes measured, each at a 2-sided significance level less than .05.

Reviewer agreement on study eligibility was quantified using the Cohen κ.¹³ Study and CDSS characteristics predicting success were analyzed and interpreted with the study as the unit of analysis. Data were summarized using descriptive summary measures, including proportions for categorical variables and mean (standard deviation) for continuous variables. Univariable and multivariable logistic regression models, adjusted for study methodological quality, were used to investigate associations between the outcomes of interest and study-specific covariates defined in our a priori hypotheses. All analyses were carried out using the SAS statistical package, version 8.2 (SAS Institute Inc, Cary, NC). We interpreted P≤.05 as indicating statistical significance; all P values are 2-sided. When reporting results from individual studies, we cited the measures of association and P values reported in the studies.

From 3997 screened citations, we retrieved 226 full-text articles, and 100 trials met our criteria for review. The chance-corrected agreement between 2 independent reviewers for article inclusion was good (κ = 0.81; 95% confidence interval [CI], 0.73-0.88).

The 100 trials examined more than 3826 practitioners or practices (median, 42; range, 2-300 [when reported]) caring for more than 92 895 patients (median, 488; range, 19-12 989 [when reported]) from 1973 to 2004.^{14 - 57 58 - 113} The number of eligible trials increased with time: 1 in 1970-1974, 4 in 1975-1979, 10 in 1980-1984, 13 in 1985-1989, 20 in 1990-1994, 26 in 1995-1999, and 26 in 2000–September 2004. Of these 100 trials, most were conducted in the United States (69%), followed by the United Kingdom (14%), Canada (5%), Australia (4%), Italy (2%), and Austria, France, Germany, Israel, Norway, and Switzerland (1% each). Sixty-nine percent of trials described funding from the public sector and 16% from the private sector. Developers of CDSS software were also study authors in 72% of trials. Ninety-seven trials described the effect of CDSS on at least 1 measure of health care practitioner performance. Fifty-two trials assessed at least 1 patient outcome. We successfully contacted authors of 91 trials, and authors of 64 trials provided additional information or confirmed the accuracy of abstracted data.^{15 - 18 ,20 - 21 ,24 - 33 ,35 - 40 ,42 - 43 ,46 - 47 ,49 ,51 - 52 ,56 ,60 - 64 ,67 - 68 ,71 ,73 - 75 ,80 - 81 ,83 - 98 ,101 ,106 ,113 - 115}

Trial methodological rigor increased with time—36% of trials before the year 2000 were cluster randomized, compared with 67% after this time (P = .01). Of all trials, 88% were randomized. Of the randomized trials, 49% were cluster randomized and 40% used a cluster as the unit of analysis or adjusted for clustering in the analysis. Twenty-four randomized trials and 1 cohort study reported a power calculation for a prespecifed difference between groups on a specific outcome. Fifteen of these trials (60%) calculated sample size based on a practitioner performance outcome, 9 (36%) based on a patient outcome, and 1 (4%) based on the cost of prescribed medications. Only 2 studies examined patient outcomes without measuring practitioner performance. Of the 88 randomized trials, 52% described an appropriate method of generating random numbers and 28% reported allocation concealment. On the 10-point methods scale, the mean score was 7 (SD, 1.7) and the range was 2 to 10.

The 100 trials had the following characteristics: 92% of trials enrolled physicians as primary users, 48% enrolled training health care practitioners (interns and residents) as users, 34% described pilot testing with users prior to implementation, 42% described user instructional training at the time of implementation, 76% took place in academic centers, and 33% were inpatient-based. In 47% of studies, the CDSS was part of an electronic medical record or computer order entry system. Most of these were early generation systems lacking the full functionality of current systems. In 15% of studies, the CDSS had a graphical user interface. Feedback from the CDSS occurred at the time of patient care in 88% of studies; in 60% the user was automatically prompted to use the system (vs the user actively initiating the system), and in 91% the CDSS suggested new orders (vs critiquing existing orders). Expert physician opinion or clinical practice guidelines usually formed the knowledge base for the CDSS.

The process of data entry into the CDSS was clear in 80% of trials, some of which used more than 1 method. Existing personnel most often entered data (attending or training physician, 38%; other health care staff [eg, nurses, clerks], 29%), although many trials used staff paid by research funds (21%) or automated data capture from an electronic medical record (30%). The method of delivering computer recommendations to the clinician was clear in 81% of trials. Most CDSSs directly provided the recommendation on a computer screen viewed by the practitioner (41% of all trials) or generated printed reports that were placed in medical charts by health care staff (29%) or by staff paid by research funds (16%). Only 13% of trials evaluated the impact of the CDSS on clinician workflow, with more than half of these CDSSs requiring more time and effort from the user compared with paper-based methods.

There were 10 trials evaluating diagnostic systems (Table 1). All studies measured practitioner performance, and the CDSS was beneficial in 4 studies (40%). Two of the 4 successful CDSSs were diagnostic systems for cardiac ischemia in the emergency department, and these decreased the rate of unnecessary hospital or coronary care admissions by 15% (P<.05).^{18 ,20} The third increased mood disorder screening in a posttraumatic stress disorder clinic by 25% (P = .008).¹⁵ The fourth improved the time to diagnosis of acute bowel obstruction (1 hour when computer was used vs 16 hours when diagnosis was made with contrast radiography; P<.001).²³ Of the 5 trials assessing patient outcomes, none reported an improvement.

There were 21 trials evaluating reminder systems for prevention (Table 2). All trials measured practitioner performance, and the CDSS was beneficial in 16 studies (76%). Performance outcomes were usually rates of screening, counseling, vaccination, testing, medication use, or the identification of at-risk behaviors. Successful use of CDSSs was typically demonstrated in ambulatory care, although 1 system was successful in hospitalized patients.⁴⁴ The single trial measuring patient outcomes failed to demonstrate an improvement in the primary analysis.³⁴ Post hoc subgroup analyses, however, demonstrated a significant reduction in winter hospitalization and emergency department visits in patients eligible for pneumococcal or influenza vaccination. One trial examined the effect of adding a cervical cancer screening reminder to an existing mammography reminder system.³⁰ This trial suggested no interaction between the 2 reminders on screening efficacy.

There were 40 studies of CDSSs for active health conditions. These CDSSs improved practitioner performance in 23 (62%) of 37 studies evaluating this outcome. Of the 27 trials measuring patient outcomes, 5 (18%) demonstrated improvements.

For diabetes care, practitioner performance was usually judged by rates of retinal, foot, urine protein, blood pressure, and cholesterol examinations, with 5 (71%) of 7 trials reporting improvements (Table 3). Similarly, in studies of cardiovascular prevention, performance was judged by blood pressure and cholesterol assessment, identification of smoking, and use of cardioprotective medications, with 5 (38%) of 13 trials reporting improvements (Table 4). One CDSS provided electrocardiogram recommendations to improve thrombolytic prescribing in emergency departments.⁶¹ Other CDSSs varied in purpose, providing recommendations for urinary incontinence, human immunodeficiency virus infection management, functional assessment, and acute respiratory distress syndrome, with 6 of 9 reporting improvements (Table 5). Clinical decision support system corollary orders were used to monitor the effects of other prescribed treatments, such as the need for renal biochemistry measurements in patients receiving amphotericin B,⁷⁹ with all 4 trials reporting improvements (Table 6). Trials testing CDSS performance to reduce unnecessary health care utilization measured the frequency of redundant testing and unnecessary hospital admissions and hospital length of stay, with 3 of 4 trials reporting improvements (Table 6).

Five CDSSs (18%) examining patient outcomes described improvements. One CDSS improved blood pressure control (70% of patients had controlled blood pressure with CDSS use vs 52% with routine care; P<.05).⁵⁴ A second CDSS reduced urinary incontinence in nursing home residents over a 10-week period (23% incontinent with CDSS vs 69% with routine care; P<.01).⁶⁶ A third CDSS improved scores of barotrauma (P<.001) and organ dysfunction (P = .04) in mechanically ventilated patients with acute respiratory distress syndrome.⁷⁰ One participating center in this trial provided data demonstrating lower tidal volumes (P≤.03) and a reduction in exposure to high plateau pressures in the group receiving CDSS-guided mechanical ventilation (P<.001).¹¹⁴ A fourth CDSS reduced patient-reported asthma exacerbations (8% vs 17%; odds ratio, [OR], 0.43; 95% CI, 0.21-0.85), emergency nebulizer use (1% vs 5%; OR, 0.13; 95% CI, 0.01-0.91), and the need for additional consultations for asthma management (22% vs 34%; OR, 0.59; 95% CI, 0.37-0.95) over 6 months.⁷³ A fifth CDSS reduced hospital length of stay (P = .02) for patients with a variety of general medical diagnoses.⁸³

In post hoc secondary or subgroup analyses, some trials described statistically significant improvements in thrombolytic prescribing with the CDSS,⁶¹ as well as patient outcomes of disease-specific emergency department visits,⁶⁵ hospital length of stay,^{45 ,54 ,116 - 117} body weight,^{54 ,116 - 117} diastolic blood pressure,^{59 ,115 ,118} serum lipids,^{51 ,58} and a reduced estimated risk of future cardiovascular events.⁵⁸

There were 29 trials of drug dosing and prescribing (Table 7 and Table 8). Single-drug dosing improved practitioner performance in 15 (62%) of 24 studies, and 2 of the 18 systems assessing patient outcomes reported an improvement (Table 7 and Table 8). Another 5 systems used computer order entry for multidrug prescribing (Table 8). Four of these systems improved practitioner performance, but none improved patient outcomes.

The 24 single-drug dosing systems ranged from a simple calculator for parenteral nutrition to more complex algorithms that considered the pharmacokinetics of warfarin, aminoglycosides, or theophylline. Most studies evaluated the serum drug level in medications with a high risk of toxicity. In a study of heparin dosing for patients receiving thrombolysis for myocardial infarction, the proportion of individuals with an adverse thrombotic or cardiac event was significantly lowered with the CDSS (0/25 with the CDSS vs 6/26 in usual care; P = .02).⁹⁷ One warfarin-dosing CDSS reduced hospital length of stay from 20 to 13 days (P = .01).⁸⁷ Two systems reduced hospital length of stay in patients receiving theophylline (from 8.7 to 6.3 days; P = .03)⁹⁸ and aminoglycosides (20.3 to 16.0 days; P = .03),¹⁰⁴ although the majority of patient outcomes measured were not improved in these trials.

Given sparse data for patient outcomes, we only assessed study-level predictors of improved practitioner performance. Studies in which users were automatically prompted to use the system described better performance compared with studies in which users had to actively initiate the system (success in 44/60 studies [73%] vs 17/36 studies [47%]; P = .02; unadjusted OR, 2.8; 95% CI, 1.2-6.6; OR adjusted for methodological quality, 3.0; 95% CI, 1.2-7.1). Similarly, studies in which the authors also created the CDSS reported better performance compared with those in which the trialists were independent of the CDSS development process (success in 51/69 studies [74%] vs 5/18 studies [28%]; P = .001; unadjusted OR, 6.7; 95% CI, 1.7-25.3; OR adjusted for methodological quality, 6.6; 95% CI, 1.7-26.7). No other predefined study-level covariate was associated with CDSS success. In a post hoc analysis of the 85 studies that measured practitioner performance and enrolled physicians, we did not find an association (P = .40) between performance and physician experience (trainee vs attending physician).

We identified 100 randomized and nonrandomized trials testing a wide variety of CDSSs, with the number of trials and their methodological quality increasing over time. Of the 97 controlled trials assessing practitioner performance, the majority (64%) improved diagnosis, preventive care, disease management, drug dosing, or drug prescribing. However, the effects of these systems on patient health remain understudied—and inconsistent when studied. Fifty-two trials assessed patient outcomes, often in a limited capacity without adequate statistical power to detect clinically important differences. Only 7 trials reported improved patient outcomes with the CDSS, and no study reported benefits for major outcomes such as mortality. Surrogate patient outcomes such as blood pressure and glycated hemoglobin were not meaningfully improved in most studies.

Recent literature has called for a better understanding of factors that predict CDSS success.¹¹⁹ Barriers to implementation include failure of practitioners to use the CDSS, poor usability or integration into practitioner workflow, or practitioner nonacceptance of computer recommendations.¹²⁰ In our review, studies in which users were automatically prompted to use the system described better performance compared with studies in which users were required to actively initiate the system. A similar finding was also reported in a meta-regression of 11 studies of computer order entry.¹²¹ Compared with manual initiation, automatic prompting may improve integration into practitioner workflow as well as provide better opportunities to correct inadvertent deficiencies in care. In this review, we also identified better performance in studies in which the trial authors also developed the CDSS software. Potential explanations of this finding include the motivational effect of a developer’s enthusiasm, creation of more usable and integrated software, better access to technical support and training, improved on-site promotion and tailoring, biases in assessing outcomes, and selective publication of successful trials. Most of the CDSSs in this review were “home grown,” and the importance of local champions to facilitate implementation cannot be underestimated.

We identified relevant controlled trials through a comprehensive search of the literature. We extended our previous review from 1998 in a number of important ways.⁵ Using better-defined inclusion criteria, we reconsidered all prior articles and identified 37 new articles. To identify CDSS and study characteristics that predicted positive effects, we abstracted relevant data from all articles in duplicate, confirmed our abstractions with a majority of primary authors, and conducted a multivariable analysis of study-level covariates.

However, limitations of this review should be appreciated. We included only English-language studies. The CDSSs were grouped into categories based on clinical applications rather than on other aspects of CDSS design.¹²² Although trial methods are improving with time, this summary is limited by the methods used in the primary studies. We were unable to use meta-analysis to pool effect sizes, given substantial differences among primary studies in the types of CDSSs and outcomes evaluated. In addition, we defined improvement as a positive effect on at least 50% of outcomes measured. This approach, along with the strict inclusion criteria of this review, may have underestimated the influence of some system and study methodological factors on CDSS success. The wide confidence intervals for the statistically significant determinants of CDSS success imply substantial imprecision in the strength of these associations, which may be noncausal. Furthermore, it is possible that CDSSs for disease management promoted the implementation of ineffective therapies, or that CDSSs of drug dosing used incorrect pharmacokinetic models. Although this appears to be an unlikely explanation for the lack of effect on patient outcomes, we did not evaluate the appropriateness of CDSS algorithms or recommendations. Finally, we summarized controlled trials of CDSSs and did not consider less rigorous but more common designs, such as before-after studies.

The decision to adopt a CDSS for local patient care is complex and is influenced by many considerations. Those responsible for CDSS implementation are typically administrators, information technology managers, and clinicians, all of whom are increasingly pushed by technology and guided by government regulations.¹²³ Important issues include CDSS user acceptance, workflow integration, compatibility with legacy applications, system maturity, and upgrade availability. Some are concerned about increased practitioner dependence on CDSSs, with eroded capacity for independent decision making.³¹ Finally, cheaper, noncomputerized alternatives may be equally or more effective in improving care and reducing medical errors.^{124 - 127}

One of the primary considerations in adopting a CDSS is its clinical effectiveness: To what extent should it be proven beneficial before mass deployment? Clearly, some testing is required, as a CDSS can have unanticipated effects when used in patient care.⁸⁵ Some highlight the need for multicenter cluster-randomized controlled trials demonstrating improvements in important patient outcomes.¹² Using such a standard, this review suggests that the majority of available systems are not yet ready for mainstream use. Most trials were unable to enroll enough clusters or patients for adequate statistical power to detect improvements in patient outcomes. Unfortunately, this situation is unlikely to change soon, given the substantial time and resources needed to conduct such trials, particularly in the area of preventive health. Furthermore, CDSSs are limited by the cumulative knowledge used to program their recommendations. It would be unrealistic to require repeat CDSS testing every time advances in the knowledge base become available. Thus, for initial consideration, it may be reasonable to require proof of CDSS effectiveness only on practitioner performance, particularly if such outcomes represent current accepted standards in care. In our review, many systems met this requirement. However, this does not preclude the need for subsequent trials or in-practice assessment to confirm system performance in improving patient health. Institutions need to measure effects on local outcomes and be prepared to iteratively modify their system in response to practice-based knowledge.²

While some perceive that CDSSs improve efficiency and reduce costs, the current supporting evidence is limited. Although some studies have assessed the costs when outcomes were improved,^{40 ,45 ,79 - 81 ,84 ,128} the cost-effectiveness of these systems remains unknown. Many studies suggested the CDSS was inefficient, requiring more time and effort from the user compared with paper-based methods.^{14 - 15 ,38 ,64 ,81 ,95 ,112} Finally, most CDSSs used research funding to facilitate implementation. As highlighted in this review, up to 21% of trials used staff paid by research funds for data entry or CDSS recommendation delivery. When investing in a commercially available system, funding for support personnel is an additional cost to be considered.

There is currently widespread enthusiasm for introducing electronic medical records, computerized physician order entry systems, and CDSSs into hospitals and outpatient settings. In other commercial, industrial, and scientific spheres of activity, computers have become ubiquitous and have improved safety, productivity, and timeliness. Given this progress, computerization of the health care environment should offer tremendous benefits. However, uptake has been slow, and multiple challenges have arisen at every phase of software development, testing, and implementation. The progress of CDSSs has mirrored these trends. Systems are proliferating, their technical performance and usability are improving, and the number and quality of evaluations is increasing. These evaluations have shown that many CDSSs improve practitioner performance. However, further research is needed to elucidate the effects of such systems on patient health.