0
We're unable to sign you in at this time. Please try again in a few minutes.
Retry
We were able to sign you in, but your subscription(s) could not be found. Please try again in a few minutes.
Retry
There may be a problem with your account. Please contact the AMA Service Center to resolve this issue.
Contact the AMA Service Center:
Telephone: 1 (800) 262-2350 or 1 (312) 670-7827  *   Email: subscriptions@jamanetwork.com
Error Message ......
Review |

Effects of Computerized Clinical Decision Support Systems on Practitioner Performance and Patient Outcomes:  A Systematic Review FREE

Amit X. Garg, MD; Neill K. J. Adhikari, MD; Heather McDonald, MSc; M. Patricia Rosas-Arellano, MD, PhD; P. J. Devereaux, MD; Joseph Beyene, PhD; Justina Sam, BHSc; R. Brian Haynes, MD, PhD
[+] Author Affiliations

Author Affiliations: Division of Nephrology (Drs Garg and Rosas-Arellano) and Department of Epidemiology and Biostatistics (Dr Garg), University of Western Ontario, London; Departments of Clinical Epidemiology and Biostatistics (Drs Garg, Adhikari, Devereaux, and Haynes and Ms McDonald) and Medicine (Drs Devereaux and Haynes), McMaster University, Hamilton, Ontario; Department of Critical Care Medicine, Sunnybrook and Women’s College Health Sciences Centre and Interdepartmental Division of Critical Care (Dr Adhikari), Population Health Sciences, Hospital for Sick Children (Dr Beyene), and Faculty of Medicine (Ms Sam), University of Toronto, Toronto, Ontario.

More Author Information
JAMA. 2005;293(10):1223-1238. doi:10.1001/jama.293.10.1223.
Text Size: A A A
Published online

Context Developers of health care software have attributed improvements in patient care to these applications. As with any health care intervention, such claims require confirmation in clinical trials.

Objectives To review controlled trials assessing the effects of computerized clinical decision support systems (CDSSs) and to identify study characteristics predicting benefit.

Data Sources We updated our earlier reviews by searching the MEDLINE, EMBASE, Cochrane Library, Inspec, and ISI databases and consulting reference lists through September 2004. Authors of 64 primary studies confirmed data or provided additional information.

Study Selection We included randomized and nonrandomized controlled trials that evaluated the effect of a CDSS compared with care provided without a CDSS on practitioner performance or patient outcomes.

Data Extraction Teams of 2 reviewers independently abstracted data on methods, setting, CDSS and patient characteristics, and outcomes.

Data Synthesis One hundred studies met our inclusion criteria. The number and methodologic quality of studies improved over time. The CDSS improved practitioner performance in 62 (64%) of the 97 studies assessing this outcome, including 4 (40%) of 10 diagnostic systems, 16 (76%) of 21 reminder systems, 23 (62%) of 37 disease management systems, and 19 (66%) of 29 drug-dosing or prescribing systems. Fifty-two trials assessed 1 or more patient outcomes, of which 7 trials (13%) reported improvements. Improved practitioner performance was associated with CDSSs that automatically prompted users compared with requiring users to activate the system (success in 73% of trials vs 47%; P = .02) and studies in which the authors also developed the CDSS software compared with studies in which the authors were not the developers (74% success vs 28%; respectively, P = .001).

Conclusions Many CDSSs improve practitioner performance. To date, the effects on patient outcomes remain understudied and, when studied, inconsistent.

Computerized clinical decision support systems (CDSSs) are information systems designed to improve clinical decision making. Characteristics of individual patients are matched to a computerized knowledge base, and software algorithms generate patient-specific recommendations. Practitioners, health care staff, or patients can manually enter patient characteristics into the computer system; alternatively, electronic medical records can be queried for retrieval of patient characteristics. Computer-generated recommendations are delivered to the clinician through the electronic medical record, by pager, or through printouts placed in a patient’s paper chart. Such systems have been developed for a myriad of clinical issues, including diagnosis of chest pain, treatment of infertility, and timely administration of immunizations. These systems provide several modes of decision support, including alerts of critical values, reminders of overdue preventive health tasks, advice for drug prescribing, critiques of existing health care orders, and suggestions for various active care issues.

As with any health care innovation, CDSSs should be rigorously evaluated before widespread dissemination into clinical practice. Various stages in this assessment process have been previously described. Iterative qualitative and quantitative assessment begin early in the software development cycle.1,2 When preliminary testing suggests that a CDSS improves clinical care or patient outcomes, confirmatory controlled trials are warranted. We previously reviewed controlled trials of computer-aided quality assurance3 and CDSSs published up to 19924 and 1998.5 This field is rapidly evolving because of technological advances, increasing access to computer systems in clinical practice, and growing concern about the process and quality of medical care. We therefore updated previous reviews to provide a cumulative summary of controlled trials evaluating the effectiveness of CDSSs on practitioner performance and patient outcomes.

Research Questions

The primary questions of this review were (1) Do CDSSs improve practitioner performance or patient outcomes? and (2) Which CDSS and study-level factors are associated with effective CDSSs? A priori, we hypothesized that studies reporting better outcomes would assess CDSSs that automatically prompted users (vs requiring the user to actively initiate the system), were built into an electronic medical record or computer order entry system (vs a stand-alone system), provided reminders (vs information on disease management, drug dosing, or diagnosis), were tested using less rigorous study methods, were studied by their software developers (vs by evaluators not involved in the CDSS design), described pilot testing, and described user training.

Studies Eligible for Review

We included English-language randomized and nonrandomized trials with a contemporaneous control group that compared patient care with a CDSS to routine care without a CDSS and evaluated clinical performance (ie, a measure of process of care) or a patient outcome. We stipulated that the CDSS had to provide patient-specific advice that was reviewed by a health care practitioner before any clinical action. Studies were excluded if the system (1) was used solely by medical students, (2) only provided summaries of patient information, (3) provided feedback on groups of patients without individual assessment, (4) only provided computer-aided instruction, or (5) was used for image analysis. Studies assessing CDSS diagnostic performance against a defined gold standard were not included in this review unless clinical use of the diagnostic CDSS was also compared with routine care. Based on these criteria, we reevaluated all studies from our previous reviews for inclusion.

Finding Relevant Studies

We have previously described our methods for finding relevant studies until March 1998.5 For this update, we examined citations from MEDLINE, EMBASE, Evidence-Based Reviews databases (Cochrane Database of Systematic Reviews, ACP Journal Club, Database of Abstracts of Reviews of Effects, and Cochrane Central Register of Controlled Trials), and Inspec bibliographic databases from 1998 through September 2004. All citations were downloaded into Reference Manager, version 10.0 (Thomson ISI ResearchSoft, Philadelphia, Pa). An experienced librarian developed the search strategies using sensitive terms for identifying clinical studies of CDSSs. We pilot-tested search strategies and modified them to ensure that they identified known eligible articles. The final strategies used the terms computer-assisted decision making, computer-assisted diagnosis, computer-assisted therapy, decision support systems, reminder systems, hospital information systems, randomized controlled trial, and cohort studies (complete strategies available from the authors). Pairs of reviewers independently evaluated the eligibility of all studies identified in our search. Disagreements were resolved by a third reviewer or by consensus. Full-text articles were retrieved if any reviewer considered a citation potentially relevant. Supplementary methods of finding studies included a review of article reference lists, articles citing included studies as listed in the Science Citation Index, PubMed related articles feature, informatics conference proceedings, information provided by primary study authors, and other recent reviews.611 Where data from a trial were distributed in more than 1 publication, we cited the principal publication.

Data Abstraction

Pairs of reviewers independently abstracted the following data from all studies meeting eligibility criteria: study setting, study methods, CDSS characteristics, patient characteristics, and outcomes. Disagreements were resolved by a third reviewer or by consensus. We attempted to contact primary authors of all included studies to confirm data and provide missing data.

All studies were scored for methodological quality on a 10-point scale consisting of 5 potential sources of bias, which we have described elsewhere.5 In brief, we considered the method of allocation to study groups (random, 2, vs quasi-random, 1, vs selected concurrent controls, 0), the unit of the allocation (a cluster such as a practice, 2, vs physician, 1, vs patient, 0), the presence of baseline differences between the groups that were potentially linked to study outcomes (of particular importance for observational studies; no baseline differences present or appropriate statistical adjustments made for differences, 2, vs baseline differences present and no statistical adjustments made, 1, vs baseline characteristics not reported, 0), the objectivity of the outcome (objective outcomes or subjective outcomes with blinded assessment, 2, vs subjective outcomes with no blinding but clearly defined assessment criteria, 1, vs subjective outcomes with no blinding and poorly defined, 0), and the completeness of follow-up for the appropriate unit of analysis (>90%, 2, vs 80 to 90%, 1, vs <80% or not described, 0). The unit of allocation was included because of the possibility of group contamination in trials in which interventions were applied to clinicians even though individual patients were allocated to the intervention and control groups.12 Contamination bias would lead to underestimating the effect of a CDSS.

The studies substantially differed in the type and number of outcomes assessed. In addition, the majority of studies did not define a single outcome for statistical testing. We aimed to efficiently summarize the benefits of CDSSs and to identify CDSS and study characteristics that predicted success. For a given study we abstracted all reported practitioner performance and patient health outcomes. Situations where the CDSS worsened outcomes were rare. Thus, for each study we defined the effects of CDSSs in terms of success, defined as an improvement in at least 50% of outcomes measured, each at a 2-sided significance level less than .05.

Statistical Analysis

Reviewer agreement on study eligibility was quantified using the Cohen κ.13 Study and CDSS characteristics predicting success were analyzed and interpreted with the study as the unit of analysis. Data were summarized using descriptive summary measures, including proportions for categorical variables and mean (standard deviation) for continuous variables. Univariable and multivariable logistic regression models, adjusted for study methodological quality, were used to investigate associations between the outcomes of interest and study-specific covariates defined in our a priori hypotheses. All analyses were carried out using the SAS statistical package, version 8.2 (SAS Institute Inc, Cary, NC). We interpreted P≤.05 as indicating statistical significance; all P values are 2-sided. When reporting results from individual studies, we cited the measures of association and P values reported in the studies.

Finding and Selecting Studies

From 3997 screened citations, we retrieved 226 full-text articles, and 100 trials met our criteria for review. The chance-corrected agreement between 2 independent reviewers for article inclusion was good (κ = 0.81; 95% confidence interval [CI], 0.73-0.88).

Description of Studies

The 100 trials examined more than 3826 practitioners or practices (median, 42; range, 2-300 [when reported]) caring for more than 92 895 patients (median, 488; range, 19-12 989 [when reported]) from 1973 to 2004.145758113 The number of eligible trials increased with time: 1 in 1970-1974, 4 in 1975-1979, 10 in 1980-1984, 13 in 1985-1989, 20 in 1990-1994, 26 in 1995-1999, and 26 in 2000–September 2004. Of these 100 trials, most were conducted in the United States (69%), followed by the United Kingdom (14%), Canada (5%), Australia (4%), Italy (2%), and Austria, France, Germany, Israel, Norway, and Switzerland (1% each). Sixty-nine percent of trials described funding from the public sector and 16% from the private sector. Developers of CDSS software were also study authors in 72% of trials. Ninety-seven trials described the effect of CDSS on at least 1 measure of health care practitioner performance. Fifty-two trials assessed at least 1 patient outcome. We successfully contacted authors of 91 trials, and authors of 64 trials provided additional information or confirmed the accuracy of abstracted data.1518,20,21,2433,3540,42,43,46,47,49,51,52,56,6064,67,68,71,7375,80,81,8398,101,106,113115

Methodological Quality Assessment

Trial methodological rigor increased with time—36% of trials before the year 2000 were cluster randomized, compared with 67% after this time (P = .01). Of all trials, 88% were randomized. Of the randomized trials, 49% were cluster randomized and 40% used a cluster as the unit of analysis or adjusted for clustering in the analysis. Twenty-four randomized trials and 1 cohort study reported a power calculation for a prespecifed difference between groups on a specific outcome. Fifteen of these trials (60%) calculated sample size based on a practitioner performance outcome, 9 (36%) based on a patient outcome, and 1 (4%) based on the cost of prescribed medications. Only 2 studies examined patient outcomes without measuring practitioner performance. Of the 88 randomized trials, 52% described an appropriate method of generating random numbers and 28% reported allocation concealment. On the 10-point methods scale, the mean score was 7 (SD, 1.7) and the range was 2 to 10.

Description of Users and CDSSs

The 100 trials had the following characteristics: 92% of trials enrolled physicians as primary users, 48% enrolled training health care practitioners (interns and residents) as users, 34% described pilot testing with users prior to implementation, 42% described user instructional training at the time of implementation, 76% took place in academic centers, and 33% were inpatient-based. In 47% of studies, the CDSS was part of an electronic medical record or computer order entry system. Most of these were early generation systems lacking the full functionality of current systems. In 15% of studies, the CDSS had a graphical user interface. Feedback from the CDSS occurred at the time of patient care in 88% of studies; in 60% the user was automatically prompted to use the system (vs the user actively initiating the system), and in 91% the CDSS suggested new orders (vs critiquing existing orders). Expert physician opinion or clinical practice guidelines usually formed the knowledge base for the CDSS.

The process of data entry into the CDSS was clear in 80% of trials, some of which used more than 1 method. Existing personnel most often entered data (attending or training physician, 38%; other health care staff [eg, nurses, clerks], 29%), although many trials used staff paid by research funds (21%) or automated data capture from an electronic medical record (30%). The method of delivering computer recommendations to the clinician was clear in 81% of trials. Most CDSSs directly provided the recommendation on a computer screen viewed by the practitioner (41% of all trials) or generated printed reports that were placed in medical charts by health care staff (29%) or by staff paid by research funds (16%). Only 13% of trials evaluated the impact of the CDSS on clinician workflow, with more than half of these CDSSs requiring more time and effort from the user compared with paper-based methods.

Systems for Diagnosis

There were 10 trials evaluating diagnostic systems (Table 1). All studies measured practitioner performance, and the CDSS was beneficial in 4 studies (40%). Two of the 4 successful CDSSs were diagnostic systems for cardiac ischemia in the emergency department, and these decreased the rate of unnecessary hospital or coronary care admissions by 15% (P<.05).18,20 The third increased mood disorder screening in a posttraumatic stress disorder clinic by 25% (P = .008).15 The fourth improved the time to diagnosis of acute bowel obstruction (1 hour when computer was used vs 16 hours when diagnosis was made with contrast radiography; P<.001).23 Of the 5 trials assessing patient outcomes, none reported an improvement.

Table Graphic Jump LocationTable 1. Trials of Computer-Assisted Diagnosis*
Reminder Systems for Prevention

There were 21 trials evaluating reminder systems for prevention (Table 2). All trials measured practitioner performance, and the CDSS was beneficial in 16 studies (76%). Performance outcomes were usually rates of screening, counseling, vaccination, testing, medication use, or the identification of at-risk behaviors. Successful use of CDSSs was typically demonstrated in ambulatory care, although 1 system was successful in hospitalized patients.44 The single trial measuring patient outcomes failed to demonstrate an improvement in the primary analysis.34 Post hoc subgroup analyses, however, demonstrated a significant reduction in winter hospitalization and emergency department visits in patients eligible for pneumococcal or influenza vaccination. One trial examined the effect of adding a cervical cancer screening reminder to an existing mammography reminder system.30 This trial suggested no interaction between the 2 reminders on screening efficacy.

Table Graphic Jump LocationTable 2. Trials of Computer-Assisted Reminders for Cancer Screening, Vaccination, and Other Types of Preventive Care
Systems for Disease Management

There were 40 studies of CDSSs for active health conditions. These CDSSs improved practitioner performance in 23 (62%) of 37 studies evaluating this outcome. Of the 27 trials measuring patient outcomes, 5 (18%) demonstrated improvements.

For diabetes care, practitioner performance was usually judged by rates of retinal, foot, urine protein, blood pressure, and cholesterol examinations, with 5 (71%) of 7 trials reporting improvements (Table 3). Similarly, in studies of cardiovascular prevention, performance was judged by blood pressure and cholesterol assessment, identification of smoking, and use of cardioprotective medications, with 5 (38%) of 13 trials reporting improvements (Table 4). One CDSS provided electrocardiogram recommendations to improve thrombolytic prescribing in emergency departments.61 Other CDSSs varied in purpose, providing recommendations for urinary incontinence, human immunodeficiency virus infection management, functional assessment, and acute respiratory distress syndrome, with 6 of 9 reporting improvements (Table 5). Clinical decision support system corollary orders were used to monitor the effects of other prescribed treatments, such as the need for renal biochemistry measurements in patients receiving amphotericin B,79 with all 4 trials reporting improvements (Table 6). Trials testing CDSS performance to reduce unnecessary health care utilization measured the frequency of redundant testing and unnecessary hospital admissions and hospital length of stay, with 3 of 4 trials reporting improvements (Table 6).

Table Graphic Jump LocationTable 3. Trials of Computer-Assisted Diabetes Management*
Table Graphic Jump LocationTable 4. Trials of Computer-Assisted Cardiovascular Disease Management and Prevention*
Table Graphic Jump LocationTable 5. Trials of Computer-Assisted Management for Other Active Health Conditions*
Table Graphic Jump LocationTable 6. Trials of Computer Use to Monitor the Effects of Other Prescribed Treatments (Corollary Orders) or to Reduce Unnecessary Health Care Utilization*

Five CDSSs (18%) examining patient outcomes described improvements. One CDSS improved blood pressure control (70% of patients had controlled blood pressure with CDSS use vs 52% with routine care; P<.05).54 A second CDSS reduced urinary incontinence in nursing home residents over a 10-week period (23% incontinent with CDSS vs 69% with routine care; P<.01).66 A third CDSS improved scores of barotrauma (P<.001) and organ dysfunction (P = .04) in mechanically ventilated patients with acute respiratory distress syndrome.70 One participating center in this trial provided data demonstrating lower tidal volumes (P≤.03) and a reduction in exposure to high plateau pressures in the group receiving CDSS-guided mechanical ventilation (P<.001).114 A fourth CDSS reduced patient-reported asthma exacerbations (8% vs 17%; odds ratio, [OR], 0.43; 95% CI, 0.21-0.85), emergency nebulizer use (1% vs 5%; OR, 0.13; 95% CI, 0.01-0.91), and the need for additional consultations for asthma management (22% vs 34%; OR, 0.59; 95% CI, 0.37-0.95) over 6 months.73 A fifth CDSS reduced hospital length of stay (P = .02) for patients with a variety of general medical diagnoses.83

In post hoc secondary or subgroup analyses, some trials described statistically significant improvements in thrombolytic prescribing with the CDSS,61 as well as patient outcomes of disease-specific emergency department visits,65 hospital length of stay,45,54,116,117 body weight,54,116,117 diastolic blood pressure,59,115,118 serum lipids,51,58 and a reduced estimated risk of future cardiovascular events.58

Systems for Drug Dosing and Drug Prescribing

There were 29 trials of drug dosing and prescribing (Table 7 and Table 8). Single-drug dosing improved practitioner performance in 15 (62%) of 24 studies, and 2 of the 18 systems assessing patient outcomes reported an improvement (Table 7 and Table 8). Another 5 systems used computer order entry for multidrug prescribing (Table 8). Four of these systems improved practitioner performance, but none improved patient outcomes.

Table Graphic Jump LocationTable 7. Trials of Computer-Assisted Anticoagulant Dosing*
Table Graphic Jump LocationTable 8. Trials of Computer-Assisted Drug Dosing and Prescribing*

The 24 single-drug dosing systems ranged from a simple calculator for parenteral nutrition to more complex algorithms that considered the pharmacokinetics of warfarin, aminoglycosides, or theophylline. Most studies evaluated the serum drug level in medications with a high risk of toxicity. In a study of heparin dosing for patients receiving thrombolysis for myocardial infarction, the proportion of individuals with an adverse thrombotic or cardiac event was significantly lowered with the CDSS (0/25 with the CDSS vs 6/26 in usual care; P = .02).97 One warfarin-dosing CDSS reduced hospital length of stay from 20 to 13 days (P = .01).87 Two systems reduced hospital length of stay in patients receiving theophylline (from 8.7 to 6.3 days; P = .03)98 and aminoglycosides (20.3 to 16.0 days; P = .03),104 although the majority of patient outcomes measured were not improved in these trials.

Study Factors Associated With CDSS Success

Given sparse data for patient outcomes, we only assessed study-level predictors of improved practitioner performance. Studies in which users were automatically prompted to use the system described better performance compared with studies in which users had to actively initiate the system (success in 44/60 studies [73%] vs 17/36 studies [47%]; P = .02; unadjusted OR, 2.8; 95% CI, 1.2-6.6; OR adjusted for methodological quality, 3.0; 95% CI, 1.2-7.1). Similarly, studies in which the authors also created the CDSS reported better performance compared with those in which the trialists were independent of the CDSS development process (success in 51/69 studies [74%] vs 5/18 studies [28%]; P = .001; unadjusted OR, 6.7; 95% CI, 1.7-25.3; OR adjusted for methodological quality, 6.6; 95% CI, 1.7-26.7). No other predefined study-level covariate was associated with CDSS success. In a post hoc analysis of the 85 studies that measured practitioner performance and enrolled physicians, we did not find an association (P = .40) between performance and physician experience (trainee vs attending physician).

We identified 100 randomized and nonrandomized trials testing a wide variety of CDSSs, with the number of trials and their methodological quality increasing over time. Of the 97 controlled trials assessing practitioner performance, the majority (64%) improved diagnosis, preventive care, disease management, drug dosing, or drug prescribing. However, the effects of these systems on patient health remain understudied—and inconsistent when studied. Fifty-two trials assessed patient outcomes, often in a limited capacity without adequate statistical power to detect clinically important differences. Only 7 trials reported improved patient outcomes with the CDSS, and no study reported benefits for major outcomes such as mortality. Surrogate patient outcomes such as blood pressure and glycated hemoglobin were not meaningfully improved in most studies.

Determinants of CDSS Success

Recent literature has called for a better understanding of factors that predict CDSS success.119 Barriers to implementation include failure of practitioners to use the CDSS, poor usability or integration into practitioner workflow, or practitioner nonacceptance of computer recommendations.120 In our review, studies in which users were automatically prompted to use the system described better performance compared with studies in which users were required to actively initiate the system. A similar finding was also reported in a meta-regression of 11 studies of computer order entry.121 Compared with manual initiation, automatic prompting may improve integration into practitioner workflow as well as provide better opportunities to correct inadvertent deficiencies in care. In this review, we also identified better performance in studies in which the trial authors also developed the CDSS software. Potential explanations of this finding include the motivational effect of a developer’s enthusiasm, creation of more usable and integrated software, better access to technical support and training, improved on-site promotion and tailoring, biases in assessing outcomes, and selective publication of successful trials. Most of the CDSSs in this review were “home grown,” and the importance of local champions to facilitate implementation cannot be underestimated.

Strengths and Weaknesses of This Review

We identified relevant controlled trials through a comprehensive search of the literature. We extended our previous review from 1998 in a number of important ways.5 Using better-defined inclusion criteria, we reconsidered all prior articles and identified 37 new articles. To identify CDSS and study characteristics that predicted positive effects, we abstracted relevant data from all articles in duplicate, confirmed our abstractions with a majority of primary authors, and conducted a multivariable analysis of study-level covariates.

However, limitations of this review should be appreciated. We included only English-language studies. The CDSSs were grouped into categories based on clinical applications rather than on other aspects of CDSS design.122 Although trial methods are improving with time, this summary is limited by the methods used in the primary studies. We were unable to use meta-analysis to pool effect sizes, given substantial differences among primary studies in the types of CDSSs and outcomes evaluated. In addition, we defined improvement as a positive effect on at least 50% of outcomes measured. This approach, along with the strict inclusion criteria of this review, may have underestimated the influence of some system and study methodological factors on CDSS success. The wide confidence intervals for the statistically significant determinants of CDSS success imply substantial imprecision in the strength of these associations, which may be noncausal. Furthermore, it is possible that CDSSs for disease management promoted the implementation of ineffective therapies, or that CDSSs of drug dosing used incorrect pharmacokinetic models. Although this appears to be an unlikely explanation for the lack of effect on patient outcomes, we did not evaluate the appropriateness of CDSS algorithms or recommendations. Finally, we summarized controlled trials of CDSSs and did not consider less rigorous but more common designs, such as before-after studies.

When to Adopt a CDSS for Practice

The decision to adopt a CDSS for local patient care is complex and is influenced by many considerations. Those responsible for CDSS implementation are typically administrators, information technology managers, and clinicians, all of whom are increasingly pushed by technology and guided by government regulations.123 Important issues include CDSS user acceptance, workflow integration, compatibility with legacy applications, system maturity, and upgrade availability. Some are concerned about increased practitioner dependence on CDSSs, with eroded capacity for independent decision making.31 Finally, cheaper, noncomputerized alternatives may be equally or more effective in improving care and reducing medical errors.124127

One of the primary considerations in adopting a CDSS is its clinical effectiveness: To what extent should it be proven beneficial before mass deployment? Clearly, some testing is required, as a CDSS can have unanticipated effects when used in patient care.85 Some highlight the need for multicenter cluster-randomized controlled trials demonstrating improvements in important patient outcomes.12 Using such a standard, this review suggests that the majority of available systems are not yet ready for mainstream use. Most trials were unable to enroll enough clusters or patients for adequate statistical power to detect improvements in patient outcomes. Unfortunately, this situation is unlikely to change soon, given the substantial time and resources needed to conduct such trials, particularly in the area of preventive health. Furthermore, CDSSs are limited by the cumulative knowledge used to program their recommendations. It would be unrealistic to require repeat CDSS testing every time advances in the knowledge base become available. Thus, for initial consideration, it may be reasonable to require proof of CDSS effectiveness only on practitioner performance, particularly if such outcomes represent current accepted standards in care. In our review, many systems met this requirement. However, this does not preclude the need for subsequent trials or in-practice assessment to confirm system performance in improving patient health. Institutions need to measure effects on local outcomes and be prepared to iteratively modify their system in response to practice-based knowledge.2

While some perceive that CDSSs improve efficiency and reduce costs, the current supporting evidence is limited. Although some studies have assessed the costs when outcomes were improved,40,45,7981,84,128 the cost-effectiveness of these systems remains unknown. Many studies suggested the CDSS was inefficient, requiring more time and effort from the user compared with paper-based methods.14,15,38,64,81,95,112 Finally, most CDSSs used research funding to facilitate implementation. As highlighted in this review, up to 21% of trials used staff paid by research funds for data entry or CDSS recommendation delivery. When investing in a commercially available system, funding for support personnel is an additional cost to be considered.

There is currently widespread enthusiasm for introducing electronic medical records, computerized physician order entry systems, and CDSSs into hospitals and outpatient settings. In other commercial, industrial, and scientific spheres of activity, computers have become ubiquitous and have improved safety, productivity, and timeliness. Given this progress, computerization of the health care environment should offer tremendous benefits. However, uptake has been slow, and multiple challenges have arisen at every phase of software development, testing, and implementation. The progress of CDSSs has mirrored these trends. Systems are proliferating, their technical performance and usability are improving, and the number and quality of evaluations is increasing. These evaluations have shown that many CDSSs improve practitioner performance. However, further research is needed to elucidate the effects of such systems on patient health.

Corresponding Author: R. Brian Haynes, MD, PhD, Clinical Epidemiology and Biostatistics, Faculty of Health Sciences, McMaster University, 1200 Main St W, Room 2C10B, McMaster University, Hamilton, Ontario, Canada L8N 3Z5 (bhaynes@mcmaster.ca).

Author Contributions: Drs Garg, Adhikari, and Rosas-Arellano had full access to all of the data in the study and take responsibility for the integrity of the data and the accuracy of the data analysis.

Study concept and design: Garg, Adhikari, Haynes.

Acquisition of data: Garg, Adhikari, McDonald, Rosas-Arellano, Sam, Haynes.

Analysis and interpretation of data: Garg, Adhikari, McDonald, Rosas-Arellano, Devereaux, Beyene, Sam, Haynes.

Drafting of the manuscript: Garg, Adhikari, Haynes.

Critical revision of the manuscript for important intellectual content: Garg, Adhikari, McDonald, Rosas-Arellano, Devereaux, Beyene, Sam, Haynes.

Statistical analysis: Garg, Adhikari, Beyene.

Obtained funding: Garg, Haynes.

Administrative, technical, or material support: Garg, Rosas-Arellano, Haynes.

Study supervision: Garg, Haynes.

Financial Disclosures: None reported.

Funding/Support: Dr Garg was supported by a Canadian Institutes of Health Research (CIHR) Clinician Scientist Training Award. Dr Devereaux was supported by a CIHR Senior Research Fellowship.

Role of the Sponsors: No funding source or sponsor had any role in the design and conduct of the study; collection, management, analysis, or interpretation of the data; or preparation, review, or approval of the manuscript.

Acknowledgment: We acknowledge the work of Linda Sheridan, who provided administrative help, and Tom Flemming, the librarian who helped with the literature searches. We thank Dereck Hunt, MD, MSc, and William Clark, MD, for their help and support.

Wyatt J, Spiegelhalter D. Evaluating medical expert systems: what to test and how?  Med Inform (Lond). 1990;15:205-217
PubMed   |  Link to Article
Sim I, Gorman P, Greenes RA.  et al.  Clinical decision support systems for the practice of evidence-based medicine.  J Am Med Inform Assoc. 2001;8:527-534
PubMed   |  Link to Article
Haynes RB, Walker CJ. Computer-aided quality assurance.  Arch Intern Med. 1987;147:1297-1301
PubMed   |  Link to Article
Johnston ME, Langton KB, Haynes RB, Mathieu A. Effects of computer-based clinical decision support systems on clinician performance and patient outcome.  Ann Intern Med. 1994;120:135-142
PubMed   |  Link to Article
Hunt DL, Haynes RB, Hanna SE, Smith K. Effects of computer-based clinical decision support systems on physician performance and patient outcomes: a systematic review.  JAMA. 1998;280:1339-1346
PubMed   |  Link to Article
Kaushal R, Shojania KG, Bates DW. Effects of computerized physician order entry and clinical decision support systems on medication safety: a systematic review.  Arch Intern Med. 2003;163:1409-1416
PubMed   |  Link to Article
Payne TH. Computer decision support systems.  Chest. 2000;118:47S-52S
PubMed   |  Link to Article
Shiffman RN, Liaw Y, Brandt CA, Corb GJ. Computer-based guideline implementation systems: a systematic review of functionality and effectiveness.  J Am Med Inform Assoc. 1999;6:104-114
PubMed   |  Link to Article
Kaplan B. Evaluating informatics applications—clinical decision support systems literature review.  Int J Med Inform. 2001;64:15-37
PubMed   |  Link to Article
Gordon RB, Grimshaw JM, Eccles M, Rowe RE, Wyatt JC. On-screen computer reminders: effects on professional practice and health care outcomes.  Cochrane Database Syst Rev. 1998;2:CD001096
Mitchell E, Sullivan F. A descriptive feast but an evaluative famine: systematic review of published articles on primary care computing during 1980-97.  BMJ. 2001;322:279-282
PubMed   |  Link to Article
Chuang JH, Hripcsak G, Heitjan DF. Design and analysis of controlled trials in naturally clustered environments: implications for medical informatics.  J Am Med Inform Assoc. 2002;9:230-238
PubMed   |  Link to Article
Fleiss JL, Cohen J. The equivalence of weighted kappa and the intraclass correlation coefficient as measures of reliability.  Educ Psychol Meas. 1973;33:613-619
Link to Article
Lewis G, Sharp D, Bartholomew J, Pelosi AJ. Computerized assessment of common mental disorders in primary care: effect on clinical outcome.  Fam Pract. 1996;13:120-126
PubMed   |  Link to Article
Cannon DS, Allen SN. A comparison of the effects of computer and manual reminders on compliance with a mental health clinical practice guideline.  J Am Med Inform Assoc. 2000;7:196-203
PubMed   |  Link to Article
Schriger DL, Gibbons PS, Langone CA, Lee S, Altshuler LL. Enabling the diagnosis of occult psychiatric illness in the emergency department: a randomized, controlled trial of the computerized, self-administered PRIME-MD diagnostic system.  Ann Emerg Med. 2001;37:132-140
PubMed   |  Link to Article
Rollman BL, Hanusa BH, Lowe HJ.  et al.  A randomized trial using computerized decision support to improve treatment of major depression in primary care.  J Gen Intern Med. 2002;17:493-503
PubMed   |  Link to Article
Pozen MW, D’Agostino RB, Selker HP, Sytkowski PA, Hood WB. A predictive instrument to improve coronary-care-unit admission practices in acute ischemic heart disease: a prospective multicenter clinical trial.  N Engl J Med. 1984;310:1273-1278
PubMed   |  Link to Article
Wyatt JR. Lessons learnt from the field trial of ACORN, an expert system to advise on chest pain. In: Barber B, Cao D, Quin D, Wagner G, eds. Proceedings of the 6th World Conference on Medical Informatics; Singapore; 1989:111-115
Selker HP, Beshansky JR, Griffith JL.  et al.  Use of the acute cardiac ischemia time-insensitive predictive instrument (ACI-TIPI) to assist with triage of patients with chest pain or other symptoms suggestive of acute cardiac ischemia: a multicenter, controlled clinical trial.  Ann Intern Med. 1998;129:845-855
PubMed   |  Link to Article
Wexler JR, Swender PT, Tunnessen WWJ, Oski FA. Impact of a system of computer-assisted diagnosis.  AJDC. 1975;129:203-205
PubMed
Wellwood J, Johannessen S, Spiegelhalter DJ. How does computer-aided diagnosis improve the management of acute abdominal pain?  Ann R Coll Surg Engl. 1992;74:40-46
PubMed
Bogusevicius A, Maleckas A, Pundzius J, Skaudickas D. Prospective randomised trial of computer-aided diagnosis and contrast radiography in acute small bowel obstruction.  Eur J Surg. 2002;168:78-83
PubMed   |  Link to Article
Turner BJ, Day SC, Borenstein B. A controlled trial to improve delivery of preventive care: physician or patient reminders?  J Gen Intern Med. 1989;4:403-409
PubMed   |  Link to Article
McPhee SJ, Bird JA, Jenkins CN, Fordham D. Promoting cancer screening: a randomized, controlled trial of three interventions.  Arch Intern Med. 1989;149:1866-1872
PubMed   |  Link to Article
McPhee SJ, Bird JA, Fordham D, Rodnick JE, Osborn EH. Promoting cancer prevention activities by primary care physicians: results of a randomized, controlled trial.  JAMA. 1991;266:538-544
PubMed   |  Link to Article
Burack RC, Gimotty PA, George J.  et al.  Promoting screening mammography in inner-city settings: a randomized controlled trial of computerized reminders as a component of a program to facilitate mammography.  Med Care. 1994;32:609-624
PubMed   |  Link to Article
Burack RC, Gimotty PA, George J, Simon MS, Dews P, Moncrease A. The effect of patient and physician reminders on use of screening mammography in a health maintenance organization: results of a randomized controlled trial.  Cancer. 1996;78:1708-1721
PubMed   |  Link to Article
Burack RC, Gimotty PA. Promoting screening mammography in inner-city settings: the sustained effectiveness of computerized reminders in a randomized controlled trial.  Med Care. 1997;35:921-931
PubMed   |  Link to Article
Burack RC, Gimotty PA, Simon M.  et al.  The effect of adding Pap smear information to a mammography reminder system in an HMO: results of randomized controlled trial.  Prev Med. 2003;36:547-554
PubMed   |  Link to Article
Chambers CV, Balaban DJ, Carlson BL, Grasberger DM. The effect of microcomputer-generated reminders on influenza vaccination rates in a university-based family practice center.  J Am Board Fam Pract. 1991;4:19-26
PubMed
Flanagan JR, Doebbeling BN, Dawson J, Beekmann S. Randomized study of online vaccine reminders in adult primary care.  Proc AMIA Symp. 1999;755-759
PubMed
Tang PC, LaRosa MP, Newcomb C, Gorden SM. Measuring the effects of reminders for outpatient influenza immunizations at the point of clinical opportunity.  J Am Med Inform Assoc. 1999;6:115-121
PubMed   |  Link to Article
McDonald CJ, Hui SL, Smith DM.  et al.  Reminders to physicians from an introspective computer medical record: a two-year randomized trial.  Ann Intern Med. 1984;100:130-138
PubMed   |  Link to Article
Tierney WM, Hui SL, McDonald CJ. Delayed feedback of physician performance versus immediate reminders to perform preventive care: effects on physician compliance.  Med Care. 1986;24:659-666
PubMed   |  Link to Article
Ornstein SM, Garr DR, Jenkins RG, Rust PF, Arnon A. Computer-generated physician and patient reminders: tools to improve population adherence to selected preventive services.  J Fam Pract. 1991;32:82-90
PubMed
Rosser WW, McDowell I, Newell C. Use of reminders for preventive procedures in family medicine.  CMAJ. 1991;145:807-814
PubMed
Tape TG, Campbell JR. Computerized medical records and preventive health care: success depends on many factors.  Am J Med. 1993;94:619-625
PubMed   |  Link to Article
Turner RC, Peden JGJ, O’Brien K. Patient-carried card prompts vs computer-generated prompts to remind private practice physicians to perform health maintenance measures.  Arch Intern Med. 1994;154:1957-1960
PubMed   |  Link to Article
Frame PS, Zimmer JG, Werth PL.  et al.  Computer-based vs manual health maintenance tracking: a controlled trial.  Arch Fam Med. 1994;3:581-588
PubMed   |  Link to Article
Overhage JM, Tierney WM, McDonald CJ. Computer reminders to implement preventive care guidelines for hospitalized patients.  Arch Intern Med. 1996;156:1551-1556
PubMed   |  Link to Article
Bonevski B, Sanson-Fisher RW, Campbell E, Carruthers A, Reid AL, Ireland M. Randomized controlled trial of a computer strategy to increase general practitioner preventive care.  Prev Med. 1999;29:478-486
PubMed   |  Link to Article
Demakis JG, Beauchamp C, Cull WL.  et al.  Improving residents’ compliance with standards of ambulatory care: results from the VA Cooperative Study on Computerized Reminders.  JAMA. 2000;284:1411-1416
PubMed   |  Link to Article
Dexter PR, Perkins S, Overhage JM, Maharry K, Kohler RB, McDonald CJ. A computerized reminder system to increase the use of preventive care for hospitalized patients.  N Engl J Med. 2001;345:965-970
PubMed   |  Link to Article
Thomas JC, Moore A, Qualls PE. The effect on cost of medical care for patients treated with an automated clinical audit system.  J Med Syst. 1983;7:307-313
PubMed   |  Link to Article
Mazzuca SA, Vinicor F, Einterz RM, Norton JA, Kalasinski LA. Effects of the clinical environment on physicians’ response to postgraduate medical education.  Am Educ Res J. 1990;27:473-488
Link to Article
Nilasena DS, Lincoln MJ. A computer-generated reminder system improves physician compliance with diabetes preventive care guidelines.  Proc Annu Symp Comput Appl Med Care. 1995;640-645
PubMed
Lobach DF, Hammond WE. Computerized decision support based on a clinical practice guideline improves compliance with care standards.  Am J Med. 1997;102:89-98
PubMed   |  Link to Article
Montori VM, Dinneen SF, Gorman CA.  et al.  The impact of planned care and a diabetes electronic management system on community-based diabetes care.  Diabetes Care. 2002;25:1952-1957
PubMed   |  Link to Article
Filippi A, Sabatini A, Badioli L.  et al.  Effects of an automated electronic reminder in changing the antiplatelet drug-prescribing behavior among Italian general practitioners in diabetic patients: an intervention trial.  Diabetes Care. 2003;26:1497-1500
PubMed   |  Link to Article
Meigs JB, Cagliero E, Dubey A.  et al.  A controlled trial of Web-based diabetes disease management: the MGH Diabetes Primary Care Improvement Project.  Diabetes Care. 2003;26:750-757
PubMed   |  Link to Article
Coe FL, Norton E, Oparil S.  et al.  Treatment of hypertension by computer and physician: a prospective controlled study.  J Chronic Dis. 1977;30:81-92
PubMed   |  Link to Article
Rogers JL, Haring OM, Goetz JP. Changes in patient attitudes following the implementation of a medical information system.  QRB Qual Rev Bull. 1984;10:65-74
PubMed
Barnett GO, Winickoff RN, Morgan MM, Zielstorff RD. A computer-based monitoring system for follow-up of elevated blood pressure.  Med Care. 1983;21:400-409
PubMed   |  Link to Article
Brownbridge G, Evans A, Fitter M, Platts M. An interactive computerized protocol for the management of hypertension: effects on the general practitioner’s clinical behaviour.  J R Coll Gen Pract. 1986;36:198-202
PubMed
McAlister NH, Covvey HD, Tong C, Lee A, Wigle ED. Randomised controlled trial of computer assisted management of hypertension in primary care.  Br Med J (Clin Res Ed). 1986;293:670-674
PubMed   |  Link to Article
Rossi RA, Every NR. A computerized intervention to decrease the use of calcium channel blockers in hypertension.  J Gen Intern Med. 1997;12:672-678
PubMed   |  Link to Article
Lowensteyn I, Joseph L, Levinton C, Abrahamowicz M, Steinert Y, Grover S. Can computerized risk profiles help patients improve their coronary risk? the results of the Coronary Health Assessment Study (CHAS).  Prev Med. 1998;27:730-737
PubMed   |  Link to Article
Hetlevik I, Holmen J, Kruger O. Implementing clinical guidelines in the treatment of hypertension in general practice: evaluation of patient outcome related to implementation of a computer-based clinical decision support system.  Scand J Prim Health Care. 1999;17:35-40
PubMed   |  Link to Article
Montgomery AA, Fahey T, Peters TJ, MacIntosh C, Sharp DJ. Evaluation of computer based clinical decision support system and risk chart for management of hypertension in primary care: randomised controlled trial.  BMJ. 2000;320:686-690
PubMed   |  Link to Article
Selker HP, Beshansky JR, Griffith JL. Use of the electrocardiograph-based thrombolytic predictive instrument to assist thrombolytic and reperfusion therapy for acute myocardial infarction: a multicenter, randomized, controlled, clinical effectiveness trial.  Ann Intern Med. 2002;137:87-95
PubMed   |  Link to Article
Ansari M, Shlipak MG, Heidenreich PA.  et al.  Improving guideline adherence: a randomized trial evaluating strategies to increase beta-blocker use in heart failure.  Circulation. 2003;107:2799-2804
PubMed   |  Link to Article
Tierney WM, Overhage JM, Murray MD.  et al.  Effects of computerized guidelines for managing heart disease in primary care.  J Gen Intern Med. 2003;18:967-976
PubMed   |  Link to Article
Weir CJ, Lees KR, MacWalter RS.  et al.  Cluster-randomized, controlled trial of computer-based decision support for selecting long-term anti-thrombotic therapy after acute ischaemic stroke.  QJM. 2003;96:143-153
PubMed   |  Link to Article
Murray MD, Harris LE, Overhage JM.  et al.  Failure of computerized treatment suggestions to improve health outcomes of outpatients with uncomplicated hypertension: results of a randomized controlled trial.  Pharmacotherapy. 2004;24:324-337
PubMed   |  Link to Article
Petrucci K, Petrucci P, Canfield K.  et al.  Evaluation of UNIS: urological nursing information systems.  Proc Annu Symp Comput Appl Med Care. 1991;43-47
PubMed
Rubenstein LV, McCoy JM, Cope DW.  et al.  Improving patient quality of life with feedback to physicians about functional status.  J Gen Intern Med. 1995;10:607-614
PubMed   |  Link to Article
Safran C, Rind DM, Davis RB.  et al.  Guidelines for management of HIV infection with computer-based patient’s record.  Lancet. 1995;346:341-346
PubMed   |  Link to Article
Dexter PR, Wolinsky FD, Gramelspacher GP.  et al.  Effectiveness of computer-generated reminders for increasing discussions about advance directives and completion of advance directive forms: a randomized, controlled trial.  Ann Intern Med. 1998;128:102-110
PubMed   |  Link to Article
East TD, Heermann LK, Bradshaw RL.  et al.  Efficacy of computerized decision support for mechanical ventilation: results of a prospective multi-center randomized trial.  Proc AMIA Symp. 1999;251-255
PubMed
Kuperman GJ, Teich JM, Tanasijevic MJ.  et al.  Improving response to critical laboratory results with automation: results of a randomized controlled trial.  J Am Med Inform Assoc. 1999;6:512-522
PubMed   |  Link to Article
Christakis DA, Zimmerman FJ, Wright JA.  et al.  A randomized controlled trial of point-of-care evidence to improve the antibiotic prescribing practices for otitis media in children.  Pediatrics. 2001;107:E15
PubMed   |  Link to Article
McCowan C, Neville RG, Ricketts IW, Warner FC, Hoskins G, Thomas GE. Lessons from a randomized controlled trial designed to evaluate computer decision support software to improve the management of asthma.  Med Inform Internet Med. 2001;26:191-201
PubMed   |  Link to Article
Eccles M, McColl E, Steen N.  et al.  Effect of computerised evidence based guidelines on management of asthma and angina in adults in primary care: cluster randomised controlled trial.  BMJ. 2002;325:941
PubMed   |  Link to Article
Lesourd F, Avril C, Boujennah A, Parinaud J. A computerized decision support system for ovarian stimulation by gonadotropins.  Fertil Steril. 2002;77:456-460
PubMed   |  Link to Article
McDonald CJ. Use of a computer to detect and respond to clinical events: its effect on clinician behavior.  Ann Intern Med. 1976;84:162-167
PubMed   |  Link to Article
Young DW. Improving the consistency with which investigations are requested.  Med Inform (Lond). 1981;6:13-17
PubMed   |  Link to Article
Fihn SD, McDonell MB, Vermes D.  et al.  A computerized intervention to improve timing of outpatient follow-up: a multicenter randomized trial in patients treated with warfarin.  J Gen Intern Med. 1994;9:131-139
PubMed   |  Link to Article
Overhage JM, Tierney WM, Zhou XH, McDonald CJ. A randomized trial of “corollary orders” to prevent errors of omission.  J Am Med Inform Assoc. 1997;4:364-375
PubMed   |  Link to Article
Tierney WM, McDonald CJ, Hui SL, Martin DK. Computer predictions of abnormal test results: effects on outpatient testing.  JAMA. 1988;259:1194-1198
PubMed   |  Link to Article
Tierney WM, Miller ME, Overhage JM, McDonald CJ. Physician inpatient order writing on microcomputer workstations: effects on resource utilization.  JAMA. 1993;269:379-383
PubMed   |  Link to Article
Hales JW, Gardner RM, Jacobson JT. Factors impacting the success of computerized preadmission screening.  Proc Annu Symp Comput Appl Med Care. 1995;728-732
PubMed
Shea S, Sideli RV, DuMouchel W, Pulver G, Arons RR, Clayton PD. Computer-generated informational messages directed to physicians: effect on length of hospital stay.  J Am Med Inform Assoc. 1995;2:58-64
PubMed   |  Link to Article
Bates DW, Kuperman GJ, Rittenberg E.  et al.  A randomized trial of a computer-based intervention to reduce utilization of redundant laboratory tests.  Am J Med. 1999;106:144-150
PubMed   |  Link to Article
Abbrecht PH, O’Leary TJ, Behrendt DM. Evaluation of a computer-assisted method for individualized anticoagulation: retrospective and prospective studies with a pharmacodynamic model.  Clin Pharmacol Ther. 1982;32:129-136
PubMed   |  Link to Article
Carter BL, Taylor JW, Becker A. Evaluation of three dosage-prediction methods for initial in-hospital stabilization of warfarin therapy.  Clin Pharm. 1987;6:37-45
PubMed
White RH, Hong R, Venook AP.  et al.  Initiation of warfarin therapy: comparison of physician dosing with computer-assisted dosing.  J Gen Intern Med. 1987;2:141-148
PubMed   |  Link to Article
White RH, Mungall D. Outpatient management of warfarin therapy: comparison of computer-predicted dosage adjustment to skilled professional care.  Ther Drug Monit. 1991;13:46-50
PubMed   |  Link to Article
Poller L, Wright D, Rowlands M. Prospective comparative study of computer programs used for management of warfarin.  J Clin Pathol. 1993;46:299-303
PubMed   |  Link to Article
Fitzmaurice DA, Hobbs FD, Murray ET, Bradley CP, Holder R. Evaluation of computerized decision support for oral anticoagulation management based in primary care.  Br J Gen Pract. 1996;46:533-535
PubMed
Vadher B, Patterson DL, Leaning M. Evaluation of a decision support system for initiation and control of oral anticoagulation in a randomised trial.  BMJ. 1997;314:1252-1256
PubMed   |  Link to Article
Vadher BD, Patterson DL, Leaning M. Comparison of oral anticoagulant control by a nurse-practitioner using a computer decision-support system with that by clinicians.  Clin Lab Haematol. 1997;19:203-207
PubMed   |  Link to Article
Ageno W, Turpie AG. A randomized comparison of a computer-based dosing program with a manual system to monitor oral anticoagulant therapy.  Thromb Res. 1998;91:237-240
PubMed   |  Link to Article
Poller L, Shiach CR, MacCallum PK.  et al.  Multicentre randomised study of computerised anticoagulant dosage.  Lancet. 1998;352:1505-1509
PubMed   |  Link to Article
Fitzmaurice DA, Hobbs FD, Murray ET, Holder RL, Allan TF, Rose PE. Oral anticoagulation management in primary care with the use of computerized decision support and near-patient testing: a randomized, controlled trial.  Arch Intern Med. 2000;160:2343-2348
PubMed   |  Link to Article
Manotti C, Moia M, Palareti G, Pengo V, Ria L, Dettori AG. Effect of computer-aided management on the quality of treatment in anticoagulated patients: a prospective, randomized, multicenter trial of APROAT (Automated PRogram for Oral Anticoagulant Treatment).  Haematologica. 2001;86:1060-1070
PubMed
Mungall DR, Anbe D, Forrester PL.  et al.  A prospective randomized comparison of the accuracy of computer-assisted versus GUSTO nomogram–directed heparin therapy.  Clin Pharmacol Ther. 1994;55:591-596
PubMed   |  Link to Article
Hurley SF, Dziukas LJ, McNeil JJ, Brignell MJ. A randomized controlled clinical trial of pharmacokinetic theophylline dosing.  Am Rev Respir Dis. 1986;134:1219-1224
PubMed
Gonzalez ER, Vanderheyden BA, Ornato JP, Comstock TG. Computer-assisted optimization of aminophylline therapy in the emergency department.  Am J Emerg Med. 1989;7:395-401
PubMed   |  Link to Article
Verner D, Seligmann H, Platt S.  et al.  Computer assisted design of a theophylline dosing regimen in acute bronchospasm: serum concentrations and clinical outcome.  Eur J Clin Pharmacol. 1992;43:29-33
PubMed   |  Link to Article
Casner PR, Reilly R, Ho H. A randomized controlled trial of computerized pharmacokinetic theophylline dosing versus empiric physician dosing.  Clin Pharmacol Ther. 1993;53:684-690
PubMed   |  Link to Article
Begg EJ, Atkinson HC, Jeffery GM, Taylor NW. Individualised aminoglycoside dosage based on pharmacokinetic analysis is superior to dosage based on physician intuition at achieving target plasma drug concentrations.  Br J Clin Pharmacol. 1989;28:137-141
PubMed   |  Link to Article
Hickling K, Begg E, Moore ML. A prospective randomised trial comparing individualised pharmacokinetic dosage prediction for aminoglycosides with prediction based on estimated creatinine clearance in critically ill patients.  Intensive Care Med. 1989;15:233-237
PubMed   |  Link to Article
Burton ME, Ash CL, Hill DP Jr, Handy T, Shepherd MD, Vasko MR. A controlled trial of the cost benefit of computerized Bayesian aminoglycoside administration.  Clin Pharmacol Ther. 1991;49:685-694
PubMed   |  Link to Article
Peck CC, Sheiner LB, Martin CM.  et al.  Computer-assisted digoxin therapy.  N Engl J Med. 1973;289:441-446
PubMed   |  Link to Article
Rodman JH, Jelliffe RW, Kolb E.  et al.  Clinical studies with computer-assisted initial lidocaine therapy.  Arch Intern Med. 1984;144:703-709
PubMed   |  Link to Article
Ryff-de Leche A, Engler H, Nutzi E, Berger M, Berger W. Clinical application of two computerized diabetes management systems: comparison with the log-book method.  Diabetes Res. 1992;19:97-105
PubMed
Horn W, Popow C, Miksch S.  et al.  Development and evaluation of VIE-PNN, a knowledge-based system for calculating the parenteral nutrition of newborn infants.  Artif Intell Med. 2002;24:217-228
PubMed   |  Link to Article
McDonald CJ. Protocol-based computer reminders, the quality of care and the non-perfectability of man.  N Engl J Med. 1976;295:1351-1355
PubMed   |  Link to Article
McDonald CJ, Wilson GA, McCabe GP Jr. Physician response to computer reminders.  JAMA. 1980;244:1579-1581
PubMed   |  Link to Article
White KS, Lindsay A, Pryor TA, Brown WF, Walsh K. Application of a computerized medical decision-making process to the problem of digoxin intoxication.  J Am Coll Cardiol. 1984;4:571-576
PubMed   |  Link to Article
Rotman BL, Sullivan AN, McDonald TW.  et al.  A randomized controlled trial of a computer-based physician workstation in an outpatient setting: implementation barriers to outcome evaluation.  J Am Med Inform Assoc. 1996;3:340-348
PubMed   |  Link to Article
Tamblyn R, Huang A, Perreault R.  et al.  The medical office of the 21st century (MOXXI): effectiveness of computerized decision-making support in reducing inappropriate prescribing in primary care.  CMAJ. 2003;169:549-556
PubMed
McKinley BA, Moore FA, Sailors RM.  et al.  Computerized decision support for mechanical ventilation of trauma induced ARDS: results of a randomized clinical trial.  J Trauma. 2001;50:415-424
PubMed   |  Link to Article
Hetlevik I, Holmen J, Kruger O, Kristensen P, Iversen H, Furuseth K. Implementing clinical guidelines in the treatment of diabetes mellitus in general practice: evaluation of effort, process, and patient outcome related to implementation of a computer-based decision support system.  Int J Technol Assess Health Care. 2000;16:210-227
PubMed   |  Link to Article
Rogers JL, Haring OM, Wortman PM, Watson RA, Goetz JP. Medical information systems: assessing impact in the areas of hypertension, obesity and renal disease.  Med Care. 1982;20:63-74
PubMed   |  Link to Article
Rogers JL, Haring OM. The impact of a computerized medical record summary system on incidence and length of hospitalization.  Med Care. 1979;17:618-630
PubMed   |  Link to Article
Hetlevik I, Holmen J, Kruger O.  et al.  Implementing clinical guidelines in the treatment of hypertension in general practice.  Blood Press. 1998;7:270-276
PubMed   |  Link to Article
Holbrook A, Xu S, Banting J. What factors determine the success of clinical decision support systems?  AMIA Annu Symp Proc. 2003;862
PubMed
Reisman Y. Computer-based clinical decision aids: a review of methods and assessment of systems.  Med Inform (Lond). 1996;21:179-197
PubMed   |  Link to Article
Kawamoto K, Lobach DF. Clinical decision support provided within physician order entry systems: a systematic review of features effective for changing clinician behavior.  AMIA Annu Symp Proc. 2003;361-365
PubMed
Berlin A, Sorani M, Sim I. Characteristics of outpatient clinical decision support systems: a taxonomic description.  Medinfo. 2004;2004:578-581
PubMed
Fox J, Thomson R. Clinical decision support systems: a discussion of quality, safety and legal liability issues.  Proc AMIA Symp. 2002;265-269
PubMed
Ofman JJ, Badamgarav E, Henning JM.  et al.  Does disease management improve clinical and economic outcomes in patients with chronic diseases? a systematic review.  Am J Med. 2004;117:182-192
PubMed   |  Link to Article
Warsi A, Wang PS, LaValley MP, Avorn J, Solomon DH. Self-management education programs in chronic disease: a systematic review and methodological critique of the literature.  Arch Intern Med. 2004;164:1641-1649
PubMed   |  Link to Article
Grimshaw JM, Thomas RE, MacLennan G.  et al.  Effectiveness and efficiency of guideline dissemination and implementation strategies.  Health Technol Assess. 2004;8:iii-iv, 1-72
PubMed
Stone EG, Morton SC, Hulscher ME.  et al.  Interventions that increase use of adult immunization and cancer screening services: a meta-analysis.  Ann Intern Med. 2002;136:641-651
Link to Article
McDonald CJ, Hui SL, Tierney WM. Effects of computer reminders for influenza vaccination on morbidity during influenza epidemics.  MD Comput. 1992;9:304-312

Figures

Tables

Table Graphic Jump LocationTable 1. Trials of Computer-Assisted Diagnosis*
Table Graphic Jump LocationTable 2. Trials of Computer-Assisted Reminders for Cancer Screening, Vaccination, and Other Types of Preventive Care
Table Graphic Jump LocationTable 3. Trials of Computer-Assisted Diabetes Management*
Table Graphic Jump LocationTable 4. Trials of Computer-Assisted Cardiovascular Disease Management and Prevention*
Table Graphic Jump LocationTable 5. Trials of Computer-Assisted Management for Other Active Health Conditions*
Table Graphic Jump LocationTable 6. Trials of Computer Use to Monitor the Effects of Other Prescribed Treatments (Corollary Orders) or to Reduce Unnecessary Health Care Utilization*
Table Graphic Jump LocationTable 7. Trials of Computer-Assisted Anticoagulant Dosing*
Table Graphic Jump LocationTable 8. Trials of Computer-Assisted Drug Dosing and Prescribing*

References

Wyatt J, Spiegelhalter D. Evaluating medical expert systems: what to test and how?  Med Inform (Lond). 1990;15:205-217
PubMed   |  Link to Article
Sim I, Gorman P, Greenes RA.  et al.  Clinical decision support systems for the practice of evidence-based medicine.  J Am Med Inform Assoc. 2001;8:527-534
PubMed   |  Link to Article
Haynes RB, Walker CJ. Computer-aided quality assurance.  Arch Intern Med. 1987;147:1297-1301
PubMed   |  Link to Article
Johnston ME, Langton KB, Haynes RB, Mathieu A. Effects of computer-based clinical decision support systems on clinician performance and patient outcome.  Ann Intern Med. 1994;120:135-142
PubMed   |  Link to Article
Hunt DL, Haynes RB, Hanna SE, Smith K. Effects of computer-based clinical decision support systems on physician performance and patient outcomes: a systematic review.  JAMA. 1998;280:1339-1346
PubMed   |  Link to Article
Kaushal R, Shojania KG, Bates DW. Effects of computerized physician order entry and clinical decision support systems on medication safety: a systematic review.  Arch Intern Med. 2003;163:1409-1416
PubMed   |  Link to Article
Payne TH. Computer decision support systems.  Chest. 2000;118:47S-52S
PubMed   |  Link to Article
Shiffman RN, Liaw Y, Brandt CA, Corb GJ. Computer-based guideline implementation systems: a systematic review of functionality and effectiveness.  J Am Med Inform Assoc. 1999;6:104-114
PubMed   |  Link to Article
Kaplan B. Evaluating informatics applications—clinical decision support systems literature review.  Int J Med Inform. 2001;64:15-37
PubMed   |  Link to Article
Gordon RB, Grimshaw JM, Eccles M, Rowe RE, Wyatt JC. On-screen computer reminders: effects on professional practice and health care outcomes.  Cochrane Database Syst Rev. 1998;2:CD001096
Mitchell E, Sullivan F. A descriptive feast but an evaluative famine: systematic review of published articles on primary care computing during 1980-97.  BMJ. 2001;322:279-282
PubMed   |  Link to Article
Chuang JH, Hripcsak G, Heitjan DF. Design and analysis of controlled trials in naturally clustered environments: implications for medical informatics.  J Am Med Inform Assoc. 2002;9:230-238
PubMed   |  Link to Article
Fleiss JL, Cohen J. The equivalence of weighted kappa and the intraclass correlation coefficient as measures of reliability.  Educ Psychol Meas. 1973;33:613-619
Link to Article
Lewis G, Sharp D, Bartholomew J, Pelosi AJ. Computerized assessment of common mental disorders in primary care: effect on clinical outcome.  Fam Pract. 1996;13:120-126
PubMed   |  Link to Article
Cannon DS, Allen SN. A comparison of the effects of computer and manual reminders on compliance with a mental health clinical practice guideline.  J Am Med Inform Assoc. 2000;7:196-203
PubMed   |  Link to Article
Schriger DL, Gibbons PS, Langone CA, Lee S, Altshuler LL. Enabling the diagnosis of occult psychiatric illness in the emergency department: a randomized, controlled trial of the computerized, self-administered PRIME-MD diagnostic system.  Ann Emerg Med. 2001;37:132-140
PubMed   |  Link to Article
Rollman BL, Hanusa BH, Lowe HJ.  et al.  A randomized trial using computerized decision support to improve treatment of major depression in primary care.  J Gen Intern Med. 2002;17:493-503
PubMed   |  Link to Article
Pozen MW, D’Agostino RB, Selker HP, Sytkowski PA, Hood WB. A predictive instrument to improve coronary-care-unit admission practices in acute ischemic heart disease: a prospective multicenter clinical trial.  N Engl J Med. 1984;310:1273-1278
PubMed   |  Link to Article
Wyatt JR. Lessons learnt from the field trial of ACORN, an expert system to advise on chest pain. In: Barber B, Cao D, Quin D, Wagner G, eds. Proceedings of the 6th World Conference on Medical Informatics; Singapore; 1989:111-115
Selker HP, Beshansky JR, Griffith JL.  et al.  Use of the acute cardiac ischemia time-insensitive predictive instrument (ACI-TIPI) to assist with triage of patients with chest pain or other symptoms suggestive of acute cardiac ischemia: a multicenter, controlled clinical trial.  Ann Intern Med. 1998;129:845-855
PubMed   |  Link to Article
Wexler JR, Swender PT, Tunnessen WWJ, Oski FA. Impact of a system of computer-assisted diagnosis.  AJDC. 1975;129:203-205
PubMed
Wellwood J, Johannessen S, Spiegelhalter DJ. How does computer-aided diagnosis improve the management of acute abdominal pain?  Ann R Coll Surg Engl. 1992;74:40-46
PubMed
Bogusevicius A, Maleckas A, Pundzius J, Skaudickas D. Prospective randomised trial of computer-aided diagnosis and contrast radiography in acute small bowel obstruction.  Eur J Surg. 2002;168:78-83
PubMed   |  Link to Article
Turner BJ, Day SC, Borenstein B. A controlled trial to improve delivery of preventive care: physician or patient reminders?  J Gen Intern Med. 1989;4:403-409
PubMed   |  Link to Article
McPhee SJ, Bird JA, Jenkins CN, Fordham D. Promoting cancer screening: a randomized, controlled trial of three interventions.  Arch Intern Med. 1989;149:1866-1872
PubMed   |  Link to Article
McPhee SJ, Bird JA, Fordham D, Rodnick JE, Osborn EH. Promoting cancer prevention activities by primary care physicians: results of a randomized, controlled trial.  JAMA. 1991;266:538-544
PubMed   |  Link to Article
Burack RC, Gimotty PA, George J.  et al.  Promoting screening mammography in inner-city settings: a randomized controlled trial of computerized reminders as a component of a program to facilitate mammography.  Med Care. 1994;32:609-624
PubMed   |  Link to Article
Burack RC, Gimotty PA, George J, Simon MS, Dews P, Moncrease A. The effect of patient and physician reminders on use of screening mammography in a health maintenance organization: results of a randomized controlled trial.  Cancer. 1996;78:1708-1721
PubMed   |  Link to Article
Burack RC, Gimotty PA. Promoting screening mammography in inner-city settings: the sustained effectiveness of computerized reminders in a randomized controlled trial.  Med Care. 1997;35:921-931
PubMed   |  Link to Article
Burack RC, Gimotty PA, Simon M.  et al.  The effect of adding Pap smear information to a mammography reminder system in an HMO: results of randomized controlled trial.  Prev Med. 2003;36:547-554
PubMed   |  Link to Article
Chambers CV, Balaban DJ, Carlson BL, Grasberger DM. The effect of microcomputer-generated reminders on influenza vaccination rates in a university-based family practice center.  J Am Board Fam Pract. 1991;4:19-26
PubMed
Flanagan JR, Doebbeling BN, Dawson J, Beekmann S. Randomized study of online vaccine reminders in adult primary care.  Proc AMIA Symp. 1999;755-759
PubMed
Tang PC, LaRosa MP, Newcomb C, Gorden SM. Measuring the effects of reminders for outpatient influenza immunizations at the point of clinical opportunity.  J Am Med Inform Assoc. 1999;6:115-121
PubMed   |  Link to Article
McDonald CJ, Hui SL, Smith DM.  et al.  Reminders to physicians from an introspective computer medical record: a two-year randomized trial.  Ann Intern Med. 1984;100:130-138
PubMed   |  Link to Article
Tierney WM, Hui SL, McDonald CJ. Delayed feedback of physician performance versus immediate reminders to perform preventive care: effects on physician compliance.  Med Care. 1986;24:659-666
PubMed   |  Link to Article
Ornstein SM, Garr DR, Jenkins RG, Rust PF, Arnon A. Computer-generated physician and patient reminders: tools to improve population adherence to selected preventive services.  J Fam Pract. 1991;32:82-90
PubMed
Rosser WW, McDowell I, Newell C. Use of reminders for preventive procedures in family medicine.  CMAJ. 1991;145:807-814
PubMed
Tape TG, Campbell JR. Computerized medical records and preventive health care: success depends on many factors.  Am J Med. 1993;94:619-625
PubMed   |  Link to Article
Turner RC, Peden JGJ, O’Brien K. Patient-carried card prompts vs computer-generated prompts to remind private practice physicians to perform health maintenance measures.  Arch Intern Med. 1994;154:1957-1960
PubMed   |  Link to Article
Frame PS, Zimmer JG, Werth PL.  et al.  Computer-based vs manual health maintenance tracking: a controlled trial.  Arch Fam Med. 1994;3:581-588
PubMed   |  Link to Article
Overhage JM, Tierney WM, McDonald CJ. Computer reminders to implement preventive care guidelines for hospitalized patients.  Arch Intern Med. 1996;156:1551-1556
PubMed   |  Link to Article
Bonevski B, Sanson-Fisher RW, Campbell E, Carruthers A, Reid AL, Ireland M. Randomized controlled trial of a computer strategy to increase general practitioner preventive care.  Prev Med. 1999;29:478-486
PubMed   |  Link to Article
Demakis JG, Beauchamp C, Cull WL.  et al.  Improving residents’ compliance with standards of ambulatory care: results from the VA Cooperative Study on Computerized Reminders.  JAMA. 2000;284:1411-1416
PubMed   |  Link to Article
Dexter PR, Perkins S, Overhage JM, Maharry K, Kohler RB, McDonald CJ. A computerized reminder system to increase the use of preventive care for hospitalized patients.  N Engl J Med. 2001;345:965-970
PubMed   |  Link to Article
Thomas JC, Moore A, Qualls PE. The effect on cost of medical care for patients treated with an automated clinical audit system.  J Med Syst. 1983;7:307-313
PubMed   |  Link to Article
Mazzuca SA, Vinicor F, Einterz RM, Norton JA, Kalasinski LA. Effects of the clinical environment on physicians’ response to postgraduate medical education.  Am Educ Res J. 1990;27:473-488
Link to Article
Nilasena DS, Lincoln MJ. A computer-generated reminder system improves physician compliance with diabetes preventive care guidelines.  Proc Annu Symp Comput Appl Med Care. 1995;640-645
PubMed
Lobach DF, Hammond WE. Computerized decision support based on a clinical practice guideline improves compliance with care standards.  Am J Med. 1997;102:89-98
PubMed   |  Link to Article
Montori VM, Dinneen SF, Gorman CA.  et al.  The impact of planned care and a diabetes electronic management system on community-based diabetes care.  Diabetes Care. 2002;25:1952-1957
PubMed   |  Link to Article
Filippi A, Sabatini A, Badioli L.  et al.  Effects of an automated electronic reminder in changing the antiplatelet drug-prescribing behavior among Italian general practitioners in diabetic patients: an intervention trial.  Diabetes Care. 2003;26:1497-1500
PubMed   |  Link to Article
Meigs JB, Cagliero E, Dubey A.  et al.  A controlled trial of Web-based diabetes disease management: the MGH Diabetes Primary Care Improvement Project.  Diabetes Care. 2003;26:750-757
PubMed   |  Link to Article
Coe FL, Norton E, Oparil S.  et al.  Treatment of hypertension by computer and physician: a prospective controlled study.  J Chronic Dis. 1977;30:81-92
PubMed   |  Link to Article
Rogers JL, Haring OM, Goetz JP. Changes in patient attitudes following the implementation of a medical information system.  QRB Qual Rev Bull. 1984;10:65-74
PubMed
Barnett GO, Winickoff RN, Morgan MM, Zielstorff RD. A computer-based monitoring system for follow-up of elevated blood pressure.  Med Care. 1983;21:400-409
PubMed   |  Link to Article
Brownbridge G, Evans A, Fitter M, Platts M. An interactive computerized protocol for the management of hypertension: effects on the general practitioner’s clinical behaviour.  J R Coll Gen Pract. 1986;36:198-202
PubMed
McAlister NH, Covvey HD, Tong C, Lee A, Wigle ED. Randomised controlled trial of computer assisted management of hypertension in primary care.  Br Med J (Clin Res Ed). 1986;293:670-674
PubMed   |  Link to Article
Rossi RA, Every NR. A computerized intervention to decrease the use of calcium channel blockers in hypertension.  J Gen Intern Med. 1997;12:672-678
PubMed   |  Link to Article
Lowensteyn I, Joseph L, Levinton C, Abrahamowicz M, Steinert Y, Grover S. Can computerized risk profiles help patients improve their coronary risk? the results of the Coronary Health Assessment Study (CHAS).  Prev Med. 1998;27:730-737
PubMed   |  Link to Article
Hetlevik I, Holmen J, Kruger O. Implementing clinical guidelines in the treatment of hypertension in general practice: evaluation of patient outcome related to implementation of a computer-based clinical decision support system.  Scand J Prim Health Care. 1999;17:35-40
PubMed   |  Link to Article
Montgomery AA, Fahey T, Peters TJ, MacIntosh C, Sharp DJ. Evaluation of computer based clinical decision support system and risk chart for management of hypertension in primary care: randomised controlled trial.  BMJ. 2000;320:686-690
PubMed   |  Link to Article
Selker HP, Beshansky JR, Griffith JL. Use of the electrocardiograph-based thrombolytic predictive instrument to assist thrombolytic and reperfusion therapy for acute myocardial infarction: a multicenter, randomized, controlled, clinical effectiveness trial.  Ann Intern Med. 2002;137:87-95
PubMed   |  Link to Article
Ansari M, Shlipak MG, Heidenreich PA.  et al.  Improving guideline adherence: a randomized trial evaluating strategies to increase beta-blocker use in heart failure.  Circulation. 2003;107:2799-2804
PubMed   |  Link to Article
Tierney WM, Overhage JM, Murray MD.  et al.  Effects of computerized guidelines for managing heart disease in primary care.  J Gen Intern Med. 2003;18:967-976
PubMed   |  Link to Article
Weir CJ, Lees KR, MacWalter RS.  et al.  Cluster-randomized, controlled trial of computer-based decision support for selecting long-term anti-thrombotic therapy after acute ischaemic stroke.  QJM. 2003;96:143-153
PubMed   |  Link to Article
Murray MD, Harris LE, Overhage JM.  et al.  Failure of computerized treatment suggestions to improve health outcomes of outpatients with uncomplicated hypertension: results of a randomized controlled trial.  Pharmacotherapy. 2004;24:324-337
PubMed   |  Link to Article
Petrucci K, Petrucci P, Canfield K.  et al.  Evaluation of UNIS: urological nursing information systems.  Proc Annu Symp Comput Appl Med Care. 1991;43-47
PubMed
Rubenstein LV, McCoy JM, Cope DW.  et al.  Improving patient quality of life with feedback to physicians about functional status.  J Gen Intern Med. 1995;10:607-614
PubMed   |  Link to Article
Safran C, Rind DM, Davis RB.  et al.  Guidelines for management of HIV infection with computer-based patient’s record.  Lancet. 1995;346:341-346
PubMed   |  Link to Article
Dexter PR, Wolinsky FD, Gramelspacher GP.  et al.  Effectiveness of computer-generated reminders for increasing discussions about advance directives and completion of advance directive forms: a randomized, controlled trial.  Ann Intern Med. 1998;128:102-110
PubMed   |  Link to Article
East TD, Heermann LK, Bradshaw RL.  et al.  Efficacy of computerized decision support for mechanical ventilation: results of a prospective multi-center randomized trial.  Proc AMIA Symp. 1999;251-255
PubMed
Kuperman GJ, Teich JM, Tanasijevic MJ.  et al.  Improving response to critical laboratory results with automation: results of a randomized controlled trial.  J Am Med Inform Assoc. 1999;6:512-522
PubMed   |  Link to Article
Christakis DA, Zimmerman FJ, Wright JA.  et al.  A randomized controlled trial of point-of-care evidence to improve the antibiotic prescribing practices for otitis media in children.  Pediatrics. 2001;107:E15
PubMed   |  Link to Article
McCowan C, Neville RG, Ricketts IW, Warner FC, Hoskins G, Thomas GE. Lessons from a randomized controlled trial designed to evaluate computer decision support software to improve the management of asthma.  Med Inform Internet Med. 2001;26:191-201
PubMed   |  Link to Article
Eccles M, McColl E, Steen N.  et al.  Effect of computerised evidence based guidelines on management of asthma and angina in adults in primary care: cluster randomised controlled trial.  BMJ. 2002;325:941
PubMed   |  Link to Article
Lesourd F, Avril C, Boujennah A, Parinaud J. A computerized decision support system for ovarian stimulation by gonadotropins.  Fertil Steril. 2002;77:456-460
PubMed   |  Link to Article
McDonald CJ. Use of a computer to detect and respond to clinical events: its effect on clinician behavior.  Ann Intern Med. 1976;84:162-167
PubMed   |  Link to Article
Young DW. Improving the consistency with which investigations are requested.  Med Inform (Lond). 1981;6:13-17
PubMed   |  Link to Article
Fihn SD, McDonell MB, Vermes D.  et al.  A computerized intervention to improve timing of outpatient follow-up: a multicenter randomized trial in patients treated with warfarin.  J Gen Intern Med. 1994;9:131-139
PubMed   |  Link to Article
Overhage JM, Tierney WM, Zhou XH, McDonald CJ. A randomized trial of “corollary orders” to prevent errors of omission.  J Am Med Inform Assoc. 1997;4:364-375
PubMed   |  Link to Article
Tierney WM, McDonald CJ, Hui SL, Martin DK. Computer predictions of abnormal test results: effects on outpatient testing.  JAMA. 1988;259:1194-1198
PubMed   |  Link to Article
Tierney WM, Miller ME, Overhage JM, McDonald CJ. Physician inpatient order writing on microcomputer workstations: effects on resource utilization.  JAMA. 1993;269:379-383
PubMed   |  Link to Article
Hales JW, Gardner RM, Jacobson JT. Factors impacting the success of computerized preadmission screening.  Proc Annu Symp Comput Appl Med Care. 1995;728-732
PubMed
Shea S, Sideli RV, DuMouchel W, Pulver G, Arons RR, Clayton PD. Computer-generated informational messages directed to physicians: effect on length of hospital stay.  J Am Med Inform Assoc. 1995;2:58-64
PubMed   |  Link to Article
Bates DW, Kuperman GJ, Rittenberg E.  et al.  A randomized trial of a computer-based intervention to reduce utilization of redundant laboratory tests.  Am J Med. 1999;106:144-150
PubMed   |  Link to Article
Abbrecht PH, O’Leary TJ, Behrendt DM. Evaluation of a computer-assisted method for individualized anticoagulation: retrospective and prospective studies with a pharmacodynamic model.  Clin Pharmacol Ther. 1982;32:129-136
PubMed   |  Link to Article
Carter BL, Taylor JW, Becker A. Evaluation of three dosage-prediction methods for initial in-hospital stabilization of warfarin therapy.  Clin Pharm. 1987;6:37-45
PubMed
White RH, Hong R, Venook AP.  et al.  Initiation of warfarin therapy: comparison of physician dosing with computer-assisted dosing.  J Gen Intern Med. 1987;2:141-148
PubMed   |  Link to Article
White RH, Mungall D. Outpatient management of warfarin therapy: comparison of computer-predicted dosage adjustment to skilled professional care.  Ther Drug Monit. 1991;13:46-50
PubMed   |  Link to Article
Poller L, Wright D, Rowlands M. Prospective comparative study of computer programs used for management of warfarin.  J Clin Pathol. 1993;46:299-303
PubMed   |  Link to Article
Fitzmaurice DA, Hobbs FD, Murray ET, Bradley CP, Holder R. Evaluation of computerized decision support for oral anticoagulation management based in primary care.  Br J Gen Pract. 1996;46:533-535
PubMed
Vadher B, Patterson DL, Leaning M. Evaluation of a decision support system for initiation and control of oral anticoagulation in a randomised trial.  BMJ. 1997;314:1252-1256
PubMed   |  Link to Article
Vadher BD, Patterson DL, Leaning M. Comparison of oral anticoagulant control by a nurse-practitioner using a computer decision-support system with that by clinicians.  Clin Lab Haematol. 1997;19:203-207
PubMed   |  Link to Article
Ageno W, Turpie AG. A randomized comparison of a computer-based dosing program with a manual system to monitor oral anticoagulant therapy.  Thromb Res. 1998;91:237-240
PubMed   |  Link to Article
Poller L, Shiach CR, MacCallum PK.  et al.  Multicentre randomised study of computerised anticoagulant dosage.  Lancet. 1998;352:1505-1509
PubMed   |  Link to Article
Fitzmaurice DA, Hobbs FD, Murray ET, Holder RL, Allan TF, Rose PE. Oral anticoagulation management in primary care with the use of computerized decision support and near-patient testing: a randomized, controlled trial.  Arch Intern Med. 2000;160:2343-2348
PubMed   |  Link to Article
Manotti C, Moia M, Palareti G, Pengo V, Ria L, Dettori AG. Effect of computer-aided management on the quality of treatment in anticoagulated patients: a prospective, randomized, multicenter trial of APROAT (Automated PRogram for Oral Anticoagulant Treatment).  Haematologica. 2001;86:1060-1070
PubMed
Mungall DR, Anbe D, Forrester PL.  et al.  A prospective randomized comparison of the accuracy of computer-assisted versus GUSTO nomogram–directed heparin therapy.  Clin Pharmacol Ther. 1994;55:591-596
PubMed   |  Link to Article
Hurley SF, Dziukas LJ, McNeil JJ, Brignell MJ. A randomized controlled clinical trial of pharmacokinetic theophylline dosing.  Am Rev Respir Dis. 1986;134:1219-1224
PubMed
Gonzalez ER, Vanderheyden BA, Ornato JP, Comstock TG. Computer-assisted optimization of aminophylline therapy in the emergency department.  Am J Emerg Med. 1989;7:395-401
PubMed   |  Link to Article
Verner D, Seligmann H, Platt S.  et al.  Computer assisted design of a theophylline dosing regimen in acute bronchospasm: serum concentrations and clinical outcome.  Eur J Clin Pharmacol. 1992;43:29-33
PubMed   |  Link to Article
Casner PR, Reilly R, Ho H. A randomized controlled trial of computerized pharmacokinetic theophylline dosing versus empiric physician dosing.  Clin Pharmacol Ther. 1993;53:684-690
PubMed   |  Link to Article
Begg EJ, Atkinson HC, Jeffery GM, Taylor NW. Individualised aminoglycoside dosage based on pharmacokinetic analysis is superior to dosage based on physician intuition at achieving target plasma drug concentrations.  Br J Clin Pharmacol. 1989;28:137-141
PubMed   |  Link to Article
Hickling K, Begg E, Moore ML. A prospective randomised trial comparing individualised pharmacokinetic dosage prediction for aminoglycosides with prediction based on estimated creatinine clearance in critically ill patients.  Intensive Care Med. 1989;15:233-237
PubMed   |  Link to Article
Burton ME, Ash CL, Hill DP Jr, Handy T, Shepherd MD, Vasko MR. A controlled trial of the cost benefit of computerized Bayesian aminoglycoside administration.  Clin Pharmacol Ther. 1991;49:685-694
PubMed   |  Link to Article
Peck CC, Sheiner LB, Martin CM.  et al.  Computer-assisted digoxin therapy.  N Engl J Med. 1973;289:441-446
PubMed   |  Link to Article
Rodman JH, Jelliffe RW, Kolb E.  et al.  Clinical studies with computer-assisted initial lidocaine therapy.  Arch Intern Med. 1984;144:703-709
PubMed   |  Link to Article
Ryff-de Leche A, Engler H, Nutzi E, Berger M, Berger W. Clinical application of two computerized diabetes management systems: comparison with the log-book method.  Diabetes Res. 1992;19:97-105
PubMed
Horn W, Popow C, Miksch S.  et al.  Development and evaluation of VIE-PNN, a knowledge-based system for calculating the parenteral nutrition of newborn infants.  Artif Intell Med. 2002;24:217-228
PubMed   |  Link to Article
McDonald CJ. Protocol-based computer reminders, the quality of care and the non-perfectability of man.  N Engl J Med. 1976;295:1351-1355
PubMed   |  Link to Article
McDonald CJ, Wilson GA, McCabe GP Jr. Physician response to computer reminders.  JAMA. 1980;244:1579-1581
PubMed   |  Link to Article
White KS, Lindsay A, Pryor TA, Brown WF, Walsh K. Application of a computerized medical decision-making process to the problem of digoxin intoxication.  J Am Coll Cardiol. 1984;4:571-576
PubMed   |  Link to Article
Rotman BL, Sullivan AN, McDonald TW.  et al.  A randomized controlled trial of a computer-based physician workstation in an outpatient setting: implementation barriers to outcome evaluation.  J Am Med Inform Assoc. 1996;3:340-348
PubMed   |  Link to Article
Tamblyn R, Huang A, Perreault R.  et al.  The medical office of the 21st century (MOXXI): effectiveness of computerized decision-making support in reducing inappropriate prescribing in primary care.  CMAJ. 2003;169:549-556
PubMed
McKinley BA, Moore FA, Sailors RM.  et al.  Computerized decision support for mechanical ventilation of trauma induced ARDS: results of a randomized clinical trial.  J Trauma. 2001;50:415-424
PubMed   |  Link to Article
Hetlevik I, Holmen J, Kruger O, Kristensen P, Iversen H, Furuseth K. Implementing clinical guidelines in the treatment of diabetes mellitus in general practice: evaluation of effort, process, and patient outcome related to implementation of a computer-based decision support system.  Int J Technol Assess Health Care. 2000;16:210-227
PubMed   |  Link to Article
Rogers JL, Haring OM, Wortman PM, Watson RA, Goetz JP. Medical information systems: assessing impact in the areas of hypertension, obesity and renal disease.  Med Care. 1982;20:63-74
PubMed   |  Link to Article
Rogers JL, Haring OM. The impact of a computerized medical record summary system on incidence and length of hospitalization.  Med Care. 1979;17:618-630
PubMed   |  Link to Article
Hetlevik I, Holmen J, Kruger O.  et al.  Implementing clinical guidelines in the treatment of hypertension in general practice.  Blood Press. 1998;7:270-276
PubMed   |  Link to Article
Holbrook A, Xu S, Banting J. What factors determine the success of clinical decision support systems?  AMIA Annu Symp Proc. 2003;862
PubMed
Reisman Y. Computer-based clinical decision aids: a review of methods and assessment of systems.  Med Inform (Lond). 1996;21:179-197
PubMed   |  Link to Article
Kawamoto K, Lobach DF. Clinical decision support provided within physician order entry systems: a systematic review of features effective for changing clinician behavior.  AMIA Annu Symp Proc. 2003;361-365
PubMed
Berlin A, Sorani M, Sim I. Characteristics of outpatient clinical decision support systems: a taxonomic description.  Medinfo. 2004;2004:578-581
PubMed
Fox J, Thomson R. Clinical decision support systems: a discussion of quality, safety and legal liability issues.  Proc AMIA Symp. 2002;265-269
PubMed
Ofman JJ, Badamgarav E, Henning JM.  et al.  Does disease management improve clinical and economic outcomes in patients with chronic diseases? a systematic review.  Am J Med. 2004;117:182-192
PubMed   |  Link to Article
Warsi A, Wang PS, LaValley MP, Avorn J, Solomon DH. Self-management education programs in chronic disease: a systematic review and methodological critique of the literature.  Arch Intern Med. 2004;164:1641-1649
PubMed   |  Link to Article
Grimshaw JM, Thomas RE, MacLennan G.  et al.  Effectiveness and efficiency of guideline dissemination and implementation strategies.  Health Technol Assess. 2004;8:iii-iv, 1-72
PubMed
Stone EG, Morton SC, Hulscher ME.  et al.  Interventions that increase use of adult immunization and cancer screening services: a meta-analysis.  Ann Intern Med. 2002;136:641-651
Link to Article
McDonald CJ, Hui SL, Tierney WM. Effects of computer reminders for influenza vaccination on morbidity during influenza epidemics.  MD Comput. 1992;9:304-312

Letters

CME
Meets CME requirements for:
Browse CME for all U.S. States
Accreditation Information
The American Medical Association is accredited by the Accreditation Council for Continuing Medical Education to provide continuing medical education for physicians. The AMA designates this journal-based CME activity for a maximum of 1 AMA PRA Category 1 CreditTM per course. Physicians should claim only the credit commensurate with the extent of their participation in the activity. Physicians who complete the CME course and score at least 80% correct on the quiz are eligible for AMA PRA Category 1 CreditTM.
Note: You must get at least of the answers correct to pass this quiz.
You have not filled in all the answers to complete this quiz
The following questions were not answered:
Sorry, you have unsuccessfully completed this CME quiz with a score of
The following questions were not answered correctly:
Commitment to Change (optional):
Indicate what change(s) you will implement in your practice, if any, based on this CME course.
Your quiz results:
The filled radio buttons indicate your responses. The preferred responses are highlighted
For CME Course: A Proposed Model for Initial Assessment and Management of Acute Heart Failure Syndromes
Indicate what changes(s) you will implement in your practice, if any, based on this CME course.

Multimedia

Some tools below are only available to our subscribers or users with an online account.

Web of Science® Times Cited: 931

Related Content

Customize your page view by dragging & repositioning the boxes below.

See Also...
Articles Related By Topic
Related Collections
PubMed Articles
JAMAevidence.com

Users' Guides to the Medical Literature
Chapter 9.6. Clinical Decision Support Systems

The Rational Clinical Examination
Evidence Summary and Review 3