Computer Technology and Clinical Work: Title and subTitle BreakStill Waiting for Godot

Process-supporting information technology (IT) has been heralded as an important building block in attempts to improve the quality and safety of health care. Two areas in particular have drawn both attention and funding. The first is clinical decision support; that is, information systems designed to improve clinicians’ decision making. The second is computerized physician order entry (CPOE) as a means for reducing medication errors. The literature in these fields has been characterized by frequent reports of success, often accompanied by predictions of a bright new (and near) future; however, the future seems never to arrive. Behind the cheers and high hopes that dominate conference proceedings, vendor information, and large parts of the scientific literature, the reality is that systems that are in use in multiple locations, that have satisfied users, and that effectively and efficiently contribute to the quality and safety of care are few and far between.¹

In this issue of JAMA, 2 articles report results in these areas. Garg et al² provide an updated systematic review evaluating the impact of computerized clinical decision support systems, and Koppel et al³ examine the specific failures introduced by a CPOE system intended (at least in part) to reduce such failures.

The summary by Garg et al of 100 trials of clinical decision support systems over a 6-year span is critical. About two thirds of the studies claimed improved clinician performance, but these assessments were often biased; when the authors were not also the system developers, less than half of the systems showed an improvement. In fact, “grading oneself” was the only factor that was consistently associated with good evaluations.

Clinical decision support systems come in many different forms, have myriad aims, and can be implemented in many ways, so it is fair to ask if these systems can really be approached as a single intervention, as Garg et al do. In addition, lack of improved performance could be due to poorly designed decision technology or to poor use of the technology by clinicians. This, in turn, could be due to a poor human-computer interface or to much simpler factors such as goal conflicts or lack of time or support among colleagues.⁴ So, it is not clear exactly what is being measured.

In a different design, the study by Koppel et al of users of a single CPOE system in a large academic medical center identified 24 different types of failures of which users were aware; roughly half the participants said these faults occurred from several times per week to daily. While this study examined only the problems, not the successes of CPOE, it serves as a cautionary balance to reports of success, particularly because it studied a mature, widely used, commercially available system, as opposed to a “one-off,” single-site system assessed by its own developers.

These results are disappointing but should not be surprising. There is a long-standing, rich, and abundant literature on the problems associated with the introduction of computer technology into complex work in other domains,^{5 - 9} as well as occasional notes in health care.^{10 - 15} Clearly, there is no reason to expect health care, which is from an organizational standpoint probably the most complex enterprise in modern society, to be immune to them. Taken together, these 2 studies^{2 - 3} suggest that important lessons about introducing new technologies into complex work seem to have been missed. For a small but important example, it has long been established in software engineering that systems cannot be adequately evaluated by their developers,¹⁶ a principle that seems to be commonly overlooked in health care. Since roughly 75% of all large IT projects in health care fail,¹⁷ inattention to these lessons is, at best, wasteful of time and resources and, at worst, harmful to patients and clinicians.

To begin to move forward, it is necessary to dispense with the commonly held notion that these problems are simply bits of bad programming or poor implementation that can easily be excised or avoided the next time around. The reality is that many of the difficulties do not result from bad parts of the systems but are inherent in the perspectives and theories of medical work (and the role of IT in this work) that are prevalent among health informaticians and those who make decisions on acquisition and implementation. In short, rather than framing the problem as “not developing the systems right,” these failures demonstrate “not developing the right systems” due to widespread but misleading theories about both technology and clinical work.

The misleading theory about technology is that technical problems require technical solutions; ie, a narrowly technical view of the important issues involved that leads to a focus on optimizing the technology. In contrast, a more useful approach views the clinical workplace as a complex system in which technologies, people, and organizational routines dynamically interact. This view holds the following: (1) Organizations are simultaneously social (eg, consisting of people, values, norms, culture) and technical (ie, without tools, equipment, procedures, technology, and facilities, the people could not work and the organization would not exist). (2) These social and technical elements are deeply interdependent and interrelated—hence, the term socio-technical systems. Every change in one element affects the other. (3) Accordingly, good design or implementation is not a technical problem but rather one of jointly optimizing the combined sociotechnical system.¹⁸

Technological change takes place as a dynamic cycle,^{19 - 20} as the effects of change reverberate through the system.²¹ New technology changes work practices, which in turn changes how the technology is used, which leads to changes in the technology, which induces new changes in work practices, and so on. Lack of attention to how the technological artifact will affect (and be affected by) the organization in which it becomes embedded lies at the core of many technological failures. A further implication of this view is that the introduction of computerized tools into health care should not be viewed as a problem in technology but rather a problem in organizational change and, particularly, one of guiding organizational change by a process of experimentation and mutual learning rather than one of planning, command, and control.^{1 ,22}

One important area of the sociotechnical system commonly ignored is the organizational dynamics and power relationships involved. Most decision support and CPOE systems have multiple dimensions and goals: to improve safety, increase quality, reduce costs, provide greater accountability, track clinician performance, collect information for management, etc. The impetus for their implementation often comes from the improvements expected to the secondary “back-end” business processes, not from improvements to the primary “front-end” work of clinical care. But the burdens of achieving these benefits for the organization as a whole are placed on the already beleaguered front-end workers, who experience few of the benefits and often have little voice in decisions about tools and vendors. These disjointed views about the benefits of new technology can produce conflicting judgments about its success or failure that must be negotiated in the organization.²⁰

The failure to attend to the social component of the sociotechnical system is compounded by misconceptions about the nature of clinical work. Technical artifacts (such as computer systems) can be viewed as embodying the implicit theories of their creators about how work is done, what characterizes workers and their environment, what problems they face, and how they will use the artifact.^{18 - 19} It follows that the success of the artifact in the real world depends on the adequacy of these implicit theories.²³ However, there is quite a large mismatch between the implicit theories embedded in these computer systems and the real world of clinical work. Clinical work, especially in hospitals, is fundamentally interpretative, interruptive, multitasking, collaborative, distributed, opportunistic, and reactive.^{1 ,13 ,20 ,24 - 25} In contrast, CPOE systems and decision support systems are based on a different model of work: one that is objective, rationalized, linear, normative, localized (in the clinician’s mind), solitary, and single-minded. Such models tend to reflect the implicit theories of managers and designers, not of front-line workers. The result of this mismatch is that many of the failed attempts at computer-based clinical systems were bound to fail because the model of health care work inscribed in these tools clashed too much with the actual nature of clinical work.¹

The mismatch problem has a special salience in the area of clinical decision support. The theories of decision making embedded in decision support tools are typically rationalized, logical, sequential, often statistical models of “singularities in time”—decision points. However, studies of workers in complex, uncertain domains suggest that in many situations, “decisions” become apparent only in retrospect^{24 ,26 - 27} and that what workers are doing might better be called sense making than decision making.²⁸ Thus, tools like clinical decision analysis are seldom used by real clinicians to make real decisions about specific patients because the task they support does not match the clinician’s task.

Simply having greater clinician participation in the design of these technologies will not fix this problem. Most domain experts have little insight into their own work processes or sources of expertise. Norman²⁹ has argued that common approaches such as expert panels, focus groups, questionnaires, and surveys are poor tools for learning about behavior because they are divorced from actual use. According to Norman, “We humans like to think that we know why we act as we do, but we don’t, however much we like to explain our actions. The fact that [our] reactions are subconscious makes us unaware of our true reactions and their causes.” Norman and others point out that trained professionals who observe real use in real situations can often tell more about individuals’ work, successes, failures, and preferences—and the reasons for them—than the individuals themselves.^{29 - 31}

In addition, this lack of self-insight is the fundamental reason why system developers cannot objectively evaluate the systems they have developed. No matter how much they may try to be objective, the very process of development and refinement has created in them hidden assumptions about “the way things work” that make it impossible for them to envision some of the ways in which things might go wrong when users who do not share those assumptions interact with the system. Organizations serving as development sites also have this problem; in the course of development, the organization adapts to the system and the system to the organization, so that system performance at the development site is not a good predictor of performance in other settings.

Useful information technology is a sine qua non to bridging the “quality chasm” that has been so clearly identified by the Institute of Medicine and others.³² Yet an information technology in and of itself cannot do anything, and when the patterns of its use are not tailored to the workers and their environment to yield high-quality care, the technological interventions will not be productive. This implies that any IT acquisition or implementation trajectory should, first and foremost, be an organizational change trajectory. This is true at both the organizational level and the national level; a national health IT infrastructure without a clear logic about how health care organizations will become engaged in this infrastructure is bound to fail. It is this challenge to which the findings of Garg et al and Koppel et al point, and nothing less.