Do We Know What Inappropriate Laboratory Utilization Is?: Title and subTitle BreakA Systematic Review of Laboratory Clinical Audits FREE

THE APPROPRIATE utilization of laboratory tests is necessary for optimal patient care. The utilization of diagnostic laboratories has increased over the last several decades in many medical jurisdictions around the world.^{1 - 3} In Ontario, Canada's largest province, the total number of tests performed annually has increased by more than 130% between 1976 and 1992.⁴ The cost of increased laboratory utilization has been considerable, rising from an annual expenditure (in inflation-adjusted dollars) of Can $33.29 per person in 1979 to Can $103.36 in 1992.⁵ This increased laboratory use is appropriate if it allows accurate diagnoses to be made, ideal therapy to be identified and monitored, and accurate prognoses to be established.

Perhaps because of this increased use, laboratory utilization has been analyzed extensively with studies focused primarily on physician-ordering practices.^{6 - 11} Many studies have suggested that inappropriate test ordering is a primary reason for increased laboratory use.^{12 - 20} If inappropriate laboratory utilization does exist, it must be corrected for several reasons. First, inappropriate testing not only causes unnecessary patient discomfort, it also increases the likelihood of generating false positive results, which cause unnecessary worry and further investigation.²¹ Second, laboratory tests are examples of "little-ticket" health care technologies,²² which, because they are commonly used, cost the health care system large amounts of money.^{22 - 25} Finally, inappropriate laboratory utilization may be associated with other inefficiencies in health care delivery. Identifying inadequacies in the use of laboratory services may disclose problems in other areas of health care.

These considerations highlight the importance of determining when inappropriate laboratory utilization occurs and raise the question, "Do we know what inappropriate laboratory utilization is?" Evidence-based users' guides to assessing the medical literature have focused on laboratory test performance characteristics,²⁶ but have not directly addressed test utilization. With this systematic review, we set out to identify published studies that measured inappropriate laboratory utilization. We used guidelines for appraising utilization reviews²⁷ to determine the validity of each study's criteria for assessing appropriateness and the application of the criteria. However, since those methodological guidelines are directed primarily toward therapeutic utilization reviews, we determined the studies' compliance with the guidelines as well as the applicability of the guidelines to diagnostic utilization reviews. Finally, we have identified directions for future research in this area.

The MEDLINE database was searched from 1966 to September 1997 by crossing several subject headings (explode laboratories, diagnostic services, diagnostic services–routine, explode diagnosis–laboratory, or explode quality assurance–health care) with several topic headings (attitude of health personnel,physicians' practice patterns, explode guidelines, or utilization review) or text words (unnecessary, duplication, efficiency, inappropriate, overutilization, underutilization, quality control, quality assurance, guidelines, utilization, or utilization review). Citations were limited to human subjects but no language restriction was used. A very inclusive search strategy was used because laboratory utilization is a broad topic applying to almost all areas of medicine. Each citation's title and abstract was reviewed and all studies potentially concerned with efficient laboratory use or methods to decrease laboratory use were retrieved.

Studies were included if they specified criteria for inappropriate laboratory utilization and actually used them in an audit. Studies were excluded if they assessed radiological or pathological tests, dealt exclusively with laboratory quality control issues (eg, culturing urine from leaky specimen containers), or assessed the appropriateness of the test's urgency (eg, was a "stat" order appropriate?). Studies that developed criteria for inappropriate laboratory utilization^{28 - 45} were excluded if the criteria were not applied to a second group of patients. Finally, we excluded studies that determined the appropriateness of screening tests (eg, preoperative screening batteries, preoperative bleeding times, blood cross-matching). We reasoned that, instead of determining the appropriateness of a test for a particular patient in a particular situation, studies addressing the appropriateness of screening tests determine whether a particular test is appropriate for a group of patients. Determining the appropriateness of screening tests has been addressed in several publications.^{46 - 47}

These inclusion and exclusion criteria were applied to the references of all articles as well as citations retrieved from EMBASE and HEALTHSTAR databases with a search strategy similar to that used in MEDLINE. Finally, hand-searching was used to screen all articles in the Toward Optimal Laboratory Use series from JAMA and all references in 3 commonly cited textbooks on laboratory test use.^{48 - 50}

Descriptive data that were abstracted from each study included country of publication, study setting, laboratory tests assessed, and the number of patients and physicians included in the study. All articles were categorized based on whether the criteria for inappropriate laboratory use were implicit (objective criteria for inappropriate use not provided or criteria very broad, leaving much latitude for interpretation by the reviewer) or explicit. Guidelines for the evaluation of utilization reviews²⁷ were applied to each study by a single reviewer. These guidelines help determine whether the criteria are valid and whether they are applied appropriately. Validity of the criteria was assessed based on 3 components: the quality of the evidence on which the criteria are based; the methods used to tap expert opinion; and whether compliance with the criteria is associated with better patient outcomes. The strength of criteria application was based on 2 components: the reliability of criteria use and whether the impact of uncertainty in their application was measured. Estimates for the prevalence of inappropriate laboratory utilization were documented. When interventions aimed at altering laboratory use were studied, only preintervention measurements of inappropriate laboratory use were determined.

Figure 1 illustrates results of the search strategy. Eighty-three articles potentially providing criteria for laboratory inappropriateness were identified and retrieved. Seventy-one of these studies were excluded because they provided no appropriateness criteria (n=24); were review articles (n=19); or studied screening tests (n=10), laboratory quality control (n=6), the appropriateness of laboratory test urgency (n=3), or other topics (n=9). Of the 24 studies without a definition of inappropriate utilization, 4 derived criteria but did not apply them to a separate group of patients,^{19 ,44 ,51 - 52} whereas the other studies measured laboratory use without an appropriateness assessment.^{30 ,53 - 71} Several of these studies referred to "inappropriate" and "needless" laboratory use in the study title or abstract without providing a definition of these terms in the manuscript.^{54 ,56 - 57} Twelve articles from the MEDLINE search were appropriate for study inclusion. Thirty-two other articles were identified by searching EMBASE and HEALTHSTAR databases, reviewing references of all retrieved articles, and hand-searching journal series and textbooks on laboratory testing. Thirty of these were in the MEDLINE database but were not captured by our initial search strategy or were overlooked during the initial citation screening. Four articles were excluded because they dealt exclusively with serum sampling time without assessing the appropriateness of ordering the test itself.^{72 - 75}

Forty-four articles meeting our inclusion criteria were identified. One study⁷⁶ presented explicit criteria for both serum calcium and lactate dehydrogenase, which are discussed separately. Criteria for inappropriate use were classified as implicit for 11 studies and explicit for 33 studies. Studies were published between 1965 and 1995. Most studies (70%) originated from the United States, with the remainder from the United Kingdom,^{77 - 80} the Netherlands,^{81 - 82} Australia,^{83 - 85} Canada,^{86 - 88} and 1 each from Egypt⁸⁹ and Thailand.⁹⁰

The 11 studies that used implicit criteria for appropriateness are listed in Table 1. All of these studies were conducted in teaching institutions and, with 2 exceptions,^{90 ,92} studied laboratory utilization for patients in hospitals. The number of patients assessed in the studies ranged from 25 to 555. Four studies cited the number of physicians ordering tests, which ranged from 3⁹³ to 18.⁹⁰ Investigations that were assessed in the studies included basic biochemistry tests alone,^{94 - 95} with hematology tests, ^{78 ,93 ,96} or with hematology and microbiology tests.^{79 ,89 - 90 ,97} It was unclear from Dickinson's article⁹² what investigations were assessed.

Each study's criteria were based partly on whether the test helped diagnose or treat patients (Table 1). As is the norm with implicit review, the criteria were subjective and relied on interpretation by the reviewer. Several studies added qualifiers to the criteria, apparently to increase the reliability of the reviews^{94 ,96 - 97} ; however, the qualifiers themselves left latitude for interpretation. Since the criteria were broad, it is not surprising that none were based on specific published evidence. Two studies^{92 ,96} used a consensus conference to develop the broad criteria for implicit review, but an explicit group process was not reported. None of the studies correlated compliance with the criteria and favorable patient outcomes. However, this methodological standard had limited applicability because the studies were assessing the ordering of basic laboratory tests in isolation.

Most studies used medical record review or a summary of the medical record review⁹⁰ to apply the criteria. All audit criteria required the chart reviewer to identify each patient's symptoms or diagnoses to determine test appropriateness. Most studies^{78 - 79 ,89 ,92 - 96} used a single reviewer for medical record abstraction. Two studies clarified uncertainties with the opinion of a second clinician⁹⁶ or interviews with the requesting physician.⁹⁵ In the study by Bold and Corrin,⁸⁰ the senior surgical residents from each team determined appropriateness of all tests, presumably based on their memory of each case.

Only 2 studies directly compared the classification of laboratory appropriateness made by different reviewers.^{78 ,97} In 1 study,⁷⁸ 42.8% of 339 laboratory tests were classified as inappropriate by a panel of 3 physicians compared with only 26.5% by a single pathologist. Bloomgarden and Sidel⁹⁷ showed that agreement between 2 physician groups was significantly beyond chance (κ=0.63, P<.001). Dickinson⁹² determined that intrarater agreement was 97% when the same reviewer reabstracted the charts after 3 months. One study allowed test appropriateness to be classified as uncertain⁷⁸ but no sensitivity analysis (ie, the effect of reclassifying uncertain categorizations on the study's conclusion) was reported on these data.

The proportion of inappropriate tests varied among studies. Dixon and Laszlo⁹⁴ and Sandler⁷⁹ had the highest estimates, citing more than 90% of tests as inappropriate. These studies arguably also had the most stringent assessment criteria (Table 1). Data provided by Young⁹³ did not allow for an estimate of test inappropriateness by individual test type. The rest of the studies concluded that between 10% and 50% of tests were inappropriate. None of the studies reported confidence intervals for the prevalence of inappropriate tests.

These studies are presented in Table 2, Table 3, and Table 4 and are categorized according to the laboratory test studied. In contrast with the implicit group, these studies concentrated on single laboratory tests. With several exceptions,^{81 - 83 ,98 - 104} most studies were conducted on patients hospitalized within teaching centers. The number of patients varied among studies (range, 15-2897) but they generally contained more than the implicit group. Finally, the sampling frames of these studies were sometimes very narrow and often included patients only if they had testing at a particular frequency,^{76 ,105} with a completed form,^{86 ,106} or for a particular indication.⁸⁴

Although the explicit criteria for inappropriateness appear diverse (Table 2, Table 3, and Table 4), several themes are common to them. Criteria were based on testing frequency,^{76 ,82 ,88 ,99 ,105 ,107 - 108} timing of the test in relation to previous medications and tests,^{84 ,86 - 87 ,102 - 103 ,108 - 113} test choice compared with possible alternatives,^{101 ,114} clinical indications for the test,^{81 ,85 ,100 ,104 ,106 ,115 - 118} or probability that a test result was abnormal.^{83 ,98} Two studies^{119 - 120} defined a single blood culture as inappropriate because distinguishing between a true positive blood culture and a contaminant can be difficult when only 1 culture is done. Flynn et al¹²¹ defined a single gentamicin level as inappropriate since peak and trough levels are usually needed for appropriate dosage adjustment.

Most criteria were based on published literature or local consensus. Several studies^{81 ,87 ,98 ,101 ,110 ,112 - 114 ,119} justified the criteria used by citing a varying number of published articles on the topic. Schoenenberger et al¹⁰⁹ documented the methods used for conducting a systematic review to derive their criteria. Three studies based criteria on guidelines published by the American College of Physicians^{108 ,115} and other specialty societies.¹¹¹ Local expert opinion was specified for criteria derivation in some studies^{76 ,106 - 107 ,109 ,116 - 118} but the methods used to elicit these opinions were not delineated. Rhyne and Gehlbach¹⁰⁰ based their criteria for thyroid testing on a survey involving physicians in training. Boon-Falleur et al⁸¹ used a probability analysis associating symptoms with thyroid dysfunction to create criteria. Morris et al⁹⁹ based their criteria on detailed data from a cohort of patients investigated at their hospital. Finally, fundamentals of physiology⁹⁸ and pharmacology^{84 ,86 ,88 ,103 ,109 - 112 ,121} were used for criteria in the remaining studies.

Since explicit criteria were used, these studies had more options for data collection necessary for criteria application. Many used data from laboratory databases^{82 - 83 ,98 - 99 ,101 ,103 ,105 ,107 - 108 ,119 - 121} or requisition forms completed by the ordering physician.^{81 ,84 ,86 ,106 ,113 - 114} The remainder of the studies^{76 ,87 - 88 ,100 ,104 ,109 ,112 ,115 - 118} required medical record review. The data source for 2 studies was uncertain.^{85 ,102} Several^{76 ,100 ,109 ,111} determined abstraction reliability with a second medical record reviewer but only Schoenenberger et al¹⁰⁹ measured agreement beyond chance (κ=0.61, P <.001). Two studies clarified data from chart review by interviewing the physician who ordered the questionable tests.^{106 ,114} None of the studies associated compliance with the criteria with positive patient outcomes. Bowman et al⁸³ and Morris et al,⁹⁹ however, showed that tests that complied with the criteria were significantly more likely to have abnormal results.

Estimates for the proportion of inappropriate tests varied extensively (range, 4.5%-95%). Tests with the highest estimates for inappropriate utilization included prothrombin time,¹¹⁵ calcium,⁷⁶ cerebrospinal fluid analysis for VDRL test,⁹⁸ and antiepileptic drug monitoring.^{88 ,109} The study by Saxena et al¹⁰⁸ probably overestimates inappropriate laboratory use because the criteria used for timing of repeat testing were stringent (ie, it assumes a meaningful difference between a serial creatine kinase level measured at 10 hours vs one measured at 12 hours). Estimates for inappropriate laboratory utilization ranged from 5% to 50% in the other studies. It is plausible that criteria relying on data from requisition forms would have higher inappropriateness rates because physician documentation on the requisition could be very incomplete. However, studies using this method^{81 ,84 ,106 ,114} did not find a higher prevalence of inappropriateness.

Appropriate laboratory utilization is a cornerstone of optimal medical practice. Patients benefit from accurate diagnoses, proper therapeutic monitoring, and precise prognostications, all of which result from the use of sensible diagnostic technologies. Inappropriate laboratory utilization can not only harm patients, it is also expensive. The authors of all studies cited in this review deserve commendation for trying to measure its prevalence. A broad range of criteria were used for different laboratory tests in distinct clinical situations. The validity of the criteria and the reliability of their application varied among the studies. The prevalence of inappropriate laboratory use also varied extensively in its magnitude.

Some studies used implicit as opposed to explicit criteria. Implicit criteria are versatile because a single definition can be applied to a broad range of tests. They allow a more complete review of laboratory utilization because the reviewer can consider multiple components of a patient's situation when determining the appropriateness of a laboratory test. However, there are many drawbacks to using implicit appropriateness criteria, including interreviewer variability in their interpretation and application. Use of implicit criteria usually demands detailed review of medical records. If physicians do not completely document their clinical reasoning or the patient's findings, laboratory tests may be judged inappropriate. The review could then become more an assessment of clinical documentation than of laboratory utilization. Also, medical record review is time-consuming and expensive. This will decrease the number of diagnostic episodes that can be assessed and therefore decrease the scope and precision of any assessment of laboratory utilization. With explicit criteria, it is often possible to collect the necessary data from databases or data requisitions, thus avoiding many of the problems associated with retrospective medical record review. Finally, explicit criteria make it possible to determine appropriateness criteria using prospectively collected data (eg, from a data requisition form). Prospectively collected data help avoid bias due to omission of physician documentation.

Two studies^{94 ,96} classified tests as inappropriate if therapy did not change as a result of test ordering. This criterion is problematic because the reviewer determining test appropriateness could not know whether or how therapy could have changed if the test result was different. Also, it is unreasonable to expect a laboratory test to single-handedly alter therapy because test results are only 1 of several parameters used in medical decision making. Finally, even tests that do not change therapy may provide useful information to physicians and patients.

Two studies^{79 - 80} based implicit reviews partly on whether the test result was abnormal. This criterion is questionable because normal test results are the most common outcome with laboratory testing. In addition, this criterion leads to an impossible double bind. To have completely appropriate laboratory utilization, one would need to know in advance that each test result would be abnormal. However, if the test result is known in advance, then ordering the test may be superfluous and inappropriate.

Many studies that used explicit criteria did not comply with guidelines for establishing the validity of utilization review criteria.²⁷ Poor study design or poor reporting could explain noncompliance with these guidelines. However, it is also arguable that these guidelines are not as readily applicable to diagnostic as to therapeutic utilization reviews. Diagnostic technologies are often not tested with the same evaluative rigor as many therapeutic technologies. This is understandable in light of the lower cost of many diagnostic tests, their use as only 1 element in clinical decision making, and the attendant uncertainties in their link to patients' outcomes. As a result, there is a paucity of high-level evidence (such as randomized controlled trials) assessing diagnostic technology on which audit criteria can be based.²⁶ This could explain why less than half of the studies cited evidence on which their audit criteria were based. Morris et al⁹⁹ compensated for this by conducting a cohort study at their own institution on which their appropriateness criteria were based.

Compliance with utilization review guidelines is problematic for audits of diagnostic technologies for other reasons. For criteria development, expert opinion should be tapped using an explicit, systematic, and reliable process when published evidence is lacking or needs supplementation.²⁷ This could be more difficult in the diagnostic arena because test appropriateness is highly dependent on the pretest probability of the condition being sought. The number of clinical situations in which pretest probabilities have been rigorously determined is relatively small. Also, the subjective determination of these probabilities¹²² and diagnostic or risk thresholds^{123 - 124} varies widely among physicians. These difficult issues may explain why only a few studies tapped expert opinion to derive criteria and why none used systematic methods in doing so.¹²⁵

Finally, the validity of the criteria is enhanced when their compliance is associated with improved patient outcomes.²⁷ For example, Kravitz et al¹²⁶ showed that patients meeting panel criteria for revascularization who did not undergo such procedures fared significantly worse than those who did. In contrast, it is difficult for diagnostic technologies to conform with this criterion because tests are often used to determine whether the primary prognosticator (ie, the disease) is present. Test use by itself may therefore not be associated with patient outcomes. To compensate for this, several studies tried to validate their criteria by associating criteria compliance with surrogate outcomes such as costs or the prevalence of abnormal test results.^{83 ,99} Other investigators dealt with the difficulty of complying with these accepted methods of establishing valid audit criteria by basing them on physiological⁹⁸ or pharmacological^{84 ,127} tenets. In our view, these are defensible alternatives, since pharmacokinetic and physiological principles can readily confirm that some tests will be redundant if repeated too frequently, whereas other tests cannot yield valid information unless performed at a prespecified interval in relation to some therapeutic intervention or physiological change in the patient's status.

For these reasons, it is understandable that audits of laboratory test utilization infrequently comply with methodological guidelines for validation of appropriateness criteria. What, then, of compliance with guidelines for application of the criteria? As discussed herein, using chart review in a "reliable and unbiased" fashion²⁷ is difficult given the limitations of physician documentation. Several studies avoided medical record review by using clinical data documented on laboratory requisitions,^{81 ,84 ,106 ,114 ,127} although none of these studies validated these data. Despite the potential problems with medical record review, only 4 studies used 2 or more reviewers and compared their abstraction results. None of the studies determined the effect of uncertainty on study results with sensitivity analysis. These issues are less of a concern when objective criteria are used.

Are the results of these utilization reviews generalizable? Almost all the studies took place in teaching centers on hospitalized patients and analyzed the ordering practices of a small number of physicians-in-training. This represents a very select patient and physician group. Hospitalized patients are less healthy than the ambulatory population and laboratory-requesting patterns for them may not represent those of ambulatory patients. Physicians in training have less experience in diagnostic strategies and may order tests less efficiently than established practitioners. Since laboratory use is dependent on both patient and physician factors, it is necessary to research other clinical settings to determine the extent of inappropriate laboratory utilization. This could be done by using the criteria that are applicable with data potentially found within laboratory and administrative databases.^{82 ,84 ,98 ,101 ,103 ,105 ,119 - 121} Certainly, allusions to extensive inappropriate laboratory use^{128 - 129} should not be made without appropriate qualifiers.

This review has several limitations that should be mentioned. First, the study deliberately excluded all radiology, pathology, and screening tests. Determining how well inappropriate utilization is measured in these areas would merit future reviews. Second, quality of study design and quality of reporting can be distinct. It is possible that the reliability of some studies was compromised when criteria for definitions or comparisons among raters were edited from the manuscripts. Finally, the diverse nature of this topic makes it possible that some eligible studies were not identified with our search strategy. The observation that less than half of the studies were identified from the initial MEDLINE search increases this concern.

The studies in this review suggest a paradigm or model for classifying criteria that determine the appropriateness of laboratory use (Figure 2). Appropriateness criteria could be based on a direct comparison among laboratory tests or a single test in isolation. The former could be based on studies using methods suggested by Reid et al²⁶ and would compare tests based on their operating characteristics such as sensitivity, specificity, and likelihood ratios. The latter criterion, determining a test's appropriateness without another test for comparison, could be based on physiological, pharmacological, or probabilistic data. Physiological criteria would be based on our knowledge of disease processes, such as the criteria of Albright et al for VDRL testing in cerebrospinal fluid.⁹⁸ Pharmacological criteria would be based on known pharmacodynamic tenets, such as those by Pearce and Day⁸⁴ and Bussey and Hoffman.¹⁰³ Clinical probability-based criteria would identify scenarios in which test results would almost never be useful. This would occur when a test's operating characteristics (eg, positive or negative likelihood ratios) are too weak to significantly alter posttest probabilities. Finally, a test may be useless if its result is highly predictable. Cohort studies could be used to identify these situations.^{29 - 30 ,32 ,34 ,37 - 40 ,43 - 44 ,130} We hope that our model will serve as a framework for the development of appropriateness criteria for future studies and that these studies will concentrate first on establishing valid criteria for determining test appropriateness before it is measured.

Where should we go from here? Identifying inappropriate laboratory use is important and requires valid methods. Promising criteria for inappropriate laboratory use need to be validated and then applied to divergent populations of patients and physicians using methodologically sound studies.²⁷ Further research is necessary to measure test operating characteristics and methods of determining pretest probabilities accurately.^{131 - 132} When inappropriate laboratory utilization is identified, methods to rectify the problem must be developed, tested, and implemented.^{133 - 134} Although appropriate laboratory testing does not necessarily translate into appropriate utilization of test results,¹³⁵ it is an important step toward optimal care of the patient and better performance of the health care system.