Retrospective Drug Utilization Review, Prescribing Errors, and Clinical Outcomes FREE

Retrospective drug utilization review programs are structured ongoing initiatives that interpret patterns of drug use in relation to predetermined criteria and attempt to minimize inappropriate prescribing.¹ The 1969 Task Force on Prescription Drugs (convened to consider a Medicare drug benefit) concluded that although the idea held promise, evidence of effectiveness was needed before widespread adoption.² This recommendation was not heeded, however, and retrospective drug utilization review has since been required of all Medicaid programs and implemented by most private-sector drug benefit programs. It has also been included in recent proposals for a Medicare drug benefit.

A typical retrospective drug utilization review process is as follows. Each month, computers screen claims data for criteria violations. Criteria are rules or expectations to which actual practice is compared. For example, a typical criterion is that a given patient should not receive more than 1 opioid analgesic at a time. Violations of criteria are known as "exceptions." If the criteria are valid, then exceptions represent prescribing errors. Exceptions are reviewed manually to determine if a physician alert should be issued. Alerts are typically conducted by mail, although a few programs do so by telephone. Alert letters typically include the name of the patient, an introduction to the retrospective drug utilization review program, text describing the criterion that has been violated (sometimes with a literature reference supporting the validity of the criterion), a line listing of the prescriptions and diagnoses that constitute the exception, and a statement that the clinical care of individual patients is at the discretion of the physician.

Retrospective drug utilization review programs are hypothesized to work through 2 mechanisms: direct effects and spillover effects.³ Direct effects apply to patients who are identified in alerts and benefit from a change in therapy. Spillover effects refer to the possibility that prescribers might apply lessons learned from receiving an alert to the care of future patients. Although a few studies have shown that drug utilization review programs can modestly reduce "undesirable" prescribing for identified patients (eg, reducing long-term use of histamine H₂ receptor antagonists⁴ ), the only study to look for clinical benefit has not appeared in the peer-reviewed literature, and it found no effect.⁵ The available studies examining spillover effects have produced conflicting results.⁶ Continuation of such programs despite a lack of evidence for their effectiveness has led some to suggest that they are a "boondoggle."¹

The aims of this study were to look for an effect of retrospective drug utilization review on the rate of exceptions, as might be caused by spillover effects; to look for a direct effect of retrospective drug utilization review in reducing adverse clinical outcomes in patients with exceptions; and to report the number of alerts made by these programs, which has not been reported in the scientific literature yet is important in interpreting the results of the first 2 aims.

This study was performed using data from 6 Medicaid programs, which used the same retrospective drug utilization review software vendor (ProVantage). We studied Medicaid programs using this product because it is commonly used, because the vendor and its Medicaid clients expressed interest in conducting a rigorous evaluation, and because the necessary archived data were available. We used all data available from the vendor prior to review implementation and had more than 1 year of data following implementation in the last state to start. The dates for which data were available for each state are shown in Figure 1.

Medicaid data are used extensively for epidemiologic and policy research.^{7 - 8} Information on dispensed prescriptions is audited and has been shown to be highly accurate.^{9 - 10} Although diagnoses are generally not recorded with the same degree of accuracy as are dispensed prescription drugs, many diagnoses have been found to be sufficiently accurate to use for research purposes.^{9 - 10} Extensive quality assurance analyses were performed on the data for this study.¹¹

This study was approved by the University of Pennsylvania's Committee on Subjects Involving Human Beings. The committee waived patient consent.

Identification of Exceptions. The vendor's software contains criteria to identify drug-disease interactions, drug-drug interactions, therapeutic duplication, excessive dose, insufficient dose, and excessive medication duration. We limited our study to drug-disease, drug-drug, and duplication criteria because these accounted for approximately 80% of alerts. We also limited the study to criteria that, according to the vendor, were used to identify exceptions in all 6 states for the entire intervention period. (These exceptions are listed at http://cceb.med.upenn.edu/cert/retrospective_dur_appendices.pdf.) Exceptions were identified retrospectively at the University of Pennsylvania using an archive of the same claims data that the programs themselves had used.

Analysis of Exceptions. For each state and for each month, we calculated the rate of exceptions per thousand prescriptions. To examine the effect of retrospective drug utilization review on the exception rate, we plotted the data visually, aligned so that implementation was considered month 0, with the preimplementation period considered as negative months and postimplementation period as positive months. We examined the plot for the presence of a negative inflection point at or following implementation, which would be indicative of a reduction in the rate of exceptions associated with retrospective drug utilization review.

Linear regression was used to calculate effect estimates and inferential statistics. We predicted a priori that any spillover effects of retrospective drug utilization review would be observed as a change in slope in the rate of exceptions vs time coincident with implementation. Therefore, we modeled the monthly rate of exceptions as a linear function of state (coded as indicator variables), calendar time in months, and retrospective drug utilization review intervention status. We also included state x time interaction terms to allow the preintervention slope to vary by state. Finally, we included an intervention x time interaction term as the parameter of interest. This term is interpretable as the before vs after intervention difference in slope of the rate of exceptions, assuming a common effect across states.¹² The primary analysis assumed a 2-month lag between implementation and onset of effect.

We also performed secondary analyses assuming lags of 1 and 4 months. We conducted a secondary analysis that adjusted for autocorrelation by fitting a generalized linear model with an autoregressive correlation matrix.¹³ The results of this analysis were very similar to those of the primary analysis (data not shown). Finally, we also performed a secondary analysis using Poisson regression,¹⁴ which gave similar results (data not shown). This was expected because the Poisson distribution is well approximated by the Gaussian distribution when the number of outcomes is large.

Identification of Eligible Patients. We studied individuals for whom an exception occurred, regardless of whether an alert was sent. This was performed to permit identification of a valid comparison group for before-after comparisons because the process of flagging exceptions for alerts is done by individuals using implicit criteria and therefore is not reproducible. Exceptions occurring after retrospective drug utilization review was implemented were considered the exposed group; those occurring before implementation were considered the unexposed group. Because of the incompleteness of Medicaid claims for those aged 65 years or older,¹¹ we also performed subanalyses that excluded this group. The primary analysis examined exceptions to the entire set of study criteria. We also examined the 5 criterion subtypes that resulted in the highest number of alerts in the states included in this analysis. This was considered a "high-dose" subgroup in which an effect might be more likely.

Determination of Outcome. All-cause hospitalization was the primary outcome. Therefore, we excluded from the individual-level analyses 3 states for which we could not distinguish inpatient from outpatient claims.

We also performed analyses of cause-specific hospitalization following specific exceptions. In particular, we examined the incidence of hospitalization for upper gastrointestinal bleeding following exceptions to gastrointestinal criteria (warfarin + gastrointestinal acid/peptic disorders, warfarin + aspirin, and NSAID + NSAID [nonsteroidal anti-inflammatory drug]); hospitalization for myocardial infarction following β-agonist + angina pectoris exceptions; and hospitalization for angina pectoris following β-agonist + angina pectoris exceptions. These criteria were selected because they were commonly used as the basis of alert letters in the study states, because exceptions were expected to be common, and because the expected outcomes could be reliably identified using claims data.

The specific International Classification of Diseases, Ninth Revision codes used to identify cause-specific hospitalizations are available at http://cceb.med.upenn.edu/cert/retrospective_dur_appendices.pdf. Because of privacy concerns, medical records were not available to permit the validation of study diagnoses. However, the accuracy of administrative claims for gastrointestinal bleeding,^{15 - 16} myocardial infarction,¹⁷ and angina pectoris¹⁸ all exceed 85%.

The primary observation period was 120 days following each exception. Secondary analyses were performed using periods of 90 and 150 days following the exception.

We restricted the analysis to exceptions for which there was at least 1 medical claim after the 120-day observation window. This approach helps ensure that outcomes are not missed because of loss of Medicaid eligibility.¹⁹

Outcome Analysis. For each outcome, we first performed a contingency table analysis for the association between intervention status and all-cause or cause-specific hospitalization to determine whether the risk of adverse clinical outcomes was influenced by retrospective drug utilization review implementation. We then used logistic regression to examine the effects of potential confounding factors. State and calendar year were included in all models. We considered other potential confounding factors individually, including them in the final model if they changed the odds ratio of interest by 10% or more.^{20 - 21} These potential confounding factors included sex, age (categorized as <15, 15-34, 35-44, 45-54, 55-64, 65-74, and ≥75), number of physicians seen in the past 3 months, and the number of pharmacies dispensing medications in the past 3 months. We also examined as potential confounders drug classes included in the Chronic Disease Score²² (see http://cceb.med.upenn.edu/cert/retrospective_dur_appendices.pdf).

Federal law requires each Medicaid program to provide the Centers for Medicare & Medicaid Services (CMS) with an annual report of retrospective drug utilization review activities.²³ These reports typically include the number of alerts per drug class, per criterion type (eg, the number of duplication alerts for opioid analgesics) for the year of interest. We requested annual reports from the coordinator of each study state during the study period, as well as from CMS. We used these reports to calculate the average annual number of alerts to drug-disease, drug-drug, and duplication criteria in each state.

The statistical analysis was performed using SAS version 8.1 (SAS Institute Inc, Cary, NC). P<.05 was set as the level of significance.

Table 1 lists the characteristics of the study states. The study sample included states from all 4 geographic regions of the United States and both large and small Medicaid programs. The timing of the available data with respect to state, calendar time, and implementation is shown in Figure 1.

Figure 2 shows the combined monthly exception rate per 1000 prescriptions, stratified by state, with implementation positioned at month 0. Overall, there was no visually apparent decline in the exception rates coincident with retrospective drug utilization review implementation. This observation was confirmed by the results of the regression models. In particular, in the primary analysis that assumed a 2-month lag, the slope of exceptions over time showed an increase, rather than a decline, in exceptions associated with retrospective drug utilization review of 0.064 exceptions per 1000 prescriptions per month, which was consistent with random variation (95% confidence interval [CI], -0.006 to 0.133). There was also no apparent decline in the slope using a 1-month lag (slope increase, 0.053 exceptions per 1000 prescriptions per month; 95% CI, -0.015 to 0.121) or a 4-month lag (slope increase, 0.001 exceptions per 1000 prescriptions per month; 95% CI, -0.067 to 0.070). A statistical test failed to reject the null hypothesis of a common effect across states (2-month lag, P = .31).

None of the subanalyses based on exception category showed a decline in slope coincident with retrospective drug utilization review. The slope for drug-disease exceptions increased by 0.015 exceptions per 1000 prescriptions per month (95% CI, -0.032 to 0.063); the slope for drug-drug exceptions increased by 0.006 exceptions per 1000 prescriptions per month (95% CI, -0.001 to 0.011); and the slope for duplication exceptions increased by 0.043 exceptions per 1000 prescriptions per month (95% CI, 0.011 to 0.075).

In the subanalysis of the 5 subgroups of criteria that resulted in the highest number of alerts (ie, the "high-dose" subgroup), the slope of the rate of exceptions associated with retrospective drug utilization review increased by 0.022 exceptions per 1000 prescriptions per month (95% CI, 0.009 to 0.034).

Table 2 summarizes the association between retrospective drug utilization review implementation and the incidence of all-cause and cause-specific hospitalization in those with an exception, using a 120-day observation window. All of the odds ratios were very close to 1.0. The results using 90-day and 150-day time windows were very similar (data not shown). Age, sex, and any of potential drug confounders did not change the odds ratios of interest by 10% or more. However, adjusting for these factors individually tended to increase rather than reduce the odds ratios associated with retrospective drug utilization review. Analyses restricted to those younger than 65 years produced results similar to the overall analysis (data not shown).

Table 3 shows the average annual number of prescriptions, exceptions, and alerts to drug-drug, drug-disease, and duplication criteria, with rate of exceptions per 1000 prescriptions, and number of alerts expressed as a percentage of exceptions. The average exception rate was similar among states, ranging from 8 to 13 exceptions per 1000 prescriptions.

The programs studied issued from 761 to 3236 alerts per year. This translates into an alert rate that ranged from 1% to 25% of exceptions. In the ecologic analysis, the alert rate per state was not a linear predictor of a change in slope of the exception rate coincident with retrospective drug utilization review implementation (P = .86), nor was a threshold evident.

We were unable to identify an effect of retrospective drug utilization review on the rate of exceptions in the population or on the risk of all-cause and cause-specific hospitalization among patients with an exception. These findings were borne out in subgroups with the highest number of interventions (ie, the high-dose groups). The proportion of exceptions resulting in alerts ranged nearly 25-fold from 1% to 25%, with smaller states having the highest rates.

This study has a number of strengths. It is the first study to examine the effects of retrospective drug utilization review programs on improving clinical outcomes in individuals with exceptions. It based inclusion on computerized criteria, avoiding lack of comparability inherent in manual methods. The ecologic analyses accounted for potential differences among states in the rate and slope of exceptions and for secular trends. Results were consistent across subgroups, including high-dose subgroups where one would expect to find the largest effect.

This study has limitations as well. The possibility of confounding cannot be ruled out in any nonrandomized study. We attempted to reduce this possibility by examining a broad range of pharmacologic variables as potential confounders, although we cannot be certain that there is no residual confounding. We ascertained outcomes using Medicaid data, which are known to provide an incomplete picture of care for those who are dually eligible for both Medicaid and Medicare.^{9 ,11} We have in large part examined the effect of this omission by performing subanalyses that exclude those aged 65 years or older, who constitute the majority of the dually eligible population. However, no analogous subanalysis could be conducted for those who are dually eligible because of a disability. Regardless, because this group comprises only 5% of the total Medicaid population, ^{24 - 25} the effect of incomplete outcomes in this group should be small. An additional limitation is that the results may not be generalizable to retrospective drug utilization review programs not included in the current study. However, because the overall structure of programs appears similar, dramatic differences in program effectiveness seem unlikely. Furthermore, although data for this study ended in 1996, a recent article suggests that the function of Medicaid retrospective drug utilization review programs has changed little since then.²⁶

The only previous study to examine clinical effects of retrospective drug utilization review has not appeared in a peer-reviewed publication and it also failed to detect an effect.⁵ One study has even suggested that retrospective drug utilization review programs could cause harm, such as discontinuation of drugs in patients who still need them, abrupt discontinuation of drugs that should be tapered, and use of undesirable alternatives.²⁷

Potential reasons for the apparent lack of effect of retrospective drug utilization review include the unknown validity of many criteria, the low alert rate, the time lag from the exception to the alert, and the modest effectiveness of alert letters in changing prescribing, particularly when letters are not based on any underlying reasons physicians may have had for prescribing the medication.²⁸

Given the lack of evidence for the effectiveness of retrospective drug utilization review, and the suggestion of possible harm, policymakers should consider withdrawing the legislative mandate for retrospective drug utilization review. This would agree with the recommendations of the 1969 Task Force on Prescription Drugs that drug review programs should not be implemented in a widespread manner until they are shown to be effective. It is possible that future drug utilization review programs can be improved to confer clinical benefits. However, programmatic efforts to improve prescribing should be shown to be effective and free of major detrimental effects before being widely adopted.