Improving Residents' Compliance With Standards of Ambulatory Care:  Results From the VA Cooperative Study on Computerized Reminders FREE

Randomized controlled trials of computerized reminders (CRs) to physicians to improve their compliance with specific standards of care (SOC) were first reported in 1976.¹ During the next 17 years, other randomized controlled trials^2- 21 were reported that compared the use of CRs with manual reminders or no reminders. In some of the trials,^11,16- 17 CRs were given to patients as well as physicians, but generally, CRs were given only to resident physicians.

Although the overall effects of the CRs were positive, there were serious limitations to the published studies. All studies were single-site trials and nearly all involved relatively few patients (eg, between 32 and 1460 patients, with the exception of 2 studies of 7000⁷ and 12,467¹⁷ patients). Since most of the computer systems were developed to meet the needs of a particular institution, there was little opportunity to extend the studies beyond the single sites. The number of conditions for which CRs were generated varied from 1 to 11, with the exception of 1 study of more than 100 conditions. Five studies involved 5 or fewer reminder conditions. All CRs were for preventive care such as influenza and pneumococcal vaccines or mammography. Two sites accounted for 11 of the previously published trials.^{1- 3,7- 9,11- 15}

The Department of Veterans Affairs centralized database and Cooperative Studies Program provided a unique opportunity to assess the generalizability and overall utility of using CRs to improve the quality of care involving multiple sites and using a host of different SOC. Specifically, we examined whether providing CRs of well-accepted SOC to resident physicians in ambulatory care clinics can increase compliance with those standards. This study also addressed (1) whether CRs can work in multiple sites around the country with the same database and (2) whether CRs can work with multiple reminders that include treatment as well as prevention interventions.

A total of 275 resident physicians from 12 Veterans Affairs (VA) medical centers participated in the study. Resident physicians were chosen for participation because they were the VA physicians who were most involved in patient primary care at the time of the study. During the course of the study, the residents cared for 18,700 unique patients, and 12,989 of these patients were eligible for at least 1 of the investigated SOC.

The SOC were chosen on the basis of their importance to the Department of Veterans Affairs ambulatory care patient population, a population composed mainly of middle-aged and elderly males with chronic diseases; and feasibility of identifying diagnoses, exclusionary factors, treatments, procedures, and instructions via the hospital computer system. In the planning stages of the study, a committee of Chiefs of Ambulatory Care Services in the VA generated a preliminary list of SOC using existing published medical guidelines and literature search strategies. More than 30 potential SOC were identified. The capability of the computer system to monitor the standards was then assessed to identify the 13 standards that were used in the trial (Table 1).

The study was a clinical trial comparing the performance of residents receiving CRs with the performance of residents not receiving CRs. Data collection for the study began January 31, 1995, and ended June 30, 1996. At each site, resident physicians were assigned to either the reminder group or the control group. For sites using a firm or team system, each firm was randomly assigned to 1 of the 2 groups, and all residents in the firm were assigned to that group. For sites not using a firm system, half-day blocks of residents were randomly assigned to the reminder or control groups. Firms or half-day blocks of residents, rather than individual residents, were randomized to reduce communication between members of the intervention and control arms. Also, a concerted effort was made at each site to have residents from the same firm or block substitute for one another when necessary to prevent contamination. A total of 153 residents were assigned to the control group, and 146 residents were assigned to the reminder group. A total of 143 residents in the control group and 132 residents in the reminder group completed the study. For the 24 residents who failed to complete the study, 12 had residencies that concluded before the completion of the study, 4 left their residencies prematurely, and 8 remained in their residencies but no longer wished to participate in the study. There was no significant difference in the drop-out rates between the treatment groups (P = .33). The 275 residents completing the study exceeded the target sample size for the study of 260 residents total.

At the beginning of the study, residents in both the reminder and the control groups were asked to complete a questionnaire exploring their knowledge of and attitudes toward the SOC being studied. Later, they attended a 1-hour instruction session where the principal investigator at each site discussed the rationale for and benefits of the SOC being studied. During the instruction session, all residents received a booklet that listed the SOC and provided the rationale and several references supporting each standard. Residents in the reminder group were also provided with an introduction to the reminder system. This consisted of an education session of 1 to 2 hours that explained the general value of reminder systems and the presentation of a videotape, designed specifically for this study, that demonstrated in detail how the reminder system worked.

Data were collected for at least 2 weeks before reminders were activated to provide a period for comparing baseline adherence rates between the reminder and control groups for a sample of residents' patients. Once the intervention period began, CRs were presented to residents in 2 ways to ensure that they would see each reminder prior to evaluating each patient. First, each examination room has a computer terminal that is connected to the hospital computer server. When a resident in the intervention group entered a patient name into the computer, all reminders pertaining to that patient were automatically presented in bold letters. Each reminder consisted of a notification that the SOC applied to the patient. Accompanying each notification was a brief rationale for the standard. Second, a computer-generated summary (typically 6-8 pages) of a patient's health, including a list of his or her medical conditions and a list of his or her most recent clinic visits, is routinely placed at the beginning of the medical chart on the day of the clinic visit. For the intervention group, this health summary was modified to include all reminders that pertained to the patient. This information appeared on page 2 of the health summary after patient-identifying information. Control group residents continued to receive standard health summaries without the reminders.

At no point in the study was any performance feedback given to the hospitals or the residents concerning individual resident adherence levels or overall hospital adherence levels. Accordingly, residents were never evaluated based on their compliance with the SOC.

Adherence to the SOC was the primary dependent variable, and adherence was measured for all patients visiting a study physician whom were eligible for 1 or more of the SOC. Adherence was measured in 2 different ways: general adherence or visit-specific adherence. General adherence used a patient's last visit as a reference point and measured the proportion of patients with 1 of the SOC who were in compliance with that SOC at a time point following the patient's last visit. This measure did not determine the specific point in time during the study when the care was provided, but only whether care had been provided by their last visit.

Visit-specific adherence, on the other hand, measured whether residents delivered appropriate care at the time of a specific patient visit by providing the care according to the SOC. This was accomplished by tracking adherence both before and after each visit. Before each visit, it was determined, based on review of prior care, whether a patient needed to receive specific services. The adherence measurement after the visit then identified whether residents provided care according to the SOC. This measure allowed multiple observations in the database for the same patient and SOC combination if a patient visited the clinic repeatedly and the suggested care had not been provided at the time of each visit.

For both measures, adherence was treated dichotomously, with adherence equal to 1 if proper care had been given to the patient and to 0 if proper care had not been given. An adherence measurement was included in the database for all SOC for which the patient was eligible.

Table 1 provides a list of the 13 SOC that were studied. Eligibility and compliance with the various SOC were determined using information from 2 sources: (1) an encounter form completed at the time of each visit and (2) each hospital's computer system. The encounter forms were optical scan forms that contained a list of 152 diagnoses, 20 procedures, and 17 patient instructions, such as a discussion of the benefits of exercise or proper nutrition. The diagnoses, procedures, and instructions listed on the encounter form were those that were most likely to be reported in the VA ambulatory clinics. Residents were instructed to mark each diagnosis that was treated, each procedure that was administered, and each instruction that was given. There were also spaces on the encounter form to enter "other" diagnoses, instructions, or procedures.

For this study, computer software was developed that downloaded the information from the encounter forms and integrated this information with hospital information about patients' demographics, prescriptions, treatments, and laboratory test results. The program then identified all eligible patients and determined whether proper care had been given to those patients. This program was run weekly at each site.

Several actions were taken to ensure that adherence was being tracked accurately. First, prior to data collection, each of the SOC was individually checked at the 12 sites by a data evaluation committee who sampled cases and checked the accuracy of the integration program with patient charts and hospital files. Because of the considerable time it took the Data Evaluation Committee to check the computer program for each SOC at each site, the exact timing of data collections varied for the different SOC within and among sites.

Second, at each site, monthly completion rates were determined to guarantee that residents were using the encounter forms. The average completion rate across all sites was 96.9% (range, 92.2%-100%).

Finally, for roughly 5% of the encounter forms that were completed each month, audits were conducted to check the accuracy of the marked information. Specifically, these audits assessed the concordance between information marked on the encounter form and information written in the medical chart. Most of the diagnoses (76.9%) were present both in the medical chart and on the encounter form. It was expected that some diagnoses would not be found on the encounter forms because the residents were instructed to only mark diagnoses that they treated at that visit.

Statistical analyses were based on the sample of 12,989 unique patients who were eligible for 1 or more of the SOC studied. An α level of .05 or 95% confidence interval (CI) was used for all statistical analyses (exact P values are also reported).

The dependent measure of interest in the study was adherence to the SOC. Because adherence is potentially influenced by both patient and physician variables, multilevel logistic regression was used to analyze adherence as a dichotomous variable at the patient level while accounting for the clustering of patients within resident physicians. This was accomplished using generalized estimating equations.²² For visit-specific adherence, multilevel logistic regression analyses that accounted for the clustering of visits within patients were used to test the statistical significance of the differences between the groups.

The mean age of the resident physicians at the start of the study was 28.4 years (SD = 3.12 years). There were more male residents (70.9%) than female residents (29.1%), and 17.5% of the residents graduated from foreign medical schools. As part of the knowledge and attitudes survey that was given prior to the study, residents used a 5-point scale (5 = highest) to rate for each SOC their level of agreement with the standard, the level of research evidence supporting the standard, and the feasibility of implementing the standard at their hospital. The overall mean ratings for the standards were 4.3, 4.0, and 3.6 for agreement, research support, and feasibility, respectively.

The mean age of the patients was 65.9 years (SD = 10.9 years), and 98.4% were male. There were no statistically significant differences in physician or patient characteristics at baseline between the reminder and control groups. There were also no significant differences in baseline general adherence between the control group and the reminder group for all SOC combined (55.3% vs 53.0%) and for 11 of the 13 SOC examined individually (Table 2). There was wide variation in the baseline adherence rates for the different standards, ranging from 4.4% for pneumococcal vaccination to 79.1% for atrial fibrillation-warfarin, aspirin, or ticlopidine.

Table 3 presents adherence rates for the reminder and control groups during the intervention period. The general adherence method was used. These rates represent the proportion of patients seen by a study physician during the intervention period who were in compliance with the SOC by their last visit. The results show that the reminder group had a statistically significantly higher adherence rate (58.8%) compared with the control group (53.5%) for all SOC combined (odds ratio [OR], 1.24; 95% CI, 1.08-1.42). Significantly higher adherence rates were also found for 5 of the 13 SOC analyzed individually, and another 6 SOC showed nonsignificant differences favoring the reminder group. The largest effect was for pneumococcal vaccination (12.7% vs 4.3%; OR, 3.26; 95% CI, 2.09-5.09).

The change in adherence rates for all SOC combined from the baseline to the intervention period favored the reminder group at 11 of the 12 participating sites. The reminder group showed statistically significantly greater improvement in adherence compared with controls (P≤.05) at 4 of the 12 sites.

Table 4 shows a comparison of residents' visit-specific adherence in the control and reminder groups for all SOC combined and for each SOC individually. As was previously shown for general adherence, the reminder group had a significantly higher rate of visit-specific adherence (17.9%) than the control group (12.2%) for all SOC combined (OR, 1.57; 95% CI, 1.45-1.71) and for 9 of the SOC examined individually. Large effects were found for pneumococcal vaccine (OR, 7.85; 95% CI, 3.83-16.08), diabetes/peripheral vascular disease-foot examination (OR, 2.57; 95% CI, 2.02-3.26), and for diabetes-eye examination (OR, 2.19; 95% CI, 1.63-2.94). The 9 SOC that showed significant benefits of reminders were all activated in the early phase of the study.

Figure 1 plots the rates of visit-specific adherence across the course of the study for the 9 SOC that were initiated early in the study. The results show that residents' responsiveness to the reminders decreased throughout the study. This treatment by time interaction was statistically significant (P < .001). Simple effect analyses showed that adherence rates significantly declined across time (P<.001) for the reminder group, but the adherence rates for the control group were unaffected by time (P = .16).

The purpose of this VA cooperative study was to test the hypothesis that CRs of well-accepted SOC can effect an increase in residents' compliance with multiple standards of ambulatory care across multiple medical centers. In addition to involving multiple centers and concurrently using 13 SOC, this clinical trial was unique in involving both preventive measures as well as treatment for specific diagnoses, having a large sample size, measuring adherence using both general and visit-specific analyses, and in using a lengthy intervention period that allowed the tracking of adherence rates over time.

The trial furthered previous research by demonstrating higher adherence with the reminder system across 12 medical centers for the combined 13 SOC. The general adherence rates increased from an already fairly high 53.0% at baseline to 58.8% during the intervention period for the reminder group (a 10.9% relative increase or a 5.8 percentage point absolute increase), while the control group showed a slight decrease from 55.3% adherence at baseline to 53.5% adherence in follow-up (a 3.3% relative decrease or a 1.8 percentage point absolute decrease). However, the increase in the general adherence rate in the CR group was not as large as that reported by some other studies.²³

Visit-specific adherence rates in both the reminder and control groups were low. Residents' failure to deliver the suggested care more regularly may be related to busy clinic schedules combined with a belief that the standards may not have an immediate impact on patients' health. The standards with the lowest adherence rates tended to be prevention rather than treatment oriented, such as smoking cessation, counseling, and pnuemococcal vaccination. Still, the fairly high number of patients in compliance with the standards by the time of the patient's last visit, as indicated by the general adherence measure, shows that patients' compliance with the standards is determined by more than residents' actions at any single ambulatory care visit. Patients often attend the ambulatory clinics multiple times in a year, giving physicians several opportunities to provide the recommended care, and in many instances, patients may receive the suggested care as an inpatient or as part of larger hospitalwide patient education efforts.

This study demonstrated that improvements in adherence could be realized for multiple standards concurrently across multiple sites. However, enthusiasm for the reminder system was tempered by the finding that physicians became less likely to respond to reminders across the time course of the study. Previous studies have shown sizable decrements in SOC compliance during wash-out periods that followed reminder deactivation. We showed a similar decrement, in spite of maintaining the reminders.

We observed that the initial positive response to CRs systematically declined across the course of the study even though the reminders remained active. There are several potential reasons for this finding. One possibility is that external changes in the practice of medicine lead to the residents discounting the reminders. However, no similar decrement in performance was observed for the control group, which would be expected if residents began questioning the standards. Another more likely explanation is that in a busy clinic the competing demands on residents' time lead to inattention to the reminders.

A lack of feedback concerning residents' performance may have contributed to the observed decline. Perhaps providing performance feedback^24- 26 and/or educational reinforcement to residents would help sustain the positive effects of reminders. Nonetheless, our data indicated that computer reminders installed at multiple sites can improve compliance with multiple SOCs.