0
We're unable to sign you in at this time. Please try again in a few minutes.
Retry
We were able to sign you in, but your subscription(s) could not be found. Please try again in a few minutes.
Retry
There may be a problem with your account. Please contact the AMA Service Center to resolve this issue.
Contact the AMA Service Center:
Telephone: 1 (800) 262-2350 or 1 (312) 670-7827  *   Email: subscriptions@jamanetwork.com
Error Message ......
Review |

Effect of Patient Reminder/Recall Interventions on Immunization Rates:  A Review FREE

Peter G. Szilagyi, MD, MPH; Clayton Bordley, MD, MPH; Julie C. Vann, PhD, MS, RN; Ann Chelminski, MD; Ronald M. Kraus, EdM; Peter A. Margolis, MD, PhD; Lance E. Rodewald, MD
[+] Author Affiliations

Author Affiliations: Department of Pediatrics, University of Rochester School of Medicine and Dentistry, Rochester, NY (Dr Szilagyi and Mr Kraus); the Children's Primary Care Research Group, Department of Pediatrics (Drs Bordley, Vann, Chelminski, and Margolis), and the Public Health Leadership Program, School of Public Health (Dr Vann), University of North Carolina, Chapel Hill; and the National Immunization Program, Centers for Disease Control and Prevention, Atlanta, Ga (Dr Rodewald).


JAMA. 2000;284(14):1820-1827. doi:10.1001/jama.284.14.1820.
Text Size: A A A
Published online

Context Immunization rates for children and adults remain below national goals. While experts recommend that health care professionals remind patients of needed immunizations, few practitioners actually use reminders. Little is known about the effectiveness of reminders in different settings or patient populations.

Objectives To assess the effectiveness of patient reminder systems in improving immunization rates, and to compare the effectiveness of different types of reminders for a variety of patient populations.

Data Sources A search was performed using MEDLINE, EMBASE, PsychINFO, Sociological Abstracts, and CAB Health Abstracts. Relevant articles, as well as published abstracts, conference proceedings, and files of study collaborators, were searched for relevant references.

Study Selection and Data Extraction English-language studies involving patient reminder/recall interventions (using criteria established by the Cochrane Collaboration) were eligible for review if they involved randomized controlled trials, controlled before-after studies, or interrupted time series, and measured immunization rates. Of 109 studies identified, 41 met eligibility criteria. Studies were reviewed independently by 2 reviewers using a standardized checklist. Results of studies are expressed as absolute percentage-point changes in immunization rates and as odds ratios (ORs). Studies with similar characteristics of patients or interventions were pooled (random effects model).

Data Synthesis Patient reminder systems were effective in improving immunization rates in 33 (80%) of the 41 studies, irrespective of baseline immunization rates, patient age, setting, or vaccination type. Increases in immunization rates due to reminders ranged from 5 to 20 percentage points. Reminders were effective for childhood vaccinations (OR, 2.02; 95% confidence interval [CI], 1.49-2.72), childhood influenza vaccinations (OR, 4.25; 95% CI, 2.10-8.60), adult pneumococcus or tetanus vaccinations (OR, 5.14; 95% CI, 1.21-21.78), and adult influenza vaccinations (OR, 2.29; 95% CI, 1.69-3.10). While reminders were most effective in academic settings (OR, 3.33; 95% CI, 1.98-5.58), they were also highly effective in private practice settings (OR, 1.79; 95% CI, 1.45-2.22) and public health clinics (OR, 2.09; 95% CI, 1.42-3.07). All types of reminders were effective (postcards, letters, and telephone or autodialer calls), with telephone reminders being most effective but costliest.

Conclusions Patient reminder systems in primary care settings are effective in improving immunization rates. Primary care physicians should use patient reminders to improve immunization delivery.

Figures in this Article

Immunization rates for children and adults are rising throughout the United States, but coverage levels have not reached national goals. In 1998, coverage levels for children aged 19 to 35 months were 79% for the combined vaccine series of 4 diphtheria-tetanus-pertussis, 3 polio, 1 measles-containing vaccine, and 3 Haemophilus influenzae type b; 87% for 3 hepatitis B; and 43% for varicella.1 Coverage levels for adults are lower; in 1997, only 65% of adults aged 65 years or older received the influenza vaccine and only 45% had ever received pneumococcal vaccine.2 Furthermore, immunization coverage levels are not evenly distributed, with lower rates occurring among impoverished populations3 and some primary care practices.4 As a result of low immunization rates, vaccine-preventable diseases still occur, as evidenced by the measles epidemic during 1989-1991; the large number of annual cases of varicella, pertussis, and hepatitis B; and the thousands of annual deaths among adults from influenza and pneumococcal infections. Concern also exists that incorporation of new vaccinations will be slow, as evidenced by the slow uptake of varicella vaccine.5

In an era of increasing complexity of immunization schedules, increasing expectations about the performance of primary care, and large demands on primary care physicians, it is important to understand and promote interventions that work in primary care settings. Recent reviews have identified several promising strategies to improve immunization rates.610 One strategy recommended by the Task Force on Community Preventive Services9 and the Standards for Immunization Practices11 involves patient reminder/recall systems.

Unfortunately, few primary care physicians actually use reminder/recall systems.12 Because many patients cannot remember the recommended immunization schedule, the burden falls on primary care physicians to ensure that their patients receive immunizations on a timely basis. Recently, the burden on the private sector has increased as more patients have begun to receive immunizations at their primary care physician's office rather than at health department immunization clinics.13

If experts are recommending reminder/recall systems and individual studies are demonstrating their effectiveness, why aren't these systems used more frequently in primary care settings? Several factors may impede their incorporation. First, health care practitioners may not perceive that individual studies apply to their own practices. Pediatricians may not focus on studies involving elderly adults, and internists may not be aware of studies involving children. Some studies have been performed in public health department clinics or academic teaching hospital clinics, and private practitioners may not think that findings from such studies can be applied to their settings. Furthermore, some vaccinations are given only once, while others require multiple booster doses, making it more difficult to extrapolate findings from individual regimens to all vaccinations.

A second barrier is that recommendations about reminder/recall systems have not been very specific.10,11 Patient reminders can be delivered by a variety of methods (eg, telephone, mail) and in different levels of intensity (eg, single or multiple reminders). The most useful recommendations are those that are specific enough to be applicable in real-world settings by large numbers of practitioners. A third barrier is that many primary care practices have lacked the computerized technology to track their patients' immunization status. Cost barriers may have also impeded use of computerized tracking and reminder systems. However, recent advances in billing systems and computerized immunization registries14 are making such technology attainable and affordable for a growing number of primary care practitioners.

We systematically reviewed the literature for studies of patient reminder/recall systems to assess their overall effectiveness and to delineate particular systems and situations that appear to be most effective in improving immunization rates. The study objectives were to (1) assess the overall effectiveness of patient reminder/recall systems in improving immunization rates; (2) compare the effectiveness of reminder/recall systems among populations that varied by baseline immunization rates, age, primary care settings, or vaccination schedules; and (3) compare the effectiveness of different types of reminder/recall interventions (eg, postcard, letter, telephone), and frequency of prompts (eg, single or multiple).

We followed the methodological review criteria established by the Cochrane Collaboration.15 Specific formats of the reviews can be found in the Cochrane Reviewers' Handbook,16 and an electronic publication containing detailed information about each study reviewed by this project is forthcoming as part of the Cochrane Database of Systematic Reviews.17

Study Selection

We sought studies involving interventions that reminded patients of immunizations that were due or immunization visits that were upcoming (reminders) or immunizations that were overdue (recall). Reminder/recall systems could be delivered by letter, postcard, telephone, autodialer (a computerized telephone dialer programmed to generate multiple telephone calls during a short time), or in person. Reminder/recall cues could also vary in their specificity (generic or patient-specific) and in their frequency (single or multiple).

The key outcome measure was immunization rates, or the proportion of the target population that was up-to-date on recommended immunizations. We included studies with outcomes for either individual vaccinations or standard combinations of recommended vaccinations (eg, all recommended vaccinations by a specific date or age).

Interventions that involved physician reminders, such as medical chart or computer prompts, were not evaluated unless they were used in combination with patient reminders. Studies with these combined interventions were analyzed separately from studies evaluating only patient reminders.

Three study designs were eligible for review: randomized controlled trials, controlled before-and-after studies, and interrupted time series studies. Studies had to meet initial published design criteria to be eligible for full review.

Search Strategy

A search was performed using the following bibliographic search engines: MEDLINE, EMBASE, PsychINFO, Sociological Abstracts, and CAB Health Abstracts; all databases were searched from their inception dates through 1998. Most studies were identified using MEDLINE. Search terms included the following text words and Medical Subject Headings: remind$, track$, autodial$, postcard$, mail$, recall$, telephone$, registry$, registries, reminder systems, appointments & schedules, appointment$, information systems, computers, immunization, immuniz$, immunization programs, vaccination, vaccin$, innoculat$, prevention health services, diphtheria, tetanus, whooping cough, poliomyelitis, polioviruses, haemophilus, influenza, measles, mumps, rubella, hepatitis b, pneumococcal infections, vaccines, tetanus toxoid, and diphtheria toxoid.

Two authors (P.G.S. and R.M.K.) reviewed the lists of titles and abstracts and used the inclusion criteria to select potentially relevant articles for full review. The reference lists of all relevant articles and reviews were back-searched for additional studies. Publications of abstracts, proceedings from scientific meetings, and files of study collaborators were also searched for references.

Inclusion Criteria

Articles were reviewed if they (1) included a patient reminder/recall system in at least 1 study arm; (2) reported primary research; (3) studied common nationally or internationally recommended childhood or adult vaccines (unusual vaccines or vaccines for travelers were excluded); (4) provided immunization coverage data; and (5) were written in English. Studies examining the effect of patient reminder/recall systems on other preventive services were included only if they reported on immunization rates separately.

Data Abstraction and Review

Each study was read independently by 2 reviewers (P.G.S. and J.C.V.). Reviewers were not blinded to authors (a recent study found no significant bias associated with such nonblinding18). Disagreements between reviewers were resolved by a formal reconciliation process to achieve consensus. Data abstraction was performed using a validated checklist developed by the Cochrane Collaboration Effective Practice and Organization of Care Group.19 For each included study, information was collected on the method of randomization or assembly of control groups, blinding, characteristics of subjects, setting and nature of the interventions, and results. Numerous quality criteria were assessed for each study design. For randomized controlled trials, which were the majority of included studies, assessment criteria included concealment of allocation, proportion of participants followed up, blinded assessment of primary outcome measures, documentation of baseline data, reliability of outcome measures, and protection of contamination between study groups.

The primary outcome measures were the percentage of patients who were immunized at the end of the study and the difference, in absolute percentage points, in immunization rates between groups receiving a reminder/recall vs control groups.

We were interested in both the overall effectiveness of patient reminder/recall and the relative effectiveness for key subgroups defined by patient age (child or adult), practice setting (academic medical center–based clinic, public health department clinic, or private practice), dates of study (before 1980, 1980-1989, or 1990-1998), type of vaccination (universal, such as all routine childhood vaccinations; or targeted, such as influenza for high-risk patients with specific chronic diseases), type of reminder/recall intervention (postcard, letter, telephone, autodialer, or combination), and frequency of intervention (single or multiple).

Analysis

Results are presented for individual studies as absolute changes in immunization rates (eg, >20-percentage-point increase in intervention vs control group rates), rather than relative rates, to allow for comparisons among studies. When possible, odds ratios (ORs) for being up to date or having received vaccinations are shown for intervention vs control patients. Studies were also subgrouped according to the key characteristics described herein. For each subgroup and for all studies combined, summary ORs were obtained using Review Manager, the computer program for analyzing Cochrane Reviews.16

We analyzed the study results as a funnel plot of the effect of reminder/recall against the sample size (which we used as a proxy for study precision since variance in immunization rates was often not available). If publication bias existed, we expected that more precise studies would be more likely to cluster around null results.20 Such clustering was not found, suggesting that positive findings of published studies were not due to publication bias.

Initially pooled results, weighted by the sample size of each study, were calculated using a fixed-effects model. Pooled weighted results were also generated for reminder type, patient age, and major vaccine category. Heterogeneity of the results of individual studies combined for each subgroup comparison was tested using a χ2 distribution with a .10 level of significance. Because heterogeneity of the results was present for overall results and within several subgroups, pooled results were ultimately computed using a random-effects model for all comparisons, with studies sorted by key characteristics. The random-effects models had wider 95% confidence intervals (CIs) than the fixed-effect models for all analyses, producing more conservative estimates of the effects of the interventions. In addition, a qualitative analysis (examining the strengths, weaknesses, and unique characteristics of each study) was performed to assess possible factors leading to heterogeneity of results.

Literature Search Results

Ninety-two studies were identified by the literature search. Seventeen additional studies were found by back-searching. Of the total 109 studies, 41 met eligibility criteria and were included in the final review (some studies had more than 1 study arm). The majority of excluded articles lacked a control group, had a study design that did not fit Cochrane criteria (eg, descriptive or ecological studies), were reviews instead of studies, or used an outcome measure other than immunizations (eg, preventive visits or services).

Overall Impact of Reminder/Recall Interventions

We categorized studies into 4 groups according to age (children or adults) and type of immunization: (1) routine childhood immunizations (15 studies2135); (2) childhood influenza vaccinations that target a high-risk group of children rather than the entire child population (2 studies36,37); (3) adult pneumococcal or tetanus immunizations (7 studies3844); and (4) adult influenza vaccinations (21 studies38,39,41,42,4561). Four evidence tables, including summaries of each study, are available from the author by e-mail and will be available on a forthcoming Cochrane review.17

Reminder/recall systems were found to be effective in 33 (80%) of the 41 studies and were generally effective for both children and adults and for both routine immunizations and targeted influenza immunizations. For routine childhood immunizations, 12 of 15 studies found positive effects, with the improvement in immunization rates ranging from 6 to 34 percentage points, and with ORs generally in the range of 1.5 to 2.5 for intervention vs control groups. The 2 studies on childhood influenza immunizations found similar improvements (>20 percentage points) in influenza immunization rates from very low baseline rates of controls. Six of 7 studies of adult pneumococcal or tetanus immunizations reported significant improvements, ranging from 4 to 27 percentage points, with most improving by at least 20 percentage points. Among the 21 studies of adult influenza immunizations (4 of these also studied pneumococcus or tetanus), 5 studies reported no improvement and 16 studies found significant improvements, often greater than 20 percentage points for patients receiving a reminder/recall intervention.

One might expect that improvement in immunization rates would be easier to achieve at lower baseline levels, with diminishing returns at higher levels; however, this was not found. In general, the degree of improvement in immunization rates due to reminder/recall was not associated with baseline immunization levels, which ranged from nearly 0% to 86%, as measured by control groups or assessed at the start of studies. Baseline rates were more than 80% in only a few cases and for influenza vaccine were extremely low.

Effectiveness of Reminder/Recall According to Patient Characteristics

Table 1 shows pooled results of the effectiveness of reminder/recall systems for randomized controlled trials, by patient age and vaccine group, study setting, and decade of study. Reminder/recall was effective for both children and adults, for both universally recommended routine childhood vaccinations and influenza vaccination recommended for high-risk children, and for all types of adult vaccinations. Odds ratios were generally higher than 2.0 for intervention vs control group immunization rates. Reminder/recall was most effective in academic settings, with somewhat lower but still positive results in private practice settings. There was no clear trend by decade of study.

Table Graphic Jump LocationTable 1. Effectiveness of Patient Reminder/Recall by Patient Characteristics
Effectiveness of Different Reminder/Recall Systems

Table 2 shows pooled results for randomized controlled trials comparing the effectiveness of different types of reminder/recall systems for routine vaccination of preschool children, child influenza vaccination, adult influenza vaccination, and other adult vaccines. Figure 1 shows ORs for immunization rates for 6 different reminder/recall systems. All types of reminder/recall systems appeared to improve immunization rates, with ORs generally between 1.5 and 2.5 and improvement in immunization rates of 5 to 20 percentage points. Telephone reminders appeared to be more effective than other types of reminders, while letter reminders generally did not appear to have an advantage over postcard reminders. The few studies that evaluated patient reminder/recall combined with physician prompts found results that were similar or slightly better than that of studies using only patient reminder/recall.

Table Graphic Jump LocationTable 2. Effectiveness of Different Types of Reminder/Recall Systems for Children and Adults
Figure. Effectiveness of Different Types of Patient Reminder/Recall Systems for All Ages Combined
Graphic Jump Location
OR indicates odds ratio; CI, confidence interval. Error bars indicate 95% CIs.

Some studies used single patient reminders while others used 2 or more reminders. For the 31 randomized clinical trials that included at least 1 study arm with single patient reminders, the OR for being up to date at the end of the study for intervention vs control groups was 2.18 (95% CI, 1.75-2.71), while for the 9 trials that had multiple reminders, the OR was 2.82 (95% CI, 1.57-5.06).

Costs of Reminder Systems

Fifteen studies reported on costs, including 8 pediatric studies22,23,26,27,3133,35 and 7 studies of adults.40,41,44,48,57,59,61 Eight studies estimated the cost-effectiveness of reminder/recall systems.22,26,27,32,35,41,48,61 Costs varied widely across studies due to (1) variability in methods of calculating costs and items included in analyses (such as existing staff or computer programming); (2) different types of reminders used, with telephone reminders being more costly than letter or postcard reminders, (3) different levels of intensity of interventions, from single postcard reminders to repeated reminders plus home visits, and (4) different study periods. Single reminders were less costly than multiple reminders but also were less effective. Costs of reminder systems used throughout a year (as in many pediatric studies) were more expensive than short-term reminders typical of influenza vaccination studies. Because of different study methods, it was not possible to combine results of costs. In several studies, the costs per patient were less than $1, particularly in short-term studies involving mailed reminders and excluding computer or study design costs.27,44,48,57,59 In the few studies that estimated cost-effectiveness, the estimated cost per additional patient vaccinated varied widely, from less than $10 per patient22,26,27,41 to more than $10 per patient,48 with some reporting even higher costs.32,35 A short-term pediatric study27 targeting 20-month-old children for receipt of immunizations noted a cost of $9.80 per child appropriately vaccinated by age 24 months using an autodialer reminder, $10.50 per child using a letter reminder, and $7 per child using a combination of approaches. A more comprehensive pediatric intervention involving patient reminders and outreach throughout the year noted a cost of $63 per child per year, with a cost-effectiveness of $316 per year per additional fully vaccinated child.35 Several studies noted that patient reminder systems had the added benefit of increasing preventive visits and receipt of preventive services in addition to immunizations,32,33,35 making cost-effectiveness difficult to assess but increasing the benefits of the immunization intervention.

Analysis of Studies That Found No Improvements

Since 20% of studies did not find improvements in immunization rates due to patient reminder/recall systems, we performed a qualitative analysis of study characteristics for the 8 studies that noted no improvements.25,31,33,39,54,56,59,60 Three of these studies involved children. Seven studies involved mailed reminders and 1 evaluated an autodialer reminder.31 Authors or reviewers noted the following concerns that might have contributed to the failure of these studies to find significant effects: 1 study clearly focused more on a 15-minute postpartum educational discussion than the reminder25; 2 studies used reminders targeting preventive visits rather than immunizations33,39; 2 studies had small sample sizes31,39; 2 studies noted significant improvements in 1 subgroup but not another54,60; in 1 study, the authors noted a possible ceiling effect with high baseline influenza immunization rates (55%-65%) among controls59; and in 1 study, reviewers noted extremely low immunization rates (<2% for adult tetanus),39 suggesting that other factors were somehow impeding immunization delivery. This qualitative analysis did not find clear trends among these studies in terms of practice setting, patient population, or type of reminder/recall used.

The findings from this systematic review of the literature support the general recommendation811 that all primary care practitioners should consider patient reminder/recall systems to improve immunization coverage levels of their practices. We found that reminder/recall was effective for both children and adults; in all types of medical settings, including private practices, academic medical centers, and public health department clinics; and for universally recommended vaccinations such as routine childhood vaccinations as well as targeted vaccinations such as influenza vaccine. In addition, all types of patient reminder/recall systems were found to be effective, with increases in immunization rates tending to be 5 to 20 percentage points. Telephone reminders were most effective, while there were no major differences in effectiveness among different types of mailed reminders. More intensive reminder/recall systems, such as those using multiple reminders, appeared to be more effective than single reminders. In studies that evaluated costs, patient reminder systems required a nontrivial expense but led to spillover benefits by increasing preventive visits or receipt of other preventive services.

This study has several limitations. First, we used the Cochrane criteria for selecting studies based on study design and methodological criteria, and some studies were excluded because they did not meet these rigorous criteria. While this strategy improved our ability to estimate the true impact of patient reminder systems, it is possible that some excluded studies may have had different findings and that the impact of reminder systems may be different when rigorous study conditions are not used.

Second, the scope of the review was limited to studies published in English. At least 1 study has found that randomized controlled trials published in English were more likely to have positive findings than studies published in German journals.62 However, such language bias was not noted in other studies.63 In addition, 9 studies included in the current review were performed outside the United States; all 9 studies found positive effects of reminder/recall systems.

A third potential limitation involves publication bias: the majority of studies were located from MEDLINE or references from other studies. Because publication bias typically results in failure to publish studies with negative or null findings,6466 it is possible that our findings of positive outcomes in 33 of 41 reviewed studies is partly affected by publication bias and that the impact of reminder/recall is lower than noted in this review. We attempted to minimize publication bias by searching the files of the investigators and immunization experts, searching references of published reviews for abstracts, and reviewing abstracts or proceedings of major scientific meetings. In addition, the funnel plot analysis discussed herein did not find that more precise studies clustered around null results, thereby increasing the plausibility of the positive findings.

A fourth set of limitations involves aspects of the systematic review process. We grouped studies according to key characteristics of either the patient population or the intervention. We defined these groups a priori, and they represent standard groupings used in other studies. However, it is possible that where differences were noted by group, factors other than the intervention might have accounted for these differences. Limitations of the standard Cochrane review criteria are published elsewhere.67

Our method of pooling data has limitations, particularly in light of the heterogeneity of some of the data that is often present in meta-analyses.68,69 Because these reminder/recall studies were performed for a variety of populations, using different interventions, in multiple settings, and across 3 decades, it is not surprising that there is interstudy heterogeneity in the results. Because of this heterogeneity, we performed a qualitative analysis of study characteristics that might explain differences in findings among the 8 studies that had negative findings, and although it was easy to find explanations for the negative findings in each study, we did not note consistent trends. We used random-effects analyses, which had consistently more conservative results (wider 95% CIs) than fixed-effects models. In 1 subgroup (adult influenza vaccinations), a single study by Baker et al57 had more than 24,000 subjects and small but significant effects of reminder/recall, while most of the other studies in that group had clinically larger positive effects of reminder/recall. The large sample size and small effect of the 1 study resulted in heterogeneity within this subgroup but also resulted in a conservative effect on the pooled results by reducing the overall impact of reminder/recall.

Because patient reminder/recall systems appear to be effective in all settings that were evaluated, we recommend that all primary care physicians seriously consider incorporating reminder/recall into their practices. Physicians should review the different types of reminder/recall systems and tailor systems to their own needs. While telephone reminders are most effective, they are also more costly and have not been studied extensively in children except for the use of autodialers, which were found to have smaller but positive effects. Practical issues relevant to choices of the reminder/recall system include characteristics of current computer systems, staffing, accuracy of patient telephone numbers or addresses, availability of computer programmers, and estimated patient responsiveness to different types of reminders. Practitioners today can tailor their billing systems to function as reminder/recall systems for simple procedures, such as selecting all patients aged 65 years or older for reminders about influenza or pneumococcal vaccination. Many billing systems have recently incorporated separate modules that can track immunization status.

A critical issue involves the complexity of rules required for a reminder/recall system. The simplest scenario involves elderly adults, because no special immunization algorithm is needed and eligible patients can be selected by birth dates. A slightly more complex scenario involves "flagging" patients with chronic problems, such as asthma, that would require influenza vaccination. More sophisticated algorithms are required to track prior immunization status, particularly for the complicated pediatric immunization schedule. A promising route involves practitioners linking with computerized immunization registries that are being developed throughout the United States. These registries already contain the necessary algorithms to assess up-to-date status of children and could be modified to deliver patient reminders. Finally, databases of managed care organizations can be modified to become reminder/recall systems. For practitioners, the usefulness of such databases depends on the proportion of a practice's patients covered by the managed care plan and the accuracy of the database information.

Overall, the technology exists to incorporate patient reminder/recall into routine primary care practice. There are additional benefits to the patient and practice beyond improving immunization rates. Studies have shown that patients who are behind in immunizations are also behind in other measures of preventive care70,71 and that reminder/recall systems targeting immunizations can also have spillover effects to improve other aspects of preventive care35 if they are used within primary care practices. Second, patients generally appreciate being reminded by their physician, and such reminder/recall systems may actually improve the patient-physician relationship. Third, in fee-for-service settings, patient reminder/recall systems can increase revenues by increasing visits.

Since patient reminder/recall systems for immunizations have been shown to be effective in a variety of settings, we recommend that future research be focused on implementation issues: how to implement reminder/recall in an efficient manner across large numbers of practices, means to effectively use computerized registries for patient reminder/recall, and demonstration on a large-scale population level of whether these interventions improve immunization rates. These questions will be particularly important for certain new vaccines, such as the conjugate pneumococcal vaccine for children or seasonal vaccines (like influenza vaccine), for which "catch-up" strategies or timing issues become important.

The use of patient reminder/recall systems provides the primary care practitioner with real-life experience at practicing population-based care by improving the care for the entire population served by the practice. Although medicine is traditionally taught and practiced one patient at a time, and preventive services such as immunizations are delivered to individual patients, the measures of success (such as immunization rates) are population-based. Such population-based primary care, while not easy to practice in a busy setting, has the potential to improve the quality of care and performance of primary care practitioners.

Centers for Disease Control and Prevention.  Vaccination coverage levels among children born from February 1995–May 1997—United States, January 1998–December 1998. Available at: http://www.cdc.gov/nip/coverage/nis1998_1.htm. Accessed July 2000.
Centers for Disease Control and Prevention.  Influenza and pneumococcal vaccination levels among adults aged greater than or equal to 65 years—United States, 1997.  MMWR Morb Mortal Wkly Rep.1998;47:797-802.
Centers for Disease Control and Prevention.  Vaccination coverage by race/ethnicity and poverty level among children aged 19-35 months—United States.  MMWR Morb Mortal Wkly Rep.1997;47:956-959.
Massoudi MS, Walsh J, Stokley S.  et al.  Assessing immunization performance of private practitioners in Maine: impact of the assessment, feedback, incentives, and exchange strategy.  Pediatrics.1999;103:1218-1223.
Centers for Disease Control and Prevention.  Notice to readers: national vaccination coverage levels among children aged 19-35 months—United States, 1998.  MMWR Morb Mortal Wkly Rep.1999;48:829-830.
Gyorkos TW, Tannenbaum TN, Abrahamowicz M.  et al.  Evaluation of the effectiveness of immunization delivery methods.  Can J Public Health.1994;85(suppl 1):S14-S30.
Shea S, DuMouchel W, Bahamonde L. A meta-analysis of 16 randomized controlled trials to evaluate computer-based clinical reminder systems for preventive care in the ambulatory setting.  J Am Med Inform Assoc.1996;3:399-409.
Udovic S, Lieu TA. Evidence on office-based interventions to improve childhood immunization delivery.  Pediatr Ann.1998;27:355-361.
Task Force on Community Preventive Services.  Vaccine-preventable diseases: improving vaccination coverage in children, adolescents, and adults.  MMWR Morb Mortal Wkly Rep.1999;48:1-15.
Shefer AM, Briss PA, Rodewald L.  et al.  Improving immunization coverage rates: an evidence-based review of the literature.  Epidemiol Rev.1999;21:96-142.
Ad Hoc Working Group for the Development of the Standards for Pediatric Immunization Practices.  Standards for pediatric immunization practices.  JAMA.1993;269:1817-1822.
Szilagyi PG, Rodewald LE, Humiston SG.  et al.  Immunization practices of pediatricians and family physicians in the United States.  Pediatrics.1994;94:517-523.
Szilagyi PG, Shone LP, Humiston SG, Kolasa MS, Rodewald LE. Staying in the medical home: decline in physician-reported referrals to health department clinics for immunizations.  Am J Prev Med.2000;18:318-324.
Linkins RW, Feikema SM. Immunization registries: the cornerstone of childhood immunization in the 21st century.  Pediatr Ann.1998;27:349-354.
Bero L, Rennie D. The Cochrane Collaboration: preparing, maintaining, and disseminating systematic reviews of the effects of health care.  JAMA.1995;274:1935-1938.
Clark M, Oxman AD. Cochrane Reviewers' Handbook 4.0. In: Review Manager [computer program]. Version 4.0. Oxford, England: The Cochrane Collaboration; 1999. Available at: http://www.cochrane.dk/cochrane/handbook/handbook.htm.
Szilagyi PG, Bordley C, Margolis P, Kraus R, Chelminski A, Vann J. Interventions to improve immunization rates (protocol for a Cochrane Review). In: The Cochrane Library [database on CD-ROM]. Oxford, England: Update Software; 2000. Issue 3.
van Rooyen S, Godlee F, Evans S, Smith R, Black N. Effect of blinding and unmasking on the quality of peer review: a randomized trial.  JAMA.1998;280:234-237.
The Cochrane Collaboration.  Reviews for the Cochrane Collaboration on Effective Professional Practice (CCEPP). In: Bero L, Grilli R, Grimshaw J, Oxman A, eds. The Cochrane Collaboration on Effective Professional Practice Module of the Cochrane Database of Systematic Reviews [database on CD-ROM ]. Oxford, England: Update Software; 1999. Issue 3.
Poole C, Greenland S. Random-effects meta-analyses are not always conservative.  Am J Epidemiol.1999;150:469-475.
Alto WA, Fury D, Condo A, Doran M, Aduddell M. Improving the immunization coverage of children less than 7 years old in a family practice residency.  J Am Board Fam Pract.1994;7:472-477.
Young SA, Halpin TJ, Johnson DA, Irvin JJ, Marks JS. Effectiveness of a mailed reminder on the immunization levels of infants at high risk of failure to complete immunizations.  Am J Public Health.1980;70:422-424.
Linkins RW, Dini EF, Watson G, Patriarca PA. A randomized trial of the effectiveness of computer-generated telephone messages in increasing immunization visits among preschool children.  Arch Pediatr Adolesc Med.1994;148:908-914.
Tollestrup K, Hubbard BB. Evaluation of a follow-up system in a county health department's immunization clinic.  Am J Prev Med.1991;7:24-28.
Oeffinger KC, Roaten SP, Hitchcock MA, Oeffinger PK. The effect of patient education on pediatric immunization rates.  J Fam Pract.1992;35:288-293.
Lieu TA, Black SB, Ray P.  et al.  Computer-generated recall letters for underimmunized children: how cost-effective?  Pediatr Infect Dis J.1997;16:28-33.
Lieu TA, Capra AM, Makol J, Black SB, Shinefield HR. Effectiveness and cost-effectiveness of letters, automated telephone messages, or both for underimmunized children in a health maintenance organization.  Pediatrics.1998;101:E3.
Ferson MJ, Fitzsimmons G, Christie D, Woollett H. School health nurse interventions to increase immunisation uptake in school entrants.  Public Health.1995;109:25-29.
Alemi F, Alemagno S, Goldhagen J.  et al.  Computer reminders improve on-time immunization rates.  Med Care.1996;34(suppl):OS45-OS51.
LeBaron CW, Starnes D, Dini EF, Chambliss JW, Chaney M. The impact of interventions by a community-based organization on inner-city vaccination coverage, Fulton County, Georgia, 1992-1993.  Arch Pediatr Adolesc Med.1998;152:327-332.
Stehr-Green PA, Dini EF, Lindegren ML, Patriarca PA. Evaluation of telephoned computer-generated reminders to improve immunization coverage at inner-city clinics.  Public Health Rep.1993;108:426-430.
Wood D, Halfon N, Donald-Sherbourne C.  et al.  Increasing immunization rates among inner-city, African American children: a randomized trial of case management.  JAMA.1998;279:29-34.
Campbell JR, Szilagyi PG, Rodewald LE, Doane C, Roghmann KJ. Patient-specific reminder letters and pediatric well-child-care show rates.  Clin Pediatr (Phila).1994;33:268-272.
Soljak MA, Handford S. Early results from the Northland immunisation register.  N Z Med J.1987;100:244-246.
Rodewald LE, Szilagyi PG, Humiston SG.  et al.  A randomized study of tracking with outreach and provider prompting to improve immunization coverage and primary care.  Pediatrics.1999;103:31-38.
Szilagyi PG, Rodewald LE, Savageau J, Yoos L, Doane C. Improving influenza vaccination rates in children with asthma: a test of a computerized reminder system and an analysis of factors predicting vaccination.  Pediatrics.1992;90:871-875.
Kemper KJ, Goldberg H. Do computer-generated reminder letters improve the rate of influenza immunization in an urban pediatric clinic?  Am J Dis Child.1993;147:717-718.
Becker DM, Gomez EB, Kaiser DL, Yoshihasi A, Hodge RJ. Improving preventive care at a medical clinic: how can the patient help?  Am J Prev Med.1989;5:353-359.
Hogg W, Bass M, Calonge N, Crouch H, Satenstein G. Randomized controlled study of customized preventive medicine reminder letters in a community practice.  Can Fam Physician.1998;44:81-88.
Rosser W, Hutchison BG, McDowell I, Newell C. Use of reminders to increase compliance with tetanus booster vaccination.  CMAJ.1992;146:911-917.
Rosser WW, McDowell I, Newell C. Use of reminders for preventive procedures in family medicine.  CMAJ.1991;145:807-814.
Siebers MJ, Hunt VB. Increasing the pneumococcal vaccination rate of elderly patients in a general internal medicine clinic.  J Am Geriatr Soc.1985;33:175-178.
Ornstein SM, Garr DR, Jenkins RG, Rust PF, Arnon A. Computer-generated physician and patient reminders: tools to improve population adherence to selected preventive services.  J Fam Pract.1991;32:82-90.
Frame PS, Zimmer JG, Werth PL, Hall WJ, Eberly SW. Computer-based vs manual health maintenance tracking: a controlled trial.  Arch Fam Med.1994;3:581-588.
Lierer VO, Morrow DG, Pariante G, Doksum T. Increasing influenza vaccination adherence through voice mail.  J Am Geriatr Soc.1989;37:1147-1150.
Brimberry R. Vaccination of high-risk patients for influenza: a comparison of telephone and mail reminder methods.  J Fam Pract.1988;26:397-400.
Spaulding SA, Kugler JP. Influenza immunization: the impact of notifying patients of high-risk status.  J Fam Pract.1991;33:495-498.
McDowell I, Newell C, Rosser W. Comparison of three methods of recalling patients for influenza vaccination.  CMAJ.1986;135:991-997.
Satterthwaite P. A randomised intervention study to examine the effect on immunisation coverage of making influenza vaccine available at no cost.  N Z Med J.1997;110:58-60.
Carter W, Beach L, Inui T. The flu shot study: using multiattribute utility theory to design a vaccination intervention.  Organ Behav Hum Decis Processes.1986;38:378-391.
Larson EB, Olsen E, Cole W, Shortell S. The relationship of health beliefs and a postcard reminder to influenza vaccination.  J Fam Pract.1979;8:1207-1211.
Lukasik MH, Pratt G. The telephone: an overlooked technology for prevention in family medicine.  Can Fam Physician.1987;33:1997-2001.
Barton M, Schoenbaum S. Improving influenza vaccination performance in an HMO setting: the use of computer-generated reminders and peer comparison feedback.  Am J Public Health.1990;80:534-536.
Puech M, Ward J, Lajoie V. Postcard reminders from GPs for influenza vaccine: are they more effective than an ad hoc approach?  Aust N Z J Public Health.1998;22:254-256.
Mullooly JP. Increasing influenza vaccination among high-risk elderly: a randomized controlled trial of a mail cue in an HMO setting.  Am J Public Health.1987;77:626-627.
Moran WP, Nelson K, Wofford JL, Velez R. Computer-generated mailed reminders for influenza immunization: a clinical trial.  J Gen Intern Med.1992;7:535-537.
Baker A, McCarthy B, Gurley V, Yood M. Influenza immunization in a managed care organization.  J Gen Intern Med.1998;13:469-475.
Buffington J, Bell KM, LaForce FM. A target-based model for increasing influenza immunizations in private practice: Genesee Hospital Medical Staff.  J Gen Intern Med.1991;6:204-209.
Buchner DM, Larson EB, White RF. Influenza vaccination in community elderly: a controlled trial of postcard reminders.  J Am Geriatr Soc.1987;35:755-760.
Margolis KL, Nichol KL, Wuorenma J, VonSternberg TL. Exporting a successful influenza vaccination program from a teaching hospital to a community outpatient setting.  J Am Geriatr Soc.1992;40:1021-1023.
Nexoe J, Kragstrup J, Ronne T. Impact of postal invitations and user fee on influenza vaccination rates among the elderly.  Scand J Prim Health Care.1997;15:109-112.
Egger M, Zellweger-Zahner T, Schneider M, Junker C, Lengeler C, Antes G. Language bias in randomised controlled trials published in English and German.  Lancet.1997;350:326-329.
Heidenreich PA, McDonald KM, Hastie T.  et al.  Meta-analysis of trials comparing β-blockers, calcium antagonists, and nitrates for stable angina.  JAMA.1999;281:1927-1936.
Chalmers TC, Frank CS, Reitman D. Minimizing the three stages of publication bias.  JAMA.1990;263:1392-1395.
Easterbrook PJ, Berlin JA, Gopalon R.  et al.  Publication bias in clinical research.  Lancet.1991;337:867-872.
Dickerson K, Min YI, Meinert CL. Factors influencing publication of research results.  JAMA.1992;267:374-378.
Jadad AR, Cook DJ, Jones A.  et al.  Methodology and reports of systematic reviews and meta-analyses: a comparison of Cochrane reviews with articles published in paper-based journals.  JAMA.1998;280:278-280.
Gottlieb MS, Carr JK, Clarkson JR. Drinking water and cancer in Louisiana: a retrospective mortality study.  Am J Epidemiol.1982;116:652-667.
Thompson SG, Pocock SJ. Can meta-analysis be trusted?  Lancet.1991;338:1127-1130.
Rodewald LE, Szilagyi PG, Shiuh T.  et al.  Is underimmunization a marker for insufficient utilization of preventive and primary care?  Arch Pediatr Adolesc Med.1995;149:393-397.
Fairbrother G, Friedman S, DuMont KA, Lobach KS. Markers for primary care: missed opportunities to immunize and screen for lead and tuberculosis by private physicians serving large numbers of inner-city Medicaid-eligible children.  Pediatrics.1996;97:785-790.

Figures

Figure. Effectiveness of Different Types of Patient Reminder/Recall Systems for All Ages Combined
Graphic Jump Location
OR indicates odds ratio; CI, confidence interval. Error bars indicate 95% CIs.

Tables

Table Graphic Jump LocationTable 1. Effectiveness of Patient Reminder/Recall by Patient Characteristics
Table Graphic Jump LocationTable 2. Effectiveness of Different Types of Reminder/Recall Systems for Children and Adults

References

Centers for Disease Control and Prevention.  Vaccination coverage levels among children born from February 1995–May 1997—United States, January 1998–December 1998. Available at: http://www.cdc.gov/nip/coverage/nis1998_1.htm. Accessed July 2000.
Centers for Disease Control and Prevention.  Influenza and pneumococcal vaccination levels among adults aged greater than or equal to 65 years—United States, 1997.  MMWR Morb Mortal Wkly Rep.1998;47:797-802.
Centers for Disease Control and Prevention.  Vaccination coverage by race/ethnicity and poverty level among children aged 19-35 months—United States.  MMWR Morb Mortal Wkly Rep.1997;47:956-959.
Massoudi MS, Walsh J, Stokley S.  et al.  Assessing immunization performance of private practitioners in Maine: impact of the assessment, feedback, incentives, and exchange strategy.  Pediatrics.1999;103:1218-1223.
Centers for Disease Control and Prevention.  Notice to readers: national vaccination coverage levels among children aged 19-35 months—United States, 1998.  MMWR Morb Mortal Wkly Rep.1999;48:829-830.
Gyorkos TW, Tannenbaum TN, Abrahamowicz M.  et al.  Evaluation of the effectiveness of immunization delivery methods.  Can J Public Health.1994;85(suppl 1):S14-S30.
Shea S, DuMouchel W, Bahamonde L. A meta-analysis of 16 randomized controlled trials to evaluate computer-based clinical reminder systems for preventive care in the ambulatory setting.  J Am Med Inform Assoc.1996;3:399-409.
Udovic S, Lieu TA. Evidence on office-based interventions to improve childhood immunization delivery.  Pediatr Ann.1998;27:355-361.
Task Force on Community Preventive Services.  Vaccine-preventable diseases: improving vaccination coverage in children, adolescents, and adults.  MMWR Morb Mortal Wkly Rep.1999;48:1-15.
Shefer AM, Briss PA, Rodewald L.  et al.  Improving immunization coverage rates: an evidence-based review of the literature.  Epidemiol Rev.1999;21:96-142.
Ad Hoc Working Group for the Development of the Standards for Pediatric Immunization Practices.  Standards for pediatric immunization practices.  JAMA.1993;269:1817-1822.
Szilagyi PG, Rodewald LE, Humiston SG.  et al.  Immunization practices of pediatricians and family physicians in the United States.  Pediatrics.1994;94:517-523.
Szilagyi PG, Shone LP, Humiston SG, Kolasa MS, Rodewald LE. Staying in the medical home: decline in physician-reported referrals to health department clinics for immunizations.  Am J Prev Med.2000;18:318-324.
Linkins RW, Feikema SM. Immunization registries: the cornerstone of childhood immunization in the 21st century.  Pediatr Ann.1998;27:349-354.
Bero L, Rennie D. The Cochrane Collaboration: preparing, maintaining, and disseminating systematic reviews of the effects of health care.  JAMA.1995;274:1935-1938.
Clark M, Oxman AD. Cochrane Reviewers' Handbook 4.0. In: Review Manager [computer program]. Version 4.0. Oxford, England: The Cochrane Collaboration; 1999. Available at: http://www.cochrane.dk/cochrane/handbook/handbook.htm.
Szilagyi PG, Bordley C, Margolis P, Kraus R, Chelminski A, Vann J. Interventions to improve immunization rates (protocol for a Cochrane Review). In: The Cochrane Library [database on CD-ROM]. Oxford, England: Update Software; 2000. Issue 3.
van Rooyen S, Godlee F, Evans S, Smith R, Black N. Effect of blinding and unmasking on the quality of peer review: a randomized trial.  JAMA.1998;280:234-237.
The Cochrane Collaboration.  Reviews for the Cochrane Collaboration on Effective Professional Practice (CCEPP). In: Bero L, Grilli R, Grimshaw J, Oxman A, eds. The Cochrane Collaboration on Effective Professional Practice Module of the Cochrane Database of Systematic Reviews [database on CD-ROM ]. Oxford, England: Update Software; 1999. Issue 3.
Poole C, Greenland S. Random-effects meta-analyses are not always conservative.  Am J Epidemiol.1999;150:469-475.
Alto WA, Fury D, Condo A, Doran M, Aduddell M. Improving the immunization coverage of children less than 7 years old in a family practice residency.  J Am Board Fam Pract.1994;7:472-477.
Young SA, Halpin TJ, Johnson DA, Irvin JJ, Marks JS. Effectiveness of a mailed reminder on the immunization levels of infants at high risk of failure to complete immunizations.  Am J Public Health.1980;70:422-424.
Linkins RW, Dini EF, Watson G, Patriarca PA. A randomized trial of the effectiveness of computer-generated telephone messages in increasing immunization visits among preschool children.  Arch Pediatr Adolesc Med.1994;148:908-914.
Tollestrup K, Hubbard BB. Evaluation of a follow-up system in a county health department's immunization clinic.  Am J Prev Med.1991;7:24-28.
Oeffinger KC, Roaten SP, Hitchcock MA, Oeffinger PK. The effect of patient education on pediatric immunization rates.  J Fam Pract.1992;35:288-293.
Lieu TA, Black SB, Ray P.  et al.  Computer-generated recall letters for underimmunized children: how cost-effective?  Pediatr Infect Dis J.1997;16:28-33.
Lieu TA, Capra AM, Makol J, Black SB, Shinefield HR. Effectiveness and cost-effectiveness of letters, automated telephone messages, or both for underimmunized children in a health maintenance organization.  Pediatrics.1998;101:E3.
Ferson MJ, Fitzsimmons G, Christie D, Woollett H. School health nurse interventions to increase immunisation uptake in school entrants.  Public Health.1995;109:25-29.
Alemi F, Alemagno S, Goldhagen J.  et al.  Computer reminders improve on-time immunization rates.  Med Care.1996;34(suppl):OS45-OS51.
LeBaron CW, Starnes D, Dini EF, Chambliss JW, Chaney M. The impact of interventions by a community-based organization on inner-city vaccination coverage, Fulton County, Georgia, 1992-1993.  Arch Pediatr Adolesc Med.1998;152:327-332.
Stehr-Green PA, Dini EF, Lindegren ML, Patriarca PA. Evaluation of telephoned computer-generated reminders to improve immunization coverage at inner-city clinics.  Public Health Rep.1993;108:426-430.
Wood D, Halfon N, Donald-Sherbourne C.  et al.  Increasing immunization rates among inner-city, African American children: a randomized trial of case management.  JAMA.1998;279:29-34.
Campbell JR, Szilagyi PG, Rodewald LE, Doane C, Roghmann KJ. Patient-specific reminder letters and pediatric well-child-care show rates.  Clin Pediatr (Phila).1994;33:268-272.
Soljak MA, Handford S. Early results from the Northland immunisation register.  N Z Med J.1987;100:244-246.
Rodewald LE, Szilagyi PG, Humiston SG.  et al.  A randomized study of tracking with outreach and provider prompting to improve immunization coverage and primary care.  Pediatrics.1999;103:31-38.
Szilagyi PG, Rodewald LE, Savageau J, Yoos L, Doane C. Improving influenza vaccination rates in children with asthma: a test of a computerized reminder system and an analysis of factors predicting vaccination.  Pediatrics.1992;90:871-875.
Kemper KJ, Goldberg H. Do computer-generated reminder letters improve the rate of influenza immunization in an urban pediatric clinic?  Am J Dis Child.1993;147:717-718.
Becker DM, Gomez EB, Kaiser DL, Yoshihasi A, Hodge RJ. Improving preventive care at a medical clinic: how can the patient help?  Am J Prev Med.1989;5:353-359.
Hogg W, Bass M, Calonge N, Crouch H, Satenstein G. Randomized controlled study of customized preventive medicine reminder letters in a community practice.  Can Fam Physician.1998;44:81-88.
Rosser W, Hutchison BG, McDowell I, Newell C. Use of reminders to increase compliance with tetanus booster vaccination.  CMAJ.1992;146:911-917.
Rosser WW, McDowell I, Newell C. Use of reminders for preventive procedures in family medicine.  CMAJ.1991;145:807-814.
Siebers MJ, Hunt VB. Increasing the pneumococcal vaccination rate of elderly patients in a general internal medicine clinic.  J Am Geriatr Soc.1985;33:175-178.
Ornstein SM, Garr DR, Jenkins RG, Rust PF, Arnon A. Computer-generated physician and patient reminders: tools to improve population adherence to selected preventive services.  J Fam Pract.1991;32:82-90.
Frame PS, Zimmer JG, Werth PL, Hall WJ, Eberly SW. Computer-based vs manual health maintenance tracking: a controlled trial.  Arch Fam Med.1994;3:581-588.
Lierer VO, Morrow DG, Pariante G, Doksum T. Increasing influenza vaccination adherence through voice mail.  J Am Geriatr Soc.1989;37:1147-1150.
Brimberry R. Vaccination of high-risk patients for influenza: a comparison of telephone and mail reminder methods.  J Fam Pract.1988;26:397-400.
Spaulding SA, Kugler JP. Influenza immunization: the impact of notifying patients of high-risk status.  J Fam Pract.1991;33:495-498.
McDowell I, Newell C, Rosser W. Comparison of three methods of recalling patients for influenza vaccination.  CMAJ.1986;135:991-997.
Satterthwaite P. A randomised intervention study to examine the effect on immunisation coverage of making influenza vaccine available at no cost.  N Z Med J.1997;110:58-60.
Carter W, Beach L, Inui T. The flu shot study: using multiattribute utility theory to design a vaccination intervention.  Organ Behav Hum Decis Processes.1986;38:378-391.
Larson EB, Olsen E, Cole W, Shortell S. The relationship of health beliefs and a postcard reminder to influenza vaccination.  J Fam Pract.1979;8:1207-1211.
Lukasik MH, Pratt G. The telephone: an overlooked technology for prevention in family medicine.  Can Fam Physician.1987;33:1997-2001.
Barton M, Schoenbaum S. Improving influenza vaccination performance in an HMO setting: the use of computer-generated reminders and peer comparison feedback.  Am J Public Health.1990;80:534-536.
Puech M, Ward J, Lajoie V. Postcard reminders from GPs for influenza vaccine: are they more effective than an ad hoc approach?  Aust N Z J Public Health.1998;22:254-256.
Mullooly JP. Increasing influenza vaccination among high-risk elderly: a randomized controlled trial of a mail cue in an HMO setting.  Am J Public Health.1987;77:626-627.
Moran WP, Nelson K, Wofford JL, Velez R. Computer-generated mailed reminders for influenza immunization: a clinical trial.  J Gen Intern Med.1992;7:535-537.
Baker A, McCarthy B, Gurley V, Yood M. Influenza immunization in a managed care organization.  J Gen Intern Med.1998;13:469-475.
Buffington J, Bell KM, LaForce FM. A target-based model for increasing influenza immunizations in private practice: Genesee Hospital Medical Staff.  J Gen Intern Med.1991;6:204-209.
Buchner DM, Larson EB, White RF. Influenza vaccination in community elderly: a controlled trial of postcard reminders.  J Am Geriatr Soc.1987;35:755-760.
Margolis KL, Nichol KL, Wuorenma J, VonSternberg TL. Exporting a successful influenza vaccination program from a teaching hospital to a community outpatient setting.  J Am Geriatr Soc.1992;40:1021-1023.
Nexoe J, Kragstrup J, Ronne T. Impact of postal invitations and user fee on influenza vaccination rates among the elderly.  Scand J Prim Health Care.1997;15:109-112.
Egger M, Zellweger-Zahner T, Schneider M, Junker C, Lengeler C, Antes G. Language bias in randomised controlled trials published in English and German.  Lancet.1997;350:326-329.
Heidenreich PA, McDonald KM, Hastie T.  et al.  Meta-analysis of trials comparing β-blockers, calcium antagonists, and nitrates for stable angina.  JAMA.1999;281:1927-1936.
Chalmers TC, Frank CS, Reitman D. Minimizing the three stages of publication bias.  JAMA.1990;263:1392-1395.
Easterbrook PJ, Berlin JA, Gopalon R.  et al.  Publication bias in clinical research.  Lancet.1991;337:867-872.
Dickerson K, Min YI, Meinert CL. Factors influencing publication of research results.  JAMA.1992;267:374-378.
Jadad AR, Cook DJ, Jones A.  et al.  Methodology and reports of systematic reviews and meta-analyses: a comparison of Cochrane reviews with articles published in paper-based journals.  JAMA.1998;280:278-280.
Gottlieb MS, Carr JK, Clarkson JR. Drinking water and cancer in Louisiana: a retrospective mortality study.  Am J Epidemiol.1982;116:652-667.
Thompson SG, Pocock SJ. Can meta-analysis be trusted?  Lancet.1991;338:1127-1130.
Rodewald LE, Szilagyi PG, Shiuh T.  et al.  Is underimmunization a marker for insufficient utilization of preventive and primary care?  Arch Pediatr Adolesc Med.1995;149:393-397.
Fairbrother G, Friedman S, DuMont KA, Lobach KS. Markers for primary care: missed opportunities to immunize and screen for lead and tuberculosis by private physicians serving large numbers of inner-city Medicaid-eligible children.  Pediatrics.1996;97:785-790.
CME
Meets CME requirements for:
Browse CME for all U.S. States
Accreditation Information
The American Medical Association is accredited by the Accreditation Council for Continuing Medical Education to provide continuing medical education for physicians. The AMA designates this journal-based CME activity for a maximum of 1 AMA PRA Category 1 CreditTM per course. Physicians should claim only the credit commensurate with the extent of their participation in the activity. Physicians who complete the CME course and score at least 80% correct on the quiz are eligible for AMA PRA Category 1 CreditTM.
Note: You must get at least of the answers correct to pass this quiz.
You have not filled in all the answers to complete this quiz
The following questions were not answered:
Sorry, you have unsuccessfully completed this CME quiz with a score of
The following questions were not answered correctly:
Commitment to Change (optional):
Indicate what change(s) you will implement in your practice, if any, based on this CME course.
Your quiz results:
The filled radio buttons indicate your responses. The preferred responses are highlighted
For CME Course: A Proposed Model for Initial Assessment and Management of Acute Heart Failure Syndromes
Indicate what changes(s) you will implement in your practice, if any, based on this CME course.
NOTE:
Citing articles are presented as examples only. In non-demo SCM6 implementation, integration with CrossRef’s "Cited By" API will populate this tab (http://www.crossref.org/citedby.html).

Multimedia

Some tools below are only available to our subscribers or users with an online account.

Web of Science® Times Cited: 178

Related Content

Customize your page view by dragging & repositioning the boxes below.

Articles Related By Topic
Related Topics
PubMed Articles