Effect of a Monetary Sanction on Immunization Rates of Recipients of Aid to Families With Dependent Children FREE

Although immunization rates have increased in general over the last decade,^1- 2 the rates for poor and minority children have not kept pace.^3- 4 Robinson and colleagues⁵ have identified several risk factors that highly correlate with failure to immunize children on schedule, including having low parental education level, large family size, low socioeconomic status, or ethnic or minority group membership; receiving services through public health clinics; being a single parent; starting the immunization series late; and having inadequate insurance coverage for immunizations. Others have indicated that the difficulties poor families have in keeping their children's immunizations up-to-date result from deficiencies in the health care system,^6- 7 and a lack of transportation, social support, and understanding of immunization schedules.^8- 9 These factors strongly suggest that families receiving welfare assistance are at risk of not having their children immunized in a timely fashion.

In the 1990s, the federal government encouraged states to seek waivers of federal welfare rules to test innovative programs and policies, including those that aimed at improving the health of public aid recipients. These waivers, which carried with them an obligation to evaluate the effects of the innovative programs and policies, allowed states to engage in experiments to encourage behaviors among welfare recipients that would lead, among other things, to better family health. Twenty-one states applied for health-related welfare waivers.¹⁰ After the passage of the Personal Responsibility and Work Opportunity Act of 1996, which gave states greater flexibility in administering their welfare programs, many states ceased evaluation activities. To our knowledge, of the states that received waivers for encouraging immunization, only Georgia and Maryland have completed program evaluations.

We present herein the results of a 4-year evaluation of the effect of the Preschool Immunization Project (PIP), a large-scale immunization initiative. The PIP was designed and implemented by the Georgia Department of Human Resources, Division of Family and Children Services, the agency responsible for administering that state's Aid to Families with Dependent Children (AFDC) program. Although the 1996 welfare reform legislation replaced AFDC with Temporary Assistance for Needy Families midway through our evaluation, we refer to Georgia's welfare program as AFDC for the purposes of consistency.

On January 1, 1993, all families (except for 1000 families in the control group) who either applied or reapplied for AFDC benefits in Georgia were told that they had to provide proof of up-to-date immunizations for their preschool-aged children (≤6 years). They were reminded of their obligation both when they applied and when they were recertified for welfare eligibility, which was required semiannually until 1996, when it became an annual requirement. If the family did not present such proof without good cause, such as having religious objections or known allergic reactions, a sanction could be applied after oral or written warnings were issued. The sanction was losing AFDC benefits normally provided for the nonimmunized child. Medicaid benefits and those for Early Periodic Screening, Diagnosis, and Treatment were not affected.

To evaluate the effect of the sanction, a randomized controlled trial was instituted in Muscogee County, Georgia, using a systematic random sampling method, by which 1500 families were randomly selected as the intervention group and 1000 families were randomly selected as the control group. Although families in the control group were encouraged to immunize their preschool children, they were neither told about the sanction nor penalized for failure to immunize them. Families in the intervention group, as all the other families receiving AFDC in Georgia, were told about the sanction for not immunizing their preschool children.

Muscogee County, Georgia, consists mainly of the city of Columbus (1995 population 178,681) and the Fort Benning Military Reservation. Almost all of the sample cases were from Columbus. The PIP was approved by the US Department of Health and Human Services, which mandated a randomized assignment evaluation of the project by an independent organization. In doing so, the US Department of Health and Human Services counsel indicated that the project and its evaluation were exempt from human subjects review.

The Georgia Department of Administrative Services used a systematic random sampling method to assign families into the intervention and control groups in November 1992, before the project began on January 1, 1993. Data from the state's Public Assistance Reporting and Information System files indicated that 3600 families receiving AFDC in Muscogee County were subject to the PIP. From those families, every third family was assigned to the control group until 1000 cases were selected. Every third family from the remaining 2600 was assigned to the intervention group until 1500 cases were selected (Figure 1).

To assess the incidence of the application of sanctions, we reviewed the Muscogee County AFDC case record files of 2265 of the 2500 cases in the study. The remaining 235 case records were not available in the Muscogee County AFDC offices.

Age-Appropriate Immunization Rates. The main outcome measure entailed examination of medical records of preschool-aged children to assess whether they had received their diphtheria and tetanus toxoids and pertussis (DTP), poliovirus, measles-mumps-rubella (MMR), Haemophilus influenzae type b (Hib), and hepatitis B (HBV) vaccines when required. Children were considered to be up-to-date if they had received age-appropriate immunizations based on the recommendations of the Advisory Committee on Immunization Practices and on Georgia PIP requirements in effect at that time. A 1-month grace period was allowed before categorizing a child as not up-to-date. We considered each immunization separately, as well as series completion at age 2 years (4 DPT, 3 poliovirus, and 1 MMR immunizations).

Conditions for review of children's immunization records were: (1) the child's parent or guardian gave written informed consent for the review of medical records; (2) a local immunization provider or providers (whether private physician or county health department) could be identified; and (3) the medical provider(s) gave permission to examine the records. For each child old enough to be eligible for at least 1 immunization at each time point examined, immunization records were recorded from all providers that could be identified as having immunized the child at any point since birth.

Five trained abstractors, from Columbus State University's Health Science Department, recorded all vaccinations. The abstractors were blinded to the status of the children. The record abstracts were edited, entered, verified, and checked for internal consistency and obvious errors.

The collection of children's immunization data (through review of their medical records) began in 1995 and continued into early 1998. All medical record reviews completed before December 1996 were updated during 1997 and 1998. The resulting immunization data for this evaluation were current as of December 31, 1996. Analyses of the medical records file were completed for the children in the sample, even those whose families no longer received AFDC benefits.

Client Burden. Ten-minute-long telephone interviews were conducted between October 1995 and December 1996. Forty families in the intervention group were asked about the extra burden imposed on them by the PIP. These interviews attempted to determine the annual amount of time spent, time lost from work, and out-of-pocket transportation costs required to address the immunization documentation requirements.

Rates of participation (in AFDC and in the study) were compared using χ² analysis. The effect of the intervention on immunization rates was assessed using logit (or log of the odds) analysis, with 4 variables entered into the equations: intervention status (intervention/control), age (in months), sex (male/female), and ethnicity (minority/nonminority). Children's up-to-date immunization status was evaluated at 5 points: baseline (January 1, 1993), after 1 year (December 31, 1993), 2 years (December 31, 1994), 3 years (December 31, 1995), and 4 years (December 31, 1996) of the demonstration's operation. All available immunization records were included in each analysis reported herein whether or not the child remained subject to PIP requirements (ie, active AFDC status and age ≤6 years). Series-completion rates by age 2 years were assessed. For this analysis, records of children who turned 2 years old during the study period (January 1, 1993-December 31, 1996) were examined for these immunization series.

Sample Characteristics. As of January 1, 1993, the intervention group included 2488 children (1.66 per family); the control group included 1662 children (1.66 per family). At that time, the average age of the preschool-aged children was 3.22 years for the intervention group and 3.34 years for the control group (Table 1). In each group 85% were black and 14% were white. The intervention group had the same proportion of males and females, but the control group had slightly fewer males proportionally (48.5%). During the study, the 278 infants born in the control group families and 483 in the intervention group families were included in this study. Toward the end of the study, the average number of preschool-aged children in intervention group families was 1.98 and 1.94 in the control group families. Age breakdowns indicate that similar proportions of children were in each age group for the 2 groups.

Participation in AFDC. Over the 4-year course of this evaluation, families in both intervention and control groups left the Muscogee County AFDC program, either by income ineligibility or by moving out of the county. In December 1993, 86.3% of the control group and 82.8% of the intervention group remained active. That difference in the percentages remaining active is statistically significant (χ²=5.8, P<.02). By December 1994, 76.4% of the control group vs 71.6% of the intervention group remained active (χ²=7.3, P<.01). By December 1995, however, that difference narrowed with 68.2% in the control group vs 66.9% in the intervention group remaining active recipients (χ²=0.6, P<.50). As of December 31, 1996, 65.5% of the control group vs 61.7% in the intervention group remained active (χ²=3.8, P<.05).

Study Participation Rates. The ability to collect information from the entire research sample for the duration of an evaluation affects the reliability of observed results. If a segment of the original sample were not involved in the analysis and if that segment were different in important ways relevant to the measures of interest, the final results could be biased. In this study, written consent to review medical records was obtained for 3001 (61.1%) of the 4911 sampled children: 1864 children (62.7%) in the intervention group and 1137 children (58.7%) in the control group. This difference is statistically significant (χ²=5.24, P<.05).

Preexisting differences between families in the intervention group that did and did not provide consent to review records were assessed by analyzing demographic characteristics at baseline. Families providing consent had more children and were more likely to be ethnic minorities.

Children in the intervention group for whom permission to review medical records was received were younger than those in the control group for whom permission was received, but all other demographic characteristics were similar for the 2 groups. The difference in children's ages observed in this analysis contrasts with the lack of difference in average age of children in the intervention and control groups at the time of original family selection (Table 1).

Families in the intervention group received 17 warnings that benefits would be decreased if parents continued not to provide proof of immunization. Families in the control group mistakenly received 3 warnings. Of the 20 warnings, 11 resulted in sanctions being applied, all to families in the intervention group, affecting a total of 18 children. Eight of the 11 sanctioned families reapplied and were approved for benefits. The period required for reactivation ranged from 1 month (as was the case for 5 families) to 6 months.

Up-to-Date Immunization Rates. There were no differences between the immunization rates of intervention and control groups at baseline. After baseline, immunization rates were higher for intervention group children for all immunizations in each year. Differences after the project's first year are statistically significant for all immunizations (P<.05), with the lone exception of Hib in the second year (Table 2). Observed intervention effects generally increased each year for the first 2 years, as the PIP regulations took root in the intervention population and were maintained in the succeeding 2 years (Figure 2).

Children's age was significantly related to up-to-date status for each immunization at each time point, with 2 exceptions (poliovirus in the third year and MMR in the first year). For DTP, poliovirus, and MMR vaccine, older children were more likely to be fully immunized. However, for Hib and HBV vaccines younger children were more likely to have completed the series requirements.

Sex was unrelated to immunization status at all time points except for the baseline Hib vaccine, for which males were more likely to be up-to-date. Ethnicity was unrelated to immunization for all vaccinations except HBV. For HBV, nonminority children were more likely to be up-to-date after baseline regardless of intervention status, age, or sex.

Analyses conducted of only those children aged 6 years or younger provided equivalent results; in fact, stronger intervention effects were observed for that sample alone.

Series Completion. Of the 510 intervention group children who became 2 years old from January 1, 1993, through December 31, 1996, 369 (72.4%) achieved series-completion. Of the 340 control group children, 206 (60.6%) achieved series-completion. This difference is statistically significant (χ²=13.4, P<.001).

The interviews with 40 families showed that the level of additional annual burden of being required to provide immunization documentation was 0.66 hours in time spent, 0.13 hours in time lost from work, and $0.41 in out-of-pocket transportation costs.

Age-appropriate vaccination coverage rates, although virtually identical at baseline, increased in the intervention group after 1 year and remained level for the remainder of the study. With just 3 exceptions, the intervention group had statistically significant and clinically meaningful higher coverage rates (by about 6-7 percentage points) for all 5 of the vaccines from the project's first through fourth years.

Immunization rates for MMR and poliovirus are high (≥80% after the first year), and the rate for DTP approaches that of these 2 immunizations (≥70%). For the Hib and HBV vaccinations, rates are far lower (<30%). The Hib and HBV vaccines have been a relatively recent requirement, so it might be expected that compliance on these 2 immunizations might take several years to reach high levels. A study of HBV vaccination rates in a similar population for which intensive in-person efforts were made to improve immunization coverage found similar rates of HBV coverage but with a more rapid rate of increase in up-to-date levels.¹¹

Given that the already high levels of immunization rates for MMR, poliovirus, and DTP present a potential ceiling effect, the increase of as much as 7 percentage points in immunization up-to-date rates achieved with these 3 vaccinations is all the more remarkable. Finally, not only are the PIP's effects positive, but the burden on families to adhere to the PIP requirements was low.

Although age of a child is significantly related to all immunization rates, the direction of the relationship is different depending on the immunization. A positive relationship between age and immunization rate was found for MMR, poliovirus, and DTP, whereas a negative relationship was found for Hib and HBV immunization rates. At the outset of the PIP, up-to-date rates for Hib and HBV immunizations were extremely low, no doubt because inclusion of these 2 in the immunization series was fairly new. It appears that as the original sample of children aged, little attempt was made to bring them up-to-date for these vaccines, but children born into the sample as the PIP progressed were being immunized more routinely with Hib and HBV vaccines.

The finding that minority children were significantly underimmunized with HBV vaccines appears contrary to 1997 data from the National Immunization Survey.⁴ In our study, that difference progressively decreased with time, so it is possible that had data been obtained in our samples for 1997, that difference would have disappeared.

Comparison of the intervention and control groups' demographic data indicate that the systematic random assignment procedure was performed correctly and had the desired result. That the intervention and control groups can be considered statistically similar at baseline makes the intervention effects found all the more compelling. On the other hand, given that the intervention-control group difference among families actively participating in AFDC at most points in the time studied is statistically significant, it is conceivable that the intervention may have influenced some families to leave AFDC.

The net effect of the potential bias that may have resulted from families not consenting to let their children's medical records be examined was determined by analyzing family demographics. Although the ages of the children in the intervention and control groups for whom medical records were examined were different, unlike the ages of the groups in the entire sample, when adjusted for in the logit analysis, that difference did not appear to have an effect on the intervention results. Despite the potential bias inherent in obtaining family consent, a record check of immunizations yields a more valid and reliable indication of immunization status than does health care clinician or parental recall,¹² measures that are often used in other studies of childhood immunization rates.

In any project requiring additional activities from the administering agency and the program recipients, program implementation cannot be expected to be perfect. Results from a process evaluation of the program showed some difficulties in implementation. Case workers said (or at least remembered) that they received relatively little training, and the sanction was not applied according to agency procedures in some instances. Among the difficulties in administering the study were (1) some control group families were treated like intervention group members; (2) other unrelated immunization promotional efforts were operating in the community during the same period diluting the effect of the intervention being evaluated; and (3) during the 4-year evaluation period, national welfare policy changes occurred that affected all state welfare programs.

Factors such as treatment crossover and dilution of the treatment intervention operated to mitigate the purity of this evaluation. Yet the intervention accomplished its objectives despite these adversities, for the efforts of the PIP are decidedly positive for ensuring childhood immunizations.

In the Georgia PIP, sanctions were seldom implemented, yet the families to whom the threat of sanctions were made had their children immunized at higher levels than did the families not subject to such sanctions. The sanction intervention undoubtedly served several purposes. Semiannually, at first, then annually, it focused attention on a desired health behavior, imparted information about what was expected of families, and threatened monetary loss for not carrying out the desired health behavior.

We found only 1 other completed welfare waiver evaluation of the role of sanctions on the immunization behavior of persons receiving public assistance, namely, Maryland's Primary Prevention Initiative (PPI). Even though many welfare waiver projects like the PIP were initiated in the early 1990s, with planned evaluations, changes in the 1996 welfare reform legislation caused many states to drop both their demonstration projects and their evaluations of them.

The Maryland PPI aimed at changing a more comprehensive range of behaviors than did Georgia's PIP, including requirements related to preventive health care (over and above immunizations), prenatal care, and attendance at school on the part of school-aged children in the family. Concerning immunization, the Maryland PPI evaluation found "no consistent evidence that PPI contributed to an increase in immunization coverage,"¹³ nor did it find an effect on any of the other desired health behaviors, nor on school attendance. The evaluators of the Maryland program attributed the total lack of effect of that program to several factors, including weaknesses in implementing the sanction, compensatory increases in other aid to families that were sanctioned, and inadvertent exposure of control families to 1 or more intervention components.¹⁴

In Georgia's PIP, there were, similarly, weaknesses in implementing the sanction, little overall intrafamilial economic impact of the sanctions, and inadvertent exposure of control families to the intervention. However, the PIP's improvement in immunization rates is the opposite of those found in Maryland. It is plausible to conclude that the Georgia project achieved its desired ends because it focused on 1 aspect of health behavior (immunizations), which was easy to understand and required clear actions from participants. Clients subjected to the requirements of Georgia's PIP more than likely could comprehend readily what was expected of them to avoid being penalized and found it relatively easy and convenient to have their children immunized. Maryland's PPI, on the other hand, aimed its efforts at a broader range of health (and nonhealth) behaviors. Participants in PPI may have found it harder to understand what was required of them, to effect that broader range of health behaviors, and thus to adhere to that project's requirements.

Linking health activities to welfare benefit payments and concomitantly placing responsibility for their children's immunizations on the recipients of welfare was somewhat controversial when this project was launched (and remains so). Linking welfare payments to health-related actions on the part of welfare recipients is arguably unfair, in that the problems it intends to address are due more to problems in the health care delivery system than to the family itself.¹⁵ Families receiving welfare may have particular difficulties keeping their children up-to-date on immunizations due to the relative scarcity of Medicaid providers, systemic barriers (such as restricted clinic hours), and lack of relevant information. In fact, the Center for Law and Social Policy called for a moratorium on the AFDC grant reduction approach "until evaluations of it and other approaches provide information about the relative effectiveness of different approaches."¹⁶ Yet the fact remains that poor children are at higher risk of vaccine-preventable illnesses. It is both a public health obligation to encourage low-income parents to have their children immunized for these diseases and a benefit to these families and to the public as a whole. There are other advantages, as well. Although immunization rates are high at school entry,¹⁷ there are delays in vaccine administration before school entry that a program like this addresses. Furthermore, immunization visits provide an opportunity for other well-baby or well-child services to be provided. Although vaccine-preventable illnesses may pose no obvious danger now, the recent measles outbreak suggests that the public health armamentarium should include effective means to increase immunizations by implementing programs similar to the PIP.

The results of our evaluation suggest that when low-income families are given the incentive to keep their children up-to-date on immunizations and are reminded regularly of this, they are able to do so. The threat of a penalty and regular reminders about the important positive measure they could take for their children provided enough of an incentive and focus for parents to have their children immunized. Even though very few families were actually penalized by having their welfare benefits reduced, the overall effect of the PIP was decidedly beneficial for this population.