An Educational Intervention to Improve Physician Reporting of Adverse Drug Reactions:  A Cluster-Randomized Controlled Trial FREE

When a new drug is licensed, drug safety information tends to be limited. Before approval, drugs are usually evaluated for a defined indication in clinical trials of relatively short duration and involving a small sample size. Study populations often exclude patients with complicated medical conditions, those receiving concurrent drug therapy, young persons, and elderly persons.^1- 2 Hence, after a drug is marketed, previously unidentified important adverse drug reactions (ADRs) may occur. Postmarketing surveillance is important for the discovery of such new ADRs, and a spontaneous reporting system is the primary method of postmarketing surveillance.^3- 6

Physicians are in a position to play a key role in reporting programs,^4- 5 but underreporting is very common, with an estimated median underreporting rate (defined as percentage of ADRs detected from intensive data collection that were not reported to relevant spontaneous reporting systems) of 94%,⁷ and occurs frequently for serious and unlabeled reactions.^8- 9 This can delay detection of important ADRs.⁴ Studies from different settings indicate inadequate knowledge about ADRs among physicians, as well as attitudes that are associated with a high degree of underreporting.^10- 15 However, we are aware of only 4 studies^16- 19 that evaluated the effectiveness of educational interventions aimed at increasing reporting among physicians, none of which was a controlled trial.

To assess whether an intervention of educational outreach visits could increase ADR reporting, we undertook a randomized controlled trial among physicians in northern Portugal. We assessed changes in the rate of ADR reporting, the quality of reporting, and the duration of the effect of this intervention.

The study was conducted in the Northern Region of Portugal, covering a population of 3.7 million, with 104 outpatient centers and 25 hospitals. Fifteen of the hospitals are general medical hospitals, which cover a designated geographic catchment area; 5 are small satellite hospitals of the general hospitals; and 5 are specialty hospitals (eg, cancer, maternity, pediatric, which cover the population of a number of general medical hospitals).

The study population included all National Health System physicians in the Northern Region of Portugal except for those not involved in any clinical activity (eg, administrators, laboratory analysis); those working in substance abuse and rehabilitation centers or specialty hospitals (because they cover multiple geographical areas); and those working at the regional pharmacosurveillance center or any department having a specific voluntary ADR reporting program (Figure 1).

We conducted a cluster-randomized controlled trial. To prevent cross-contamination between the intervention and control groups, 15 spatial clusters were used as units of assignment, rather than physicians. Each cluster consisted of 1 reference hospital plus the outpatient center and any other hospital that might be in its catchment area. Within each cluster were all physicians working at the general hospitals and outpatient centers in the selected geographic area.

For economic efficiency, the clusters were distributed by unequal randomization^20- 21 with an intervention:control-group ratio of approximately 1:3. Using a computer-generated procedure, 4 clusters were assigned to the intervention and 11 to the control group.

The educational intervention was based on a prior case-control study of the same population of physicians designed to identify attitudes associated with probability of ADR reporting.¹⁵ The categories of attitudes were based on a study by Inman,²² and included complacency, the belief that very serious adverse drug reactions are well documented by the time a drug is marketed; insecurity, the belief that it is nearly impossible to determine whether a drug is responsible for a particular adverse reaction; diffidence, the belief that reporting an adverse drug reaction would only be done if there was certainty that it was related to the use of a particular drug; indifference, the belief that the single case an individual physician might observe could not contribute to medical knowledge; and ignorance, the belief that it is only necessary to report serious or unexpected ADRs. Lack of time for form completion or diagnosis was also a reason associated with underreporting.

Based on this, we developed a continuing medical education multifaceted intervention,²³ comprising an outreach visit, reminder card, and report form. The outreach visit consisted of a 1-hour 2-part presentation that took place during weekly staff meetings to ensure that the greatest number of physicians could be present; the groups were made up of 10 to 20 physicians.

The first part of the presentation included definitions of pharmacosurveillance and ADRs; review of international studies on drug-related morbidity and mortality, hospital admissions, and the cost to health systems and patients; and description of the methods used in pharmacosurveillance and in spontaneous reporting systems, explaining that underreporting constituted the system's principal limitation. The second part addressed the 5 attitudes associated with underreporting, emphasizing that only 5 minutes are required to complete the report form (electronic presentation file is available on request). In addition, a reminder card similar to the report form and containing the principal messages of the presentation was distributed to approximately 90% of physicians attending the sessions.

For trials of educational interventions in Portugal, approval may be given by either the hospital teaching committee or hospital ethics committee at the discretion of the hospital clinical director. Ethics committee approval and individual informed consent are required only for clinical trials. Before each outreach visit, a letter was sent to the director of the selected outpatient center or the clinical director of the selected hospital, presenting the study objectives and requesting their consent to participate in research. In each case, the intervention was approved by the relevant hospital teaching committee and was included in each center's continuing medical education program.

The intervention visits were conducted from March 2004 through July 2004. The follow-up period began immediately after the intervention and ended in June 2005. The 5 outcomes assessed for each physician and each month of follow-up included an indicator of reporting quantity (total number of reports) plus the following 4 indicators of the quality (relevance) of the reports: number of serious ADRs (one that results in death; is life-threatening; is a congenital anomaly; requires hospital admission or prolongation of stay in hospital; or results in persistent or great disability, incapacity, or both)⁹; number of ADRs with attribution of definitive or probable causality^24- 26; number of unexpected ADRs (unknown ADRs that are not described in the summary of product characteristics)^26- 27; and number of ADRs concerning medications that have been on the market for fewer than 5 years.²⁶ All data came from the Northern Pharmacosurveillance Unit (under the Portuguese Health Authority) and were certified in accordance with World Health Organization guidelines. The Pharmacosurveillance Unit expert responsible for codifying adverse reactions (J.P.) was blinded to the physician study group assignment. Confidentiality was maintained, with data only being available for study purposes under a code number assigned to each physician that precluded any further identification.

Adverse drug reaction reporting is a passive process in which every report that is generated is received by the Northern Pharmacosurveillance Unit, which then furnished it to the researchers. Because of this, there was 100% assessment of ADR outcomes in the study population, and effectively no loss to follow-up. The only potential source of error would be if physicians in the study left clinical practice or died, and this information is not available. If this occurred, it would not have affected the accuracy of the number of ADR reports but could distort the per-physician rates.

The sample size was limited by the resources available. The preintervention monthly reporting rate among Portuguese physicians was estimated as 0.001 per month.²⁸ We considered a clinically meaningful increase in reporting to be 60 reports per 1000 physician-years (500% increase). This was chosen because the total reporting rate when combined with pharmacists' reports would approach that of countries with the highest reporting rates, such as those of the United Kingdom.²⁸ We calculated that with 15 clusters (mean of 400 physicians per cluster), the study would have a power of 80% (2-sided P<.05) to detect a 500% increase in reporting if 13 measures of the response variable were taken for each physician, with a correlation among these of 0.01.^21,29- 31 This assumed an overdispersion in the response variable of 10%, an intracluster correlation coefficient of 0.005, and an intervention:control cluster ratio of 1:3.

All statistical analyses were carried out on an intention-to-treat basis.^32- 33 Generalized linear mixed model, using penalized quasilikelihood, were applied to the statistical analysis.³⁴ This method allows for longitudinal data analysis adjusted for the baseline values of the dependent variable.

To construct the models, we used the number of ADR reports as the dependent variable, with individual observations (per month per physician) as level 1, physicians as level 2, and spatial clusters (as indicator variable) as level 3; random effects were considered, both among physicians and among spatial clusters. Because the dependent variable was a count outcome, a Poisson generalized linear mixed model was used. Models were adjusted for age, specialty, and work setting (hospital vs primary care). Because the Poisson assumption (that the mean and variance of the dependent variable are equal) was not met in our data, the models were adjusted taking the overdispersion parameter into account.³⁴ To measure the intervention effect, a dichotomous indicator variable was created. This variable (period) assumed a value of 0 for the baseline period and a value of 1 for months between the start of the intervention and the end of the follow-up. The intervention effect was evaluated on the basis of the interaction between the group (1 for intervention group, 0 for control group) and period variables.

For analysis of the duration of the effect, another indicator variable was constructed with 5 categories (0 for baseline period; 1 for the first 4-month period after intervention; and 2, 3, and 4 for the ensuing 4-month periods, respectively). The intervention effect in each 4-month period was evaluated on the basis of the interaction between this indicator variable and the group variable. The results of the generalized linear mixed model were validated by comparing them against results from comparable models obtained by running generalized estimating equations. Results are expressed as relative risks (RRs) and their 95% confidence intervals (CIs). The estimated absolute change in reporting rates attributable to intervention, adjusted for baseline rate, age, specialty, and work setting, were calculated by multivariate-adjusted risk differences using link identity in generalized estimating equation models.³⁵ Analyses were performed using S-Plus 6.2 (Insightful Corp, Seattle, Wash).

Of 6950 physicians in the northern health region, the following were excluded from the study: 35 for working as administrators, 5 for working at laboratory analysis centers, 24 for working in substance and rehabilitation centers, 2 for working at a pharmacosurveillance center, 1 for working at a protocol center, and 432 for working in specialty hospitals (Figure 1). The study population included 6451 physicians, of whom 1388 were randomly assigned to the intervention group and 5063 to the control group. A total of 655 (47.2%) of those in the intervention group attended the intervention visit. Follow-up duration was 13 months for 865 (62.3%), 14 months for 57 (4.1%), 15 months for 242 (17.5%), and 16 months for 224 (16.1%). The baseline characteristics of the intervention and control groups are shown in Table 1. There were statistically significant differences between the groups with respect to age, specialties, and work setting.

Compared with the control group, at baseline the intervention group had lower rates of reporting for all categories of ADR, measured as reports per 1000 physician-years; however, none of these were statistically significant (Table 2). Comparing total ADR reporting (per 1000 physician-years), the intervention group increased from 7.6 (95% CI, 4.0-12.6) at baseline to 100.2 (95% CI, 85.2-116.4) in the postintervention period, while the control group increased from 11.3 (95% CI, 8.9-14.1) to 14.5 (95% CI, 12.0-18.0), respectively (P<.001). When adjusted for baseline values, medical specialty, and work setting, the RR for total ADR reporting associated with the intervention was 10.23 (95% CI, 3.81-27.51; P<.001; Table 3, Model 1). The adjusted increase in the rate of total ADR reports attributable to intervention was 90.19 (95% CI, 54.51-125.87).

The monthly trend in total ADR reporting is shown in Figure 2 and Figure 3. The reporting rate in the intervention group increased sharply in the initial phase of the intervention, and the RR for total ADR reporting in the first 4-month period after intervention was 27.78 (95% CI, 8.36-92.23; P<.001; Table 3, Model 2). The magnitude of the effect decreased in subsequent periods, remaining statistically significant through the first 12 months after the intervention but nonsignificant in months 13 through 16.

The intervention was also associated with an increase in the percentage of physicians submitting reports. In the intervention group, this increased from 0.57% (95% CI, 0.14%-1.01%) to 5.25% (95% CI, 4.07%-6.50%), while the control group increased from 0.98% (95% CI, 0.70%-1.27%) to 1.01% (95% CI, 0.72%-1.29%). The difference between the groups was statistically significant (P<.001).

The effects of the intervention on the reporting rate of serious, high-causality, unexpected, and new-drug-related adverse drug reactions are also shown in Table 2, Table 3, and Figure 2 and Figure 3. The RR associated with intervention for serious ADRs was 6.32 (95% CI, 2.09-19.16; P = .001); for high-causality ADRs, 8.75 (95% CI, 3.05-25.07; P<.001); for unexpected ADRs, 30.21 (95% CI, 4.54-200.84; P<.001); and for new-drug-related ADRs, 8.04 (95% CI, 2.10-30.83; P = .002). Applying generalized estimating equations to these analyses provided very similar results. The adjusted increase in the rate of ADR reports attributable to intervention for serious ADRs was 30.16 (95% CI, 18.84-41.47); for high-causality ADRs, 64.90 (95% CI, 38.38-91.42); for unexpected ADRs, 28.04 (95% CI, 16.25-39.83); and for new-drug-related ADRs, 42.17 (95% CI: 21.58-62.76).

In this cluster-randomized controlled trial, physicians increased their ADR reporting rate 10-fold (95% CI, 3.81-27.51) in the year following an hour-long educational intervention. The effect was maximal in the first 4 months, but the attenuated effect remained significant up to 1 year after the intervention. It was no longer significant at 13 through 16 months, but relatively few physicians were followed up during that interval, during which time only a small number of ADRs were reported. The intervention also led to an improvement in the quality of reports by increasing the reporting rate for serious, high-causality, unexpected, and new-drug-related ADRs.

Although the CIs of our results are wide, the magnitude of changes in physician performance were much greater than those found in other studies of changing physician behavior.^36- 38 Although a number of factors may account for the effect of an intervention aimed at changing professional behavior,^{23,36,39- 41} ADRs as the target of our intervention³⁹ may have been important in our trial and may also account for the high effect magnitudes reported by other investigators who addressed this same topic in uncontrolled studies.^16- 19 Other studies have used multiple interventions, such as mailings,^17- 19 newsletters,^17- 18 oral presentations,^17- 18 verbal reminders,¹⁹ articles in staff newsletters,^17- 18 advertisements,¹⁷ and coordination between physician and the hospital pharmacist.^16,19 In our trial, the only cointerventions were a low-cost information leaflet and report form, which can act both as a facilitating factor and as a reminder.⁴²

The intervention was expressly designed to address knowledge and attitude gaps that had been previously identified in the same target population,¹⁵ which permits specific, concrete, and more effective messages to be developed.³⁹ We designed the intervention to be as interactive as possible. Furthermore, these messages were delivered by means of an interactive outreach visit, which may increase effectiveness.⁴⁰

Although the relative increase in reporting rates was high, the absolute magnitude of the changes must be considered in evaluating the clinical importance of the results. The World Health Organization has recommended a reporting rate of 200 to 300 per million population per year.⁴³ If the absolute improvement achieved in our trial were to be applied to the entire Northern Region of Portugal, the physician-based reporting rate would increase from 26 to 160 reports per million population per year. Combining these with the reports made by pharmacists (25 per million population in the study area) would reach the World Health Organization target. Countries, such as the United States, Canada, Germany, and Italy, have low numbers of ADR reports submitted directly to national drug agencies by physicians.⁴⁴ If the absolute rate increase that we found were to hold in these countries, this would represent a significant improvement. This may also be true in countries such as Sweden and the United Kingdom, which currently have greater absolute reporting rates but where underreporting is very high.^45- 46

The effects of educational interventions may diminish over time. Other studies have found behavioral changes that persist from 9 months to 2 years.⁴⁷ In our study, the differences between groups lasted for 12 months. This raises a question of whether periodically repeating an intervention would have a booster effect or whether the target group would show a lesser or absent response on reexposure. Our study does not provide that information.

Our study design has a number of strengths. The use of a control group served to eliminate potential sources of bias, such as seasonal variation or increases in reporting due to vaccination campaigns, and served to minimize the effects of secular changes in behavior.⁴⁸ These potential biases have limited the interpretation of other before-and-after studies of this issue.^16- 19 Randomization minimizes the potential for selection bias, and a cluster-based distribution reduces the risk of cross-contamination between groups. Imbalances in baseline group composition were addressed by adjusting for variables that were unequally distributed, as well as adjusting for baseline differences in the dependent variables and comparing the changes in the intervention group against those in the control group.^49- 50

Important limitations also must be considered in interpreting the results of our study. First, we did not have information on some potentially relevant factors, such as medical school attended, years since graduation, and postgraduate degrees or training of participating physicians; we were therefore unable to include these in our models. Similarly, we had no data on the possible right censoring of physicians who died or became clinically inactive during the follow-up. However, we believe that it is extremely unlikely that more than a very small percentage of the study population would have died or left practice during the 16-month followup. Moreover, the percentage should have been similar in both the intervention and control groups, biasing the results toward a null effect.

Second, only 47.2% of the physicians assigned to the intervention group actually attended the outreach visit. However, because we analyzed the data on an intention-to-treat basis, the significant differences found between groups cannot be explained by a bias related to low attendance. The true potential effect of this intervention might be greater if attendance could be increased, although our results may reflect its pragmatic effectiveness.³³ The other consideration of intention-to-treat analysis is that if any cross-contamination between groups actually occurred, the analysis would have been biased toward a null effect. Because we found significant differences between groups, the true effect would be at least as strong as what we reported, and possibly stronger.

Third, there may be limited generalizability of these findings. The intervention was tailored to the specific study population and may not be as effective in a different group. The positive results may reflect in part unique aspects of the Portuguese health care system or the regional physician culture. Willingness to report ADRs may be less in countries in which there is greater concern about malpractice liability.

Fourth, it is difficult to know exactly what the target number of ADRs, or percentage of physicians reporting ADRs, should be in the absence of information about the true number of adverse events that are occurring. It is therefore difficult to judge the degree of success represented by our results.

Finally, more reporting may not necessarily be better. Although we measured ADRs that met accepted criteria for high quality, we cannot tell from this study the effect that any of these had on clinical care. In addition to identifying new and possibly serious drug effects, such a program has the potential for creating false alarms. The cost-effectiveness of programs such as ours, including all of the possible downstream benefits and harms, must be considered, but we did not evaluate that; it should be the subject of future investigation.

However, recent cases of drug withdrawals have highlighted concerns about drug-safety monitoring,^51- 52 and emphasize the importance of postmarketing surveillance. Direct reporting by physicians has proved successful in identifying new ADRs.^5- 6 Our study shows that physicians can respond to short outreach visits designed to increase reporting. If these results can be replicated over time in other settings, many countries may be able to substantially enhance ADR reporting—in terms of both volume and relevance—by means of educational strategies designed to meet specific physicians' training needs, and thereby improve drug-safety monitoring.