Completeness of Safety Reporting in Randomized Trials: Title and subTitle BreakAn Evaluation of 7 Medical Areas FREE

Randomized trials with adequate sample size offer a unique opportunity to assess the frequency and severity of adverse events in a controlled and objective setting, with the most comprehensive and systematic accumulation of pertinent information. Such information is important in estimating benefit-harm ratios in the application of medical interventions. However, compared with the heightened scrutiny of the conduct, analysis, and reporting of randomized trials in general,^{1 - 3} assessment of the reporting of adverse events and toxicity has only recently drawn some attention.

In a preliminary analysis of the quality of safety reporting, we observed that toxicity data may sometimes be presented erratically or may be missing altogether.⁴ A limitation of this preliminary work was that it considered trials in only 1 medical domain, ie, drug therapy for human immunodeficiency virus (HIV) infection. However, the adequacy of safety reporting may be different in other medical areas. Therefore, in the present study, we extended our evaluation of safety reporting to 7 different areas of drug therapy. In doing so, we also sought to understand the settings and predictors that lead to suboptimal safety reporting, and to gain insight for improving these important deficiencies.

Randomized trials of drug therapies qualified for the analysis if they had a sample size of at least 100 patients and at least 50 patients allocated to a study arm. Smaller trials give very uncertain estimates for even the most frequent adverse events, and may completely miss even relatively common toxicity. With 100 patients, when no subjects are observed to experience or report a specific adverse effect, the upper limit of the 95% confidence interval (CI) for the true frequency of this unobserved effect is still 3%. With 50 patients assigned to an experimental arm, the upper limit of the 95% CI is as high as 6%.⁵

We analyzed safety reporting in randomized drug trials in the following 7 medical areas: (1) HIV therapy (all therapeutic trials, excluding immunization [passive immunotherapy, vaccines] and treatment of complications); (2) antibiotic therapy for acute sinusitis (all comparisons of antibiotics among themselves or with placebo or no therapy); (3) thrombolysis for acute myocardial infarction (AMI) (all comparisons of different thrombolytic regimens against placebo or no therapy); (4) use of nonsteroidal anti-inflammatory drugs (NSAIDs) for rheumatoid arthritis (comparisons of NSAIDs among themselves or with placebo or no therapy); (5) treatment of hypertension in elderly persons (all comparisons of antihypertensive regimens among themselves or with placebo or no therapy in this age group); (6) treatment of Helicobacter pylori with antibiotics (all comparisons of any antibiotic regimen until 1994 and all comparisons involving proton-pump inhibitors [omeprazole] with antibiotics until 1999); and (7) selective decontamination of the gastrointestinal tract (SDGIT) (all comparisons against placebo or no therapy).

These topics represent a diverse set of medical questions that have a significant public health impact (ie, the diseases are very common and/or have major morbidity). Also, a comprehensive list of the pertinent trials would be easily retrievable from systematic databases and/or meta-analyses performed by either our group or prior investigators. We purposely included topics spanning the acute and chronic care settings and both inpatient and outpatient settings to maximize generalizability. However, we focused on drug therapies and excluded surgical interventions and vaccines, since safety may be assessed differently for such interventions.

We identified the pertinent trials for each topic using several sources. For the HIV, sinusitis, AMI, hypertension, and H pylori topics, we used comprehensive databases of randomized controlled trials that have been developed by members of our research group as part of previous work on evidence reports, meta-analyses, and methodologic studies.^{4 ,6 - 11} Typically, MEDLINE and EMBASE searches using the names of specific medications and a large array of terms pertaining to randomized trials were complemented with hand-searching of journals specializing in the given area, identification of trials referenced in retrieved publications, and communications with experts. For these 5 topics, databases cover until late 1997 or early 1998 (1999 for H pylori). For the H pylori topic, we also retrieved trials from a meta-analysis published by a different team that addressed all trials of antibiotic regimens published until 1994.¹²

For the rheumatoid arthritis and SDGIT topics, we used the trial databases from meta-analyses published by other teams.^{13 - 15} These cover randomized trials published until 1993^{13 - 14} and 1989,¹⁵ respectively. For the topic of hypertension in the elderly, we also consulted the respective systematic review in the Cochrane Library.¹⁶ We did not systematically update these meta-analyses, since the aim of the project was not a thorough meta-analysis to estimate the treatment effects based on the most current data.

We decided to use both qualitative and quantitative components of adverse event reporting; these components may offer complementary information. The list of these parameters was originally developed for the evaluation of HIV-related drug trials.⁴ Although it is difficult to specify which aspects of safety reporting are most important, some aspects are probably indispensable, if the reported information is to be used and interpreted for clinical purposes. First, data should be given with numbers. Generic statements (eg, "few patients had side effects") cannot be objectively appraised and may be misinterpreted. Second, the severity of adverse effects should be stated and, at minimum, the frequency of severe or life-threatening toxic events should be provided per study arm. Standardized scales for grading toxicity and definitions for severity are both important for this purpose. Third, data should be given separately for each specific type of severe adverse effect so clinicians can determine what kind of harm is involved.

Based on the above, we selected the following 2 qualitative components: (1) whether the number of withdrawals and discontinuations of study treatment due to toxicity are reported, and whether the number was given for each specific type of adverse effect leading to withdrawal; and (2) whether the severity of the described clinical adverse events or abnormalities of laboratory tests (laboratory-determined toxicity) were adequately defined, only partially defined, or inadequately defined.

Adequate definition of severity requires either detailed description of the severity or reference to a known scale of toxicity severity (typically with grades being 1 = mild, 2 = moderate, 3 = severe, 4 = life-threatening), with separate reporting of at least severe or life-threatening events. At least 2 adverse effects (clinical or laboratory) have to be defined in this way, with numbers or rates given for each study arm.

Partial definition of severity means that reports of severity combine moderate with severe or life-threatening toxicity counts, or that the number of severe or life-threatening toxicity cases are separately specified for only 1 of many reported clinical adverse events and laboratory abnormalities per study arm.

Inadequate definition of severity includes protocols reporting the total number of severe clinical or laboratory-determined toxic effects without giving numbers on specific types of adverse events per arm, those that lumped numbers for all grades of toxicity without separating any grades for any specific adverse events, those providing only generic statements, and those not reporting adverse effects at all. The common characteristic of all these situations is that information is missing on the frequency of severe adverse effects—information that is directly relevant for the estimation of benefit-harm ratios.

Quantitative measures assess the relative emphasis given to safety in the results of published trials. We specified these measures as the extent of space (in printed pages) devoted to safety in the results section, and the proportion it represents of the whole results section; and the space devoted to safety as compared with the space devoted to the names and affiliations of authors, participants, and contributors in the same trial report. The space allocated to toxicity is not necessarily correlated with the quality of reporting, but it complements the qualitative assessment, because it is an objective estimate of the relative importance that safety has in the overall clinical trial report. In our study, we measured the space for each section with a resolution of 0.05 page. When there were N columns per page and a printed page had a length of Y centimeters, a section spanning a length of S centimeters in 1 such column was calculated as occupying S/(N × Y) pages. Y refers to the printed area, excluding upper and lower margins.

We collected information on trial parameters that may have affected safety reporting. In particular, we wanted to evaluate: (1) whether dose-comparison studies may place more emphasis on safety; (2) whether high-impact journals,¹⁷ and articles with significant results for efficacy, used less space for safety; (3) whether safety reporting was given more emphasis in larger trials, long-term trials, or masked trials; (4) whether sponsorship, type of population, and location of the trial affected reporting; (5) whether safety was less emphasized for drugs that had already been used for a different indication; and (6) whether the first trial for a new indication devoted more emphasis to safety. Furthermore, we evaluated whether the situation improved over time. In making these evaluations, we noted several trial characteristics (Table 1). Additional information about the characteristics of the considered trials, the data extracted from the trials, and the trials included in the database is available at: http://www.nemc.org/dccr/Projects/safetyreporting/supplements.htm.

All characteristics listed in Table 1 were also used as predictors in least-squares regression models using the percentage of space devoted to safety reporting in the results section as the dependent variable. We estimated univariate models for each predictor adjusting for medical area (with dummy variables for medical areas). We also considered interaction terms between covariates and medical areas, but they did not improve model fit substantially. Multivariate models were also considered, either using all variables that were significant (P<.05) in univariate analyses by forced entry, or starting from all variables with P<.25 on the univariate analyses and stepwise eliminating variables with P>.10 in the resulting models. The results were similar when univariate regressions were performed separately in each field and regression coefficients for each predictor were combined with general variance models.¹⁸

We also performed logistic regressions to identify predictors of adequate reporting of clinical adverse events and laboratory-determined toxicity. All analyses were adjusted for medical area using dummy variables. Both univariate and multivariate models were considered. Again, we reached identical final results, whether considering multivariate models using all variables that were significant (P<.05) in univariate analyses by forced entry, or starting from all variables with P<.25 on the univariate analyses and stepwise eliminating variables with P>.10 in the resulting models.

Two independent data extractors separately evaluated the 60 trial reports of HIV drug therapy. We observed very high interrater agreement. For the assessment of the adequacy of reporting of clinical adverse effects and laboratory-defined toxicity, the κ coefficients were 0.72 and 0.85, respectively. There were no instances in which 1 extractor considered the reporting adequate and the other inadequate. Moreover, we observed no important discrepancies in the data extraction of quantitative parameters. Thereafter, 1 reviewer examined trial reports on the other 6 areas.

Analyses were performed in SPSS (SPSS Inc, Chicago, Ill). P values are 2-tailed.

A total of 192 trials from 7 different medical areas (N = 130 074 patients) were included (Table 1). Trial characteristics differed across the selected areas. Trials with a sample size of more than 1000 patients had been performed only in HIV therapy and thrombolysis for AMI. A total of 117 trials were double blind, but the percentage of double-blind trials across areas varied from 37% to 100%. Trials involving dose comparisons were available in 4 areas. The large majority of the trials on acute sinusitis showed no statistically significant differences for efficacy, while statistically significant results were more common than nonsignificant results in the other areas. The percentage of trials with government funding varied from 0% to 60%. The percentage of trials with long-term follow-up ranged from 0% to 100%. Children were evaluated in the areas of HIV and sinusitis. There was wide variability in the proportion of trials where the drugs had been already used for another indication (0% to 100%). Trials had been conducted both in the United States and elsewhere and they covered a wide range of publication years. The percentage of trials published in journals with an impact factor higher than 7 ranged from 0% to 60% across medical areas. Overall, in accordance with our aim, the large diversity in trial characteristics across medical areas ensured the generalizability of the results.

Only 39% of trials had adequate reporting of clinical adverse effects and only 29% had adequate reporting of laboratory-determined toxicity. A further 11% (clinical adverse effects) and 8% (laboratory-determined toxicity) had partially adequate reporting. The numbers of discontinuations due to toxicity per study arm were mentioned in 75% of the trial reports, but specific reasons for these discontinuations were given only 46% of the time. For all these outcomes, there was statistically significant heterogeneity for the rates of adequate reporting (vs partially adequate and inadequate reporting combined) between the 7 medical areas (Table 2).

Of the 95 trials with inadequate reporting of clinical adverse events, 3 gave the total number of serious or life-threatening events but failed to specify their types, 52 gave numbers for various adverse effects but without separating severe adverse events, 11 only offered generic statements without specific numbers, and 29 did not report specifically on clinical adverse effects (although 12 of these mentioned discontinuations due to adverse effects, but no other relevant information). Among the 121 trials with inadequate reporting of laboratory-determined toxicity, 1 gave the total number of serious or worse toxicity but failed to specify its types, 8 gave numbers for various adverse effects but without separating severe toxicity, 16 offered generic statements without specific numbers, and 96 did not report anything on toxicity.

The absolute space allocated to safety information was limited (median, 0.3 page; interquartile range, 0.1-0.7 page). The median was less than half a page in all areas except for arthritis trials (Table 3). A similar picture emerged when we studied the percentage of the results dedicated to safety. Overall, the space given to safety information was the same as or less than the space given for the names of authors and their affiliations. In 92 trials the authors/affiliations space was larger than the safety space, in 21 trials it was similar (within 0.05 of the length of a page), and in 79 trials safety reporting took more space (P = .16 by Wilcoxon test). Safety reporting took more space than the names of authors in trials of therapy for sinusitis (P<.001) and rheumatoid arthritis (P<.001), but it took less space than the names of authors in trials of SDGIT (P = .002) and HIV therapy (P = .06). More than half the trials included at least 1 table for safety information, while only 5% of reports included figures for such information.

In univariate analyses, the percentage of space in the results section devoted to safety was significantly larger in trials also making dose comparisons; similarly, the amount of space was larger in trials involving only dose comparisons. Conversely, emphasis on safety decreased when the trial found statistically significant results for efficacy (Table 4). There were also trends for more emphasis on safety in double-blind trials, and less emphasis on safety in trials studying drugs with a prior indication, but these were not significant. The results of multivariate models were consistent with the univariate findings (Table 4).

In univariate regressions, the odds of adequate reporting of clinical adverse effects was 2.83-fold higher (95% CI, 1.34-6.00) in double-blind trials vs single-blind or unmasked trials, and it increased 4.14-fold (95% CI, 1.57-10.9) for each 10-fold increase in sample size. It also improved over time (increased 1.07-fold every year, [95% CI, 1.00-1.14] ). On the contrary, long-term trials were probably less likely to have adequate reporting of clinical adverse events than short-term trials (odds ratio [OR], 0.40; 95% CI, 0.16-1.01). Trends were also observed for better clinical reporting in trials in which dose comparisons were involved (OR, 1.67; 0.73-3.81), and worse clinical reporting in trials where there was already a prior indication for the tested medication (OR, 0.50; 95% CI, 0.18-1.39). The results of the final multivariate model were similar (double-blinding: OR, 2.51, 95% CI, 1.13-5.57; per 10-fold increase in sample size: OR, 4.52, 95% CI, 1.51-13.6; per year: OR, 1.06, 95% CI, 0.99-1.13; long-term trials: OR, 0.27, 95% CI, 0.10-0.74).

Adequate reporting of laboratory-determined toxicity was less likely when there was a prior indication for the studied medication (OR, 0.33; 95% CI, 0.13-0.88) and possibly when the efficacy results reached statistical significance (OR, 0.61; 95% CI, 0.26-1.42) and in pediatric trials (OR, 0.39; 95% CI, 0.09-1.74). Adequate reporting of laboratory-determined toxicity was more likely in trials performed mostly in the United States (OR, 2.29; 95% CI, 1.04-5.04) and possibly when there was government funding (OR, 1.87; 95% CI, 0.77-4.57). In multivariate modeling, prior indication (OR, 0.33; 95% CI, 0.12-0.90) and US location (OR, 2.29; 95% CI, 1.03-5.10) remained significant independent predictors.

An evaluation of safety reporting in randomized trials across 7 different medical areas proves that safety reporting is often inadequate and neglected. Key information that would take minimal space to report is often missing. The extent of neglect varies significantly across medical areas. However, in the 7 medical areas we examined, we found no instances where safety reporting can be deemed satisfactory. With 1 exception, safety reporting takes less than a half page in the average trial report; at least as much space is taken by the listing of the names and affiliations of the trial contributors and authors. Safety gets more space in trials in which dose comparisons are involved. This is not surprising, since such trials usually aim to show that a certain dose has similar efficacy, but shows superior tolerability and fewer adverse effects. Otherwise, adverse effects are even more neglected in trials that report statistically significant results for efficacy.

Adequate reporting of clinical adverse events was seen in only 39% of trial reports. Many trials reported clinical adverse events without distinction of severity. There was some evidence that the situation may be improving over time, and that double-blind studies and large studies may pay more attention to clinical adverse events. Long-term trials were less likely to report such events adequately, perhaps because long-term trials are conducted with a strong emphasis on clinical efficacy rather than safety outcomes.

Adequate reporting of laboratory-determined toxicity occurred only in 29% of the trial reports. Half of the trial reports failed to mention laboratory-determined toxicity altogether. Trials of drugs with prior indications fared significantly worse in this aspect, just as they did for the reporting of clinical adverse events. For drugs with prior established indications, authors may not feel compelled to repeat what might be considered standard knowledge, even if the population and indication under study are different. The better reporting of laboratory-determined toxicity from trials performed in the United States is more difficult to explain. If not a chance finding due to the large number of associations examined in our study, it could reflect a difference in reporting across continents or a difference in the phase of trials performed in the United States vs other countries.

Our figures may be overestimating the actual emphasis that is given to safety as compared with efficacy. While further subsequent publications for efficacy involving subgroup analyses, surrogate marker analyses, and specialized analyses are very frequent in pivotal randomized trials, we could not identify other follow-up publications from these trials focusing comprehensively on toxicity results (with 2 exceptions in the area of HIV therapy and 1 exception in thrombolysis for AMI). Moreover, since we examined 7 different medical areas, the results of our evaluation are likely to be generalizable across medical specialties. Other investigators also have presented preliminary data that safety reporting is neglected in trials of anesthesia and pain management.¹⁹

We should acknowledge that comparative clinical trials such as those included in our evaluation have several limitations in providing information about medication adverse effects. They are unlikely to reveal uncommon but important adverse effects occurring in fewer than 1 in 1000 patients. Important adverse events are often recognized many years after a medication has passed the clinical trial stage and has been used extensively in the community.²⁰ Nevertheless, randomized clinical trials of adequate sample size offer the best (and only) opportunity for assessing the frequency and severity of common side effects from a new medication in a controlled setting.

While there have been major strides in standardizing the collection, analysis, and reporting of efficacy data in clinical trials,^{1 - 3} efforts to assess and improve the quality of analysis and reporting of safety data are lagging behind. This important deficiency needs to be corrected, if we wish to use quantitative objective evidence both for the efficacy and the toxicity of specific treatments in making therapeutic decisions. Such information may complement the data obtained from postmarketing reporting.²¹ Postmarketing reporting is very important, but is highly dependent on the reporting of adverse events; such reporting can be spontaneous, sporadic, and erratic. Standardization of safety reporting may allow the performance of more reliable meta-analyses of safety information that may complement the meta-analyses performed on efficacy parameters. Meta-analysis of toxicity data has been hampered in the past because of inadequate safety reporting, as also admitted by other authors.²²

In our experience, most high-quality trials amass an enormous amount of information about safety and adverse effects during their conduct as part of regulatory requirements. Yet, the selective filtering of all these data into a quarter of a page can hardly be adequate. The simple storage of such information in company archives does not help the educated clinician who wants to critically interpret the efficacy vs toxicity data in each large trial considering the specific trial population, dosing, concomitant medications used, setting, study design, and other factors. The reliability and generalizability of the information conveyed in medication brochures is uncertain and cannot be critically evaluated.

The set of parameters we have developed to evaluate safety reporting offers a standardized evaluation tool. It may complement the CONSORT statement that has been developed in an effort to standardize the reporting of the design and efficacy outcomes of randomized clinical trials.² Descriptors for such an addendum to the CONSORT statement might be summarized as follows: (a) Specify the number of patients withdrawn from the study because of adverse effects, per study arm and per type of adverse effect; (b) Provide the number of specific adverse effects per study arm and per type of adverse effect. Give exact numbers, especially for high-grade (severe) clinical adverse events and laboratory-determined toxicity; and (c) Tabulation of safety information per study arm and severity grade is encouraged, as well as detailed description of cases of unusual or previously unrecorded adverse effects. This addendum may be used in future research and may offer a guide to investigators and journals for the concise, focused reporting of adverse effects and toxicity.