Higher Standards for Epidemiologic Studies—Replication Prior to Publication?

To the Editor: In his commentary on epidemiologic studies and meta-analysis in drug development and surveillance, Dr Temple¹ suggests that medical journals impose 3 higher standards on epidemiologic studies. Two of these should invoke little concern. Who is not in favor of any scientist doing a better job describing methods and their limitations? However, Temple's third suggestion—that epidemiologic studies with relative risks of 2.0 or less not be published until replicated in a different environment—is alarming.

There are 3 reasons that mandating replication prior to publication is a bad idea. First, doing so flies in the face of the central ethical obligation of epidemiologists to report their findings to society, peers, and study participants.² Second, it trades one known bias—publication bias—for others presumed but unknown. Third, it would be a logistical nightmare.

Each deserves elaboration, but equally compelling is a fourth reason, emerging from the idea that singles out epidemiologic results as unreliable. For the record, Temple's views have changed on this account. In 1995, he asserted that his "basic rule is if the relative risk isn't at least three or four, (then) forget it."³ Why he is now willing to ratchet down his minimum level of acceptability to 2.0 is not so much mysterious as it is reflective of the subjective nature of claims about the causal relevance of so-called weak associations.

Modest relative risks are not be considered weak evidence, because we have carefully examined the historical record, as Temple realizes. Neither is it true that theoretically important causal hypotheses necessarily imply large relative risks; the study of biology always has admitted small effects observed over the long run. Modest relative risks evoke concern because unknown and thus unmeasured confounders and biases lurk in the recesses of unexplained variability that surround all deliberate observations of nature. Nevertheless, to maintain that smaller magnitudes of associations are somehow more subject to these unknown disturbances than larger values, we must assume either that perturbations in the opposite (non-null) direction would not (by the same logic) be more likely to affect weak associations, or that we are experiencing an "era of weak associations" in which nothing but increasingly weak associations are discovered, making disruptions of large magnitudes increasingly unlikely. If either assumption fades in the light of historical inquiry, then so-called weak associations are stronger than we think. For these 4 reasons, singling out epidemiology for a new policy of "no publication without replication" is unwarranted and unreasonable.