Comparison of Effect Sizes Associated With Biomarkers Reported in Highly Cited Individual Articles and in Subsequent Meta-analyses FREE

Many new biomarkers are continuously proposed^1- 3 as potential determinants of disease risk, prognosis, or response to treatment. The plethora of statistically significant associations^4- 5 increases expectations for improvements in risk appraisal.⁶ However, many markers get evaluated only in 1 or a few studies.⁷ Among those evaluated more extensively, few reach clinical practice.⁸

This translational attrition requires better study. Are the effect sizes proposed in the literature accurate or overestimated?⁹ It is interesting to address this question in particular for biomarker studies that are highly cited. Many of these risk factors are also evaluated in meta-analyses¹⁰ that allow overviews of the evidence. However, some meta-analyses may suffer bias from selective reporting, especially among small data sets^11- 13; then large studies may provide more unbiased evidence.

Here, we examined biomarkers that had been evaluated in at least 1 highly cited study and for which at least 1 meta-analysis had been performed for that same association. We aimed to compare the effect size of these associations in the most highly cited studies vs what was observed in the largest studies and the corresponding meta-analyses.

We considered biomarkers that had a relative risk (RR) estimate presented numerically in the abstract of an article that had received more than 400 citations in ISI Web of Science until December 2010. The threshold of 400 citations was decided a priori, to target approximately the top 3% of biomarker studies published in influential journals. Of those, we focused further on biomarkers with published meta-analyses on the same association. Eligible RR metrics included risk ratios, odds ratios, and hazard ratios. Log odds scores were excluded.

We considered all markers of disease risk, prognosis, or treatment response representing body fluid, tissue, or imaging measurements. We did not consider demographic, anthropometric, social, environmental, psychological, and behavioral factors unless represented by a fluid, tissue, or imaging measurement. We did not consider associations with other markers and nonclinical outcomes.

Meta-analyses were eligible if they had addressed the same association (same marker, same outcome) as the one highlighted in the abstract of the highly cited study. When several eligible RR estimates appeared in the abstract of the highly cited study, we selected the one presented first.

Screening citation counts for all biomarker studies is extremely difficult because there are probably more than 100 000 studies published to date. Instead, we focused our searches on 24 nonreview biomedical journals that receive very high numbers of citations (per 2009 Journal Citation Reports) and that may publish on new biomarkers. These journals are (in decreasing total citations) Nature, PNAS, Science, New England Journal of Medicine, Cell, Lancet, Circulation, Cancer Research, Blood, Journal of Immunology, JAMA, Journal of Clinical Oncology, Journal of Clinical Investigation, Neurology, BMJ, Nature Genetics, Journal of Experimental Medicine, Journal of Clinical Endocrinology and Metabolism, Journal of the American College of Cardiology, Cancer, Gastroenterology, Clinical Cancer Research, Pediatrics, and Journal of the National Cancer Institute (JNCI).

We searched for potentially eligible highly cited studies in ISI Web of Science (last update December 26, 2010) using the search risk ratio OR relative risk OR odds ratio OR hazard NOT random*. The last item aimed to exclude randomized trials, which use RRs predominantly for treatment effects rather than biomarker associations. When 2 or more highly cited articles from the same cohort or study were identified, we included only the one published earlier. To test the sensitivity of the search strategy, we also searched without any journal limits for articles published in 2002: 9326 items were retrieved for that calendar year alone, vs 696 with the search limited to the 24 selected journals; the journal-unlimited search yielded 12 eligible highly cited biomarker studies, 10 of which (83%) were from the 24 journals.

For each potentially eligible highly cited biomarker study, we searched PubMed (last update January 3, 2011) for meta-analyses of the selected association highlighted in the abstract of the highly cited study (same biomarker and outcome). The search used the biomarker name and synonyms, limited to publication type = “meta-analysis.” Whenever multiple eligible meta-analyses existed, we identified whichever included the most studies. In each meta-analysis, we also identified the largest study (the one with greatest weight [smallest variance]).

For each highly cited study, we extracted information on journal, publication year, number of citations, and information on the selected RR: the biomarker of interest, condition/outcome implicated, sample size and number of outcome events, RR estimate, and corresponding exposure contrast.

For each meta-analysis, we extracted information on the journal; first author; total sample size and number of events; sample size and number of events in the largest study; the years of the published studies; summary RR estimate in the meta-analysis and 95% confidence interval (CI) by random effects¹⁴ (or fixed-effect calculations, if random effects were not presented at all); RR in the largest study and its 95% CI; RR in the index highly cited study and its 95% CI, as given in the meta-analysis; and type of RR and exposure contrast used in the meta-analysis. When RR estimates and CIs were only shown graphically in forest plots, we perused the corresponding primary study publications. If this remained unclear, we used the open-source software Engauge Digitizer (version 4.1) to extract numbers from the graphically presented information.

Biomarkers are sometimes tested independently, but their true clinical value is best appreciated when one can document whether they confer incremental information beyond what other variables and other biomarkers can provide. We examined all the evaluated highly cited studies and all the respective meta-analyses to record how many used unadjusted effects, how many provided adjusted effects adjusting for factors other than biomarkers, and how many adjusted also for other biomarkers.

Finally, we noted whether the RR estimate and 95% CI for the highly cited study as extracted from each meta-analysis were identical to those reported in the abstract of the highly cited study. Whenever not identical, we noted whether this was due to differences in exposure contrast, adjusting covariates, or sample size (a larger or smaller data set included in the meta-analysis than in the original highly cited publication). For comparison against the largest study and the meta-analysis results, we used for consistency the RR estimate of the highly cited study as reported in the meta-analysis. Whenever a meta-analysis did not include the highly cited study, it was not considered eligible, because it addressed potentially a different question than the highly cited study. However, we accepted as eligible those meta-analyses that (1) had replaced the highly cited study in their calculations with data from a larger data set that included the data from the highly cited study; (2) had excluded the highly cited study because of some technical issue (eg, Hardy-Weinberg violation in a genetic study), but the data addressed the same question; or (3) presented collaborative meta-analyses of individual participant data that did not have the raw data from the highly cited study even though it addressed the same question. In these 3 cases, for comparison against the largest study and meta-analysis result, we used the RR estimate of the highly cited study as reported in its original abstract. When the exposure contrast of categorical variables was different between the highly cited study and the corresponding meta-analysis, the RR was approximated for the same contrast as used in the meta-analysis, after converting exposure contrasts to standard deviation equivalents.

Data extraction was performed independently by 2 investigators. Discrepancies were discussed to reach consensus.

We compared the magnitude of the effect sizes (RRs) in the highly cited study against the meta-analysis and against the largest study using the same exposure contrast. Exposures were coined consistently to represent RR values greater than 1.00 in the highly cited study, eg, if a highly cited study gave an RR of 1.5 for above vs below median values of the biomarker, this was coined to become an RR of 0.67 for below vs above median values of the biomarker. We estimated for how many associations the RRs were in opposite direction, larger, more than twice as large, more than 4 times as large, or different beyond chance in the highly cited vs the largest study and in the highly cited study vs the meta-analysis. We also calculated the ratio of the RR estimates (relative relative risk [RRR]) and their 95% CIs. All analyses were conducted in Stata version 10.1 (Stata Corp, College Station, Texas). All P values are 2-tailed.

Among 14 025 articles, 377 had received more than 400 citations. Of those, we identified 113 highly cited biomarker studies that listed at least 1 RR for a biomarker in the abstract.

Of the 113 studies, 13 were meta-analyses by themselves and 100 were primary studies. We identified published systematic reviews (n = 44) that could potentially correspond to 61 of the 100 associations. On further scrutiny, 26 associations were excluded, because the RR in the abstract was possibly wrong and could not be identified in the full text of the highly cited article (n = 1), systematic reviews did not provide any summary estimate for the eligible association (n = 6), another highly cited study from the same population cohort had been published earlier (n = 1), the highly cited study addressed different risk factors (n = 6) or different outcomes (n = 3) or had a different design (n = 4) from the studies considered eligible for the meta-analysis, or no study-specific RRs could be retrieved from the meta-analysis (n = 5). Therefore, 35 highly cited studies^15- 49 published between 1991 and 2006 remained eligible (Table 1). For each of 2 associations (C-reactive protein and coronary heart disease, Helicobacter pylori and gastric cancer) there were 3 eligible highly cited studies; thus, 31 independent meta-analyses^50- 80 were considered.

The 35 eligible highly cited articles had received a median of 645 (interquartile range [IQR], 526-1054) citations vs 609 (IQR, 503-804) for the 65 excluded articles (P = .83). The median publication year of eligible articles was 1996 (IQR, 1995-2000) vs 1998 (IQR, 1995-2001) for those excluded (P = .07).

The highly cited biomarkers (Table 1) included genetic risk factors (n = 11 associations), blood proteins (n = 3), other blood biomarkers (n = 8), infectious agent biomarkers (n = 6), and others (n = 7). Cancer-related (n = 14) and cardiovascular-related (n = 12) outcomes predominated. The median sample size for the 35 highly cited studies was 518 (IQR, 218-1327), and median number of events was 197 (IQR, 103-504). The median RR was 2.50 (IQR, 1.70-4.08). Diverse exposure contrasts were involved (Table 1). The 35 studies were published in 10 different journals (New England Journal of Medicine, n = 13 studies; JAMA, n = 4; JNCI, n = 4; Nature, n = 3; Lancet, n = 3; other, n = 8).

Thirty-one of the 35 associations were statistically significant, while 4 were not (ACE deletion-insertion polymorphism and ischemic heart disease,²⁹ penicillin resistance and pneumococcal pneumonia mortality,³² insulinlike growth factor 1 levels and breast cancer in postmenopausal women,⁴⁰ and total estradiol levels and breast cancer⁴³). In 3 of these 4, other statistically significant associations were also reported in the abstract of the same highly cited study.

Table 2 shows the characteristics of the meta-analyses corresponding to the eligible highly cited studies. These meta-analyses, published between 1998 and 2010, included a median of 24 (IQR, 12-42) primary studies. Five were meta-analyses of individual patient data.^{57,65,68,71,74} For 18 associations, the highly cited was published early (within the first 2 years in the accumulation of evidence), while in the other 17 it was published later (median, 6 years [IQR, 5-13] in the accumulation of evidence). The median reported sample size was 12 128 (IQR, 4267-30 650); median number of events was 4790 (IQR, 2862-10 451).

The median sample size of the largest studies was 1820 (IQR, 721-5457) and median number of events was 509 (IQR, 123-1121). In 3 cases (C-reactive protein levels and coronary heart disease,²⁵ NS5A_2209-2248 [amino acid sequence 2209 to 2248 of nonstructural protein 5A] and interferon response in hepatitis C virus infection,³¹ and bone mineral density and fracture³⁷), the largest study was the highly cited one. Excluding these 3 cases, the median number of citations in the largest studies was only 79 (IQR, 34-159). The largest studies were published a median of 5 years (IQR, 2-8) after the highly cited study, but in 3 cases they were published before the highly cited studies.

For 15 of the 35 studies, the biomarker effect was assessed in unadjusted analyses, and in the other 20 it was adjusted for other variables; in 7 of these studies, the adjusting variables also included other biomarkers. Similarly, for the meta-analyses, adjustments for other variables occurred in 21 cases and 7 included also other biomarkers (eTable).

Table 3 shows comparatively the effect sizes in each highly cited study, meta-analysis, and largest study. Of note, in 23 of the 35 associations, there were some differences between the RR reported in the original highly cited study and how it was represented in the meta-analysis in terms of exposure contrast (n = 18), adjusting covariates (n = 9), and/or sample size (n = 7). However, the difference was equally likely to yield a smaller or larger estimate in the meta-analysis representation of the study results than in the original abstract of the highly cited study.

For 30 of the 35 associations (86%), the highly cited studies had a stronger effect estimate than the largest study, for 3 the largest and highly cited study coincided, and only twice was the effect estimate stronger in the largest than in the highly cited study (Figure). The RR estimate was in the opposite direction in the highly cited than in the largest study in 5 associations, and the increase was more than 2-fold greater in another 20 associations (more than 4-fold in 13 associations). Both early- and late-published highly cited studies showed more extreme results than those found in meta-analyses of these biomarkers (eFigure 1). Differences were beyond chance (RRR 95% CIs excluding 1.00) in 9 associations: 5 for which the highly cited study was published early (in the first 2 years of accumulation of published evidence) and 4 for which the highly cited study was published late. Of the 9 discrepancies, 3 involved highly cited studies for which the effect sizes had been adjusted for other variables.

For 29 of the 35 highly cited studies (83%), the corresponding meta-analysis found a smaller effect (Figure). The RR estimate was in the opposite direction in the highly cited than in the meta-analysis in 4 associations and the increase was more than 2-fold greater in another 14 associations (more than 4-fold in 7 associations). Both early- and late-published highly cited studies showed larger effects than those in meta-analyses (eFigure 2). Differences were beyond chance in 11 associations: 8 in which the highly cited study was published early (in the first 2 years of accumulation of published evidence) and 3 in which the highly cited study was published late. Of the 11 discrepancies, 6 involved highly cited studies where the effect estimates had been adjusted for other variables.

Only 15 associations were nominally statistically significant based on the largest studies and of those 7 had a point estimate RR greater than 1.37. Thirty-two of the 35 associations showed nominally statistically significant increased risk based on the meta-analyses and of those 18 had a point estimate RR greater than 1.37.

This empirical evaluation of 35 top-cited biomarker studies suggests that many of these highlighted associations are exaggerated. In some cases, these markers may have no predictive ability, if one trusts the subsequent replication record, in particular the results of the largest studies on the same associations. Less than half of these biomarkers have shown nominally significant results in the largest studies that have been conducted on them, and only 1 in 5 has shown an RR greater than 1.37. There are several true associations, but they correspond predominantly to small or modest effects with uncommon exceptions. Such effects, even if genuine, may have only incremental translational value for clinical use.

The results of highly cited studies were often in stark contrast against both the largest study on the same association and the corresponding meta-analysis. Occasionally, the contrast was more prominent against the largest study. Meta-analyses of risk factors may have more inflated effects themselves, because typically they include also the highly cited studies and they may suffer from publication and other selective reporting biases.^106- 111 It is probably less common to see smaller effect sizes in large studies due to poorer quality of biomarker and outcome measurements in grand-scale investigations or different population characteristics.

Several reasons could explain false-positive and inflated results among the examined highly cited investigations.^9,112 Many of these studies were relatively small and among the first to report on the association of interest. Discoveries made in small studies are prone to overestimate or underestimate the actual association.⁹ Interest in publishing major discoveries leads to selective reporting from chasing significance. Half of the highly cited studies were published early and this may also have given them a citation advantage (more time to accrue citations). However, half of the highly cited studies were not published early in the accumulation of evidence, and often these attractive articles were among the late-appearing studies on the question of interest. There are some cases where the highly cited studies were even published after the largest study in the field. More extreme estimates of associations can be seen in both early studies and late-appearing studies.

The potential exaggeration of effects was seen in almost all cases that we analyzed, but some exceptions may also occur. Even in those uncommon exceptions where the effect estimate was smaller in the highly cited study than in the meta-analysis, a similar mechanism of preference for attention-drawing results may have applied, but simply a “negative” result was felt to carry more notoriety. For example, in 1 occasion the highly cited study showed that penicillin resistance unexpectedly did not increase the risk of death from pneumococcal pneumonia,³² while the meta-analysis showed an increase in mortality.⁶³ However, in the large majority, notoriety is associated with large rather than paradoxical null effects.

We should acknowledge that several of the highly cited biomarkers probably have genuine associations; a minority have even large effects, eg, the associations of gene loci such as CFH with age-related macular degeneration,¹⁸ prothrombin G2010A and venous thrombosis,⁴⁹ and CYP2C9 and bleeding risk³⁶; the associations of H pylori with gastric cancer and atrophic gastritis^21- 23,38; or the association of NS5A mutations with interferon response.³¹ Nevertheless, even in these cases, highly cited studies described often optimistic effect estimates. Some of these biomarkers have started being used in clinical practice.¹¹³ Their cost-utility depends on their cost and on their discriminating ability.^114- 115 If discriminating ability is overestimated, markers may become overpriced. Biomarkers with large populations of potential users could cause major escalation of health care costs with limited benefits. Obviously, adopting markers with no discriminating ability would be even worse. Clinical biomarker use requires robust evidence and safeguards.^3,116- 117

Novel biomarkers that purport to assist in complex problems related to decisions about diagnosis and prognosis with relatively simple objective measures are often viewed oversimplistically as major advances. However, biomarkers may start having a potential clinical utility when it can be demonstrated that they can provide incremental information beyond already known predictors and any other known biomarkers.¹¹⁸ Regardless of such adjustments, the results of highly cited biomarker studies that appear in major journals are often substantially overestimated. This should lead to reinforcing healthy skepticism about interpreting this literature. This does not mean that no biomarkers of any use are possible to discover, but that the standards for claiming success should be higher. These standards should include not only prospective design, careful analysis plans, and meticulous reporting, but also extensive replication and validation of proposed biomarkers in large independent studies and assessment of their incremental ability. Until such studies are available, emphasis on single studies with highly promising results may be premature.

Some limitations should be discussed. First, we selected only highly cited studies that presented RRs in their abstracts. This removed subjectivity in selecting the most important RR estimate among the many presented in the full text of these articles. Moreover, many highly cited biomarker studies do not report RR estimates in their abstract and were thus excluded from our sample. It is unlikely, however, that the replication record of such biomarkers would be better. Second, we were able to retrieve meta-analyses corresponding to only a third of the highly cited primary studies. It is unlikely that biomarkers lacking a corresponding meta-analysis are enriched in successful replications. If anything, meta-analyses are more likely to conduct for biomarkers that had more and successful replications. Third, the results may not be directly extrapolated to studies of biomarkers that do not get much cited. Citation bias, the preference to cite extreme results, is described in diverse fields.^119- 122 However, less-cited biomarkers are probably less influential in scientific circles and less likely to draw enough attention to become applied in practice. Finally, there is no consensus on what threshold characterizes a highly cited study. One can set also citation criteria that adjust for the time since publication to compare citations of articles published in very different years. However, we preferred to select articles that have clearly achieved cumulative recognition in the literature rather than those that may (or may not) achieve this in the future.

Merely because an article is highly cited does not indicate that the reason for citing it is that it is considered the best science. It is difficult to judge the quality of biomarker studies, mostly because reporting of methods in this literature has been largely elliptical to date.^11,13,123 This further reinforces the notion that citing investigators pay more attention to reporting more extreme estimates rather than the robustness of the methods, which is often either intangible or difficult to compare between different studies. Readers should be cautious when authors cite single studies and not meta-analyses, and authors should be more careful in what they cite.

While we acknowledge these caveats, our study documents that results in highly cited biomarker studies often significantly overestimate the findings seen from meta-analyses. Evidence from multiple studies, in particular large investigations, is necessary to appreciate the discriminating ability of these emerging risk factors. Rapid clinical adoption in the absence of such evidence may lead to wasted resources.