0
We're unable to sign you in at this time. Please try again in a few minutes.
Retry
We were able to sign you in, but your subscription(s) could not be found. Please try again in a few minutes.
Retry
There may be a problem with your account. Please contact the AMA Service Center to resolve this issue.
Contact the AMA Service Center:
Telephone: 1 (800) 262-2350 or 1 (312) 670-7827  *   Email: subscriptions@jamanetwork.com
Error Message ......
Original Contribution |

Persistence of Contradicted Claims in the Literature FREE

Athina Tatsioni, MD; Nikolaos G. Bonitsis, MD; John P. A. Ioannidis, MD
[+] Author Affiliations

Author Affiliations: Department of Hygiene and Epidemiology, (Drs Tatsioni, Bonitsis, and Ioannidis) and the Department of Dermatology (Dr Bonitsis), University of Ioannina School of Medicine; and the Biomedical Research Institute, Foundation for Research and Technology-Hellas (Dr Ioannidis), Ioannina, Greece; Institute for Clinical Research and Health Policy Studies, Department of Medicine, Tufts University School of Medicine, Boston, Massachusetts (Drs Tatsioni and Ioannidis).

More Author Information
JAMA. 2007;298(21):2517-2526. doi:10.1001/jama.298.21.2517.
Text Size: A A A
Published online

Context Some research findings based on observational epidemiology are contradicted by randomized trials, but may nevertheless still be supported in some scientific circles.

Objectives To evaluate the change over time in the content of citations for 2 highly cited epidemiological studies that proposed major cardiovascular benefits associated with vitamin E in 1993; and to understand how these benefits continued being defended in the literature, despite strong contradicting evidence from large randomized clinical trials (RCTs). To examine the generalizability of these findings, we also examined the extent of persistence of supporting citations for the highly cited and contradicted protective effects of beta-carotene on cancer and of estrogen on Alzheimer disease.

Data Sources For vitamin E, we sampled articles published in 1997, 2001, and 2005 (before, early, and late after publication of refuting evidence) that referenced the highly cited epidemiological studies and separately sampled articles published in 2005 and referencing the major contradicting RCT (HOPE trial). We also sampled articles published in 2006 that referenced highly cited articles proposing benefits associated with beta-carotene for cancer (published in 1981 and contradicted long ago by RCTs in 1994-1996) and estrogen for Alzheimer disease (published in 1996 and contradicted recently by RCTs in 2004).

Data Extraction The stance of the citing articles was rated as favorable, equivocal, and unfavorable to the intervention. We also recorded the range of counterarguments raised to defend effectiveness against contradicting evidence.

Results For the 2 vitamin E epidemiological studies, even in 2005, 50% of citing articles remained favorable. A favorable stance was independently less likely in more recent articles, specifically in articles that also cited the HOPE trial (odds ratio for 2001, 0.05 [95% confidence interval, 0.01-0.19; P < .001] and the odds ratio for 2005, 0.06 [95% confidence interval, 0.02-0.24; P < .001], as compared with 1997), and in general/internal medicine vs specialty journals. Among articles citing the HOPE trial in 2005, 41.4% were unfavorable. In 2006, 62.5% of articles referencing the highly cited article that had proposed beta-carotene and 61.7% of those referencing the highly cited article on estrogen effectiveness were still favorable; 100% and 96%, respectively, of the citations appeared in specialty journals; and citations were significantly less favorable (P = .001 and P = .009, respectively) when the major contradicting trials were also mentioned. Counterarguments defending vitamin E or estrogen included diverse selection and information biases and genuine differences across studies in participants, interventions, cointerventions, and outcomes. Favorable citations to beta-carotene, long after evidence contradicted its effectiveness, did not consider the contradicting evidence.

Conclusion Claims from highly cited observational studies persist and continue to be supported in the medical literature despite strong contradictory evidence from randomized trials.

Figures in this Article

Some research findings that have received wide attention in the scientific community, as proven by the high citation counts of the respective articles, are eventually contradicted by subsequent evidence.1 A number of such high-profile contradictions pertain to differences between nonrandomized and randomized studies. For example, the effect of vitamin E on cardiovascular disease prevention has been in the center of a major debate in clinical research over the last 2 decades. Vitamin E is known to have antioxidant activity, and a long list of citations in the preclinical literature on antioxidants24 suggested that these agents may be beneficial for cancer and cardiovascular disease. Two highly cited publications suggested in the 1990s that vitamin E could decrease cardiovascular disease risk by almost half in men and in women.5,6 However, subsequent randomized trials showed no benefit or even suggested increased harm.7,8 Several other highly prominent contradictions have also been recorded pertaining to the effects of other dietary components and hormones.915 The prominent refutation of the epidemiological studies has spurred considerable controversy for observational epidemiology in general.1621

Such debate offers opportunities to study what happens to the scientific literature, when a highly prominent claim is refuted. How quickly are such beliefs abandoned? Is there still literature citing the contradicted studies despite their refutation? What counterarguments are used by the citing articles to defend the original claims? To answer these questions, we performed citation content analysis for the 2 most highly cited articles that proposed vitamin E benefits. We evaluated the change in favorable vs unfavorable citations over time and recorded the counterarguments that were used to continue supporting the belief in vitamin E effectiveness. To assess the generalizability of our findings, we also examined the extent to which 2 other major contradicted claims, the preventive effectiveness of beta-carotene for cancer and estrogens for Alzheimer dementia, continue to be supported in the current literature.

Evolution of Evidence for Vitamin E

We focused on 2 highly cited articles published in 1993. These articles presented data from 2 observational cohorts5,6 and showed consistently that vitamin E was associated with major decreases in the relative risk (RR) of cardiovascular events (0.63, 95% confidence interval [CI], 0.47-0.84 in men and RR, 0.59; 95% CI, 0.38-0.91 in women for those receiving vitamin E for 2 years vs none). These 2 articles are the most-cited papers on benefits from vitamin E supplementation and they have received 1395 and 1234 citations, respectively, until the end of 2006. Based on these articles, vitamin E was considered cardioprotective for many years. Several smaller studies suggested direct or indirect evidence supporting this claim.

A randomized trial of 2002 patients (CHAOS) published in 1996 also found a 47% relative risk reduction for cardiovascular events.7 However, many randomized trials subsequently found no cardiovascular benefit. The most-cited contradicting trial (HOPE) was published in January 2000 and found an RR of 1.05 (95% CI, 0.95-1.16) for cardiovascular events,8 an effect entirely incompatible with estimates of the epidemiological studies. A meta-analysis published in late 2004 concluded that at high doses, vitamin E significantly increased the risk of death (RR, 1.04; 95% CI, 1.01-1.07).22 Publication of the CHAOS and HOPE trials have also accumulated a large number of citations (1172 and 704 citations by the end of 2006, respectively) and the meta-analysis is also highly cited, despite the short time since its publication (226 citations by the end of 2006, the most-cited article published in in the field of clinical medicine in 2004 according to Essential Science Indicators). A recent meta-analysis23 even concluded that among high-quality trials, vitamin E increases mortality regardless of dose (RR, 1.04; 95% CI, 1.01-1.07 in low-bias trials). Vitamin E supplementation is not currently recommended by practice guidelines.24,25

Citation Curves. We downloaded annual citation counts from Thomson Scientific ISI Web of Knowledge for each of the 2 highly cited epidemiological studies between 1993 and 2006 and also assessed the number articles citing at least 1 of the 2 studies. For reference standard, we examined the total annual citation curves for all the articles published in the same year (1993) and in the same journal as the 2 highly cited epidemiological studies.

Selection of the Citing Articles. We sampled citations to the 2 highly cited epidemiological studies at 3 different and equidistant years: 1997, 2001, and 2005. The first selected year (1997) represents the peak of annual citations and may be perceived to be the time when the evidence was the strongest in favor of vitamin E (shortly after the additional support offered by the CHAOS trial published in 1996).7 The second selected year (2001) corresponds with an early period after major refutation (1-2 years after the HOPE results).8 The third selected year (2005) corresponds with a late period after major refutation; meta-analysis had even shown increased harm with vitamin E. To allow for a fairly similar number of citations analyzed at each selected year, we sampled every third citation in 1997, every second citation in 2001, and all citations in 2005 among citations made to either or both highly cited epidemiological articles.

Our purpose was not to study the overall literature on a research topic in which contradiction of the original studies has arisen. The boundaries of such a literature review are very difficult, if not impossible, to define. On the contrary, we aimed to examine the citing behavior of the scientific literature toward the original studies that have been contradicted. The content analysis of this set of citations is likely to yield a set of references that is enriched in positions that allude to or even try to defend the original claims. This can give insights on how extensively, and with what arguments, these claims are defended despite the ensued contradiction.

Characteristics of the Citing Articles. For each eligible citation, we retrieved the full text of the citing article. We recorded the first author, journal, and country(ies) of investigators. We classified each article depending on whether it had primary data or not (reviews, meta-analyses, editorials, letters, other), and articles with primary data were further categorized depending on whether they were derived from a randomized trial or from nonrandomized studies. Additionally, we retrieved the 2005 impact factor for each journal that published an eligible article26 and recorded whether (per Web of Knowledge classification) the journal was classified in the general/internal medicine category vs some specialty (including both clinical and basic sciences). We also recorded which of the 2001 and 2005 articles had also cited the HOPE trial,8 the most highly cited contradicting publication to date on this topic.

Citation Content Analysis. We assessed how many times each of the 2 highly cited epidemiological studies was cited with a reference in each citing article. For each time that each article was cited, we recorded the exact phrase or sentence in which the reference(s) appeared and any preceding or following sentences that elaborated on the same argument(s). When these articles were cited multiple times in the same citing article, we captured the text on all of these appearances.

We first excluded citations that were erroneous (irrelevant, apparently an error of the authors), and those that were not pertinent to cardiovascular disease prevention and vitamin E, but instead to some other aspect of the 2 highly cited articles (eg, association of vitamin C with chronic diseases that was also commented in the original highly cited articles) or some other generic issue (eg, referring to similar methods or questionnaires being used as in the vitamin E studies). When the context of the citation was pertinent to the association between vitamin E and cardiovascular disease prevention, we categorized the overall stance of the citing article as favorable, equivocal, or unfavorable.

The categorization depended on whether the arguments were suggesting that vitamin E had beneficial effects (favorable), both favorable and unfavorable arguments existed without any clear preference given to either (equivocal), or vitamin E was claimed to be ineffective or harmful (unfavorable). When both favorable and unfavorable arguments were presented but the authors eventually took sides in one direction, the article was accordingly categorized as either favorable or unfavorable. For categorization, we cumulatively considered all the expounded arguments in each citing article.

Data extraction was performed by 2 independent investigators; discrepancies were resolved by consensus and arbitration by a third investigator.

Quantitative Analyses

The primary outcome was the proportion of articles citing the highly cited epidemiological studies that were favorable, equivocal, and unfavorable about vitamin E effectiveness for cardiovascular disease prevention.

The main hypothesis was that these proportions should markedly change between 1997, 2001, and 2005, unless beliefs in vitamin E effectiveness remained unchanged. Secondary hypotheses evaluated whether any additional characteristics of the citing article besides year of publication (country of origin, article type, impact factor, journal field, article also cited the contradicting HOPE trial) were related to its stance.

The primary hypothesis was evaluated with the Jonckheere-Terpstra test for multiple-ordered categories. The secondary hypotheses were evaluated with the Kruskal-Wallis analysis of variance for single-ordered variables and the Jonckheere-Terpstra test for multiple-ordered variables.

We also performed analyses to examine the independent association of different characteristics on the overall stance of a citing article. Unfavorable and equivocal citations were merged because they occurred fewer times than favorable ones. We used univariate logistic regressions to examine the association between each of the characteristics mentioned previously with a favorable stance. Variables with a P value of less than .10 in univariate analyses were considered also in a multivariate analysis. Categorical independent variables were treated with multiple dummy variables. The regression used step-wise backward elimination of variables that had a P value of greater than .05. Forward selection of variables yielded similar models. For the multivariate analyses, we first constructed a variable that considered both the publication year and whether citation to the HOPE trial was made (categories: 1997, 2001 and citing HOPE, 2001 not citing HOPE, 2005 and citing HOPE, and 2005 not citing HOPE), since year and citation to the HOPE trial are by default strongly correlated (citing HOPE not applicable for 1997 articles; HOPE was published in 2000). Quantitative analyses were performed using SPSS version 13.0 (SPSS Inc, Chicago, Illinois) and StatXact (Cytel Corp, Boston, Massachusetts). P values were 2-tailed, and a P value of less than .05 was considered statistically significant.

Content Analysis for Citations to the HOPE Trial. Articles selected because they cite the 2 highly cited epidemiological studies may be more likely to be favorable to vitamin E use compared with articles that would be selected because they cite contradicting studies. Therefore, we also created a separate group of articles in which we sampled every third article that cited in 2005 the HOPE trial,8 regardless of whether it cited the 2 highly cited epidemiological studies or not. Through the same process, we identified the proportion of favorable, equivocal, and unfavorable citations to vitamin E use.

Qualitative Counterarguments. We created a qualitative list of the different types of arguments that have been made to counter the accumulating evidence that vitamin E is harmful or not effective. We categorized counterarguments according to allusion to biases and genuine differences in study participants, interventions, cointerventions, and outcomes using the PICO structure.27

Beta-Carotene for Cancer Prevention and Estrogen for Prevention of Alzheimer Dementia

To examine the generalizability of our main findings on vitamin E, we also investigated 2 other examples for which observational claims have been subsequently contradicted by large randomized trials. We used as highly cited articles the most-cited articles that had proposed these claims. We selected a claim that had been made a long time ago and had also been contradicted long ago (beta-carotene for cancer prevention), and a claim that had been contradicted very recently (estrogen for dementia prevention). We then examined the current stance (favorable, equivocal, or unfavorable) of citing articles. We chose for citation content analysis the calendar year 2006, ie, a decade and 2 years, respectively, after the major contradicting studies were published.

Beta-carotene was initially supported by many epidemiological studies and laboratory investigations as a potent chemoprevention against cancer. The most-cited article in this literature is an influential review of the epidemiological and other nonrandomized studies that was published in 1981.9 This review received 1119 citations by the end of 2006. Randomized trials, nevertheless, found no benefit or harm with beta-carotene use. The 3 most-cited trials on this topic are the Alpha-Tocopherol, Beta Carotene Cancer Prevention Study Group10 (primary outcome, lung cancer; published in 1994; 1640 citations through 2006), the Beta-Carotene and Retinol Efficacy Trial11 (primary outcome, lung cancer using carotene combined with retinol; published in 1996; 1296 citations through 2006), and the Physicians's Health Study12 (primary outcome, all cancers; published in 1996; 1087 citations through 2006). These trials found relative risks of 1.18 (95% CI, 1.03-1.36), 1.28 (95% CI, 1.04-1.57) and 0.98 (95% CI, 0.91-1.06), respectively. Beta-carotene supplementation is not recommended by any guidelines currently.

Estrogens were also supported by many epidemiological studies and laboratory investigations as strong neuroprotective agents that could diminish the risk of dementia. The most-cited article is an observational study13 published in 1996 that found a 60% (95% CI, 15-78) RR reduction in postmenopausal women taking estrogens. This study has received 915 citations through 2006. Early randomized trials could not replicate these benefits and in mid-2004, the Women's Health Initiative Memory Study RCT published its results showing a trend for increased risk of dementia with estrogens in postmenopausal women (RR, 1.49; 95% CI, 0.83-2.66)14 and worsening of cognition.15 Estrogens are also not recommended as preventive intervention for dementia currently.

For each of these 2 topics, we constructed citation curves for the highly cited epidemiological articles, retrieved the articles citing the articles in 2006, evaluated the stance (favorable, equivocal, or unfavorable) of the citing articles, and captured counterarguments raised to defend the effectiveness of these interventions using the same methods as for vitamin E. For 3 articles, the 2 independent reviewers disagreed on the stance of the citation and the third investigator arbitrated on the discrepancy.

Vitamin E for Cardiovascular Disease Prevention

Citation Curves Over Time. The citation curve for the 2 vitamin E epidemiological articles largely paralleled the citation curve for all the articles published in the same journal in 1993: early rapid increase, peak in 1997 or 1998, and slow decline until 2001 (Figure 1). However, in 2002 and beyond, the relative decrease in citations was much steeper for the 2 vitamin E articles than for the total citations to all articles published in the same journal. The citation rate in 2006 for all articles published in 1993 continued to be more than half (55%) of the citation rate in 1997, while the 2 vitamin E articles had decreased to 20% of their peak annual citation rate by that time.

Figure 1. Standardized Citation Counts for the 2 Highly Cited Epidemiological Studies on Vitamin E
Graphic Jump Location

Annual citation counts are standardized against the calendar year 1997 (citations received in 1997 are set at the standardized value of 100). The articles published in the same journal in 1993 received a total of 110 383 citations during 1993-2006.

Characteristics of Eligible Citing Articles. We selected for citation analysis 176 citing articles, of which 56 articles were published in 1997, 61 in 2001, and 59 in 2005 (Table 1). We could not retrieve 2 articles from 2001 and 2 from 2005; thus, we finally analyzed 172 publications (Table 1).

Table Graphic Jump LocationTable 1. Characteristics of Eligible Citing Articles for the 2 Highly Cited Vitamin E Studiesa

Seventy-six articles (44.2%) included at least 1 author from an institution located in the United States. Ninety-seven (56.4%) articles included primary data, 23 (23.7%) of which pertained to data from randomized trials. The citing articles were published in journals with a median impact factor 2.310 (interquartile range, 1.52-4.04). Twenty-six (16.0%) appeared in general or internal medicine journals. Seventy-two (62.1%) of the articles in 2001 and 2005 cited also the contradicting HOPE trial (Table 1).

Of the 172 articles, one had entirely erroneously cited 1 of the 2 articles5 and the citations in another 6 articles were not pertinent to vitamin E and cardiovascular disease prevention. Thus, 165 articles were eligible for categorizing a stance on vitamin E in cardiovascular prevention.

Overall Stance and Evolution Over Time. Overall, 101 citing articles (61.2%) were favorable, 36 (21.8%) were equivocal, and 28 (17.0%) were unfavorable (Table 1). Categorization by 2 independent investigators was concordant (weighted κ 0.91, 95% CI, 0.87-0.94).

Citing articles showed significant difference in their stance over time (P = .0002). The proportion of unfavorable articles increased from 1.9% in 1997 to 14.3% in 2001 and to 33.9% in 2005. Despite a decrease in the proportion of favorable articles, these still represented 50% of the total in 2005. The stance of the articles overall was also significantly more favorable, less unfavorable, or both, when articles were not originating from the United States (P = .046), when articles included nonrandomized primary data (P < .001), when specialty journals were involved (P < .001), and when the HOPE trial was not cited (P < .001) (Table 2).

Table Graphic Jump LocationTable 2. Association of Characteristics of the Citing Articles With Their Overall Stance Toward Vitamin E Effectiveness in Cardiovascular Disease Prevention

Independent Associations of Citing Article Characteristics With Favorable Stance. In multivariate analyses, the odds of a citing article having a favorable stance were approximately 20 times lower in 2001 and 2005, as compared with 1997, when the HOPE trial was also cited (odds ratios were 0.05 and 0.06, respectively), but not necessarily when the HOPE trial was not cited (Table 3). Moreover, the odds of a favorable stance were 12 times lower in articles published in general and internal medicine journals than in articles published in other journals (Table 3).

Table Graphic Jump LocationTable 3. Association of Characteristics of the Citing Articles With a Favorable Stance Toward Vitamin E Effectiveness in Cardiovascular Disease Prevention

Overall Stance of Articles Citing the HOPE Trial in 2005. In a sample of 29 articles published in 2005 that had cited the HOPE study, 6 (20.7%) were still favorable to vitamin E, 11 (37.9%) were equivocal, and 12 (41.4%) were unfavorable. Eight of these articles had also cited one or both of the 1993 highly cited epidemiological studies. Excluding these 8 articles, there were 6 (28.6%) favorable citations to vitamin E, 9 (42.9%) equivocal citations, and 6 (28.6%) unfavorable citations.

Qualitative List of Counterarguments. Typical examples of counterarguments are shown in Box 1.2834 Alluded biases included study selection bias in meta-analyses or information bias due to incomplete recording of outcome events. Genuine diversity between studies in favor of vitamin E and trials with negative or harmful effects focused on baseline patient characteristics (ie, genetic background, dietary habits, stage of atherosclerotic disease, oxidative stress status, and lifestyle of study participants); vitamin E intervention—type, dosage, and bioavailability (ie, use of synthetic vs natural form of tocopherol, use of small vs higher doses of tocopherol, ingestion of vitamin E with vs without lipid-rich meals, use of balanced intake vs single antioxidant supplementation, or discrepancies in antioxidant levels in blood or tissues before and after supplementation); concomitant interventions (ie, patients supplemented with a harmful cointervention or lacking an additional useful antioxidant cointervention); and duration of follow-up (short-term vs long-term follow up studies). Diverse biological mechanisms were invoked in support.

Box 1. Qualitative List of Counterarguments Made to Defend Vitamin E Effectiveness Despite Contradictory Evidence From Randomized Trials

Biases

Selection bias: meta-analysis did not put its results in perspective by reviewing the context of research on vitamin E including the many positive observational and interventional studies28

Information bias: mortality estimates from CHAOSa came from a research letter, not a peer-reviewed study, and included data after the study was officially ended, and thus subject to information bias28

Genuine Diversity

Participant Characteristics

Genetic characteristics: genetic background of study subjects might have contributed to the differential results29

Dietary habits: discrepancies may be explained by differences in the antioxidant content of the basal diet of the sample population under investigation30

Stage of disease: some antioxidants, eg, vitamin E, might be more effective in the early phase of atherosclerosis, but much less so in the advanced clinically overt stage present in the majority of patients evaluated in clinical trials31

Oxidative stress status: studies that have included healthy subjects with decreased oxidative stress while vitamin E reduced oxidative stress in smokers (a condition of increased oxidative stress)31

Lifestyle characteristics: lifestyle of study subjects might have contributed to the differential results29

Intervention: Vitamin E Form, Dose, Bioavailability

Vitamin E form: some trials utilized synthetic tocopherol, whose efficacy is not equivalent to the natural form31

Vitamin E dose: an adequate intake in the lowest intake category or a low interindividual variation intake may explain some of the negative findings32

Vitamin E bioavailability: no control on how antioxidant vitamins were ingested: the bioavailability of vitamin E is higher when it is taken with lipid-rich meals. Antioxidant levels were not consistently measured in blood or tissues before and after supplementation: the same intake may produce different levels in distinct individuals31

Co-interventions

Beta-carotene (harmful co-intervention): most of the evidence for an elevated mortality risk came from two trials that administered vitamin E together with beta-carotene33

Lack of appropriate cointervention: . . . single antioxidant supplementation might not be a good strategy, since antioxidant defenses normally behave as a network: therefore balanced intake is likely important31

Outcomes

Duration of follow-up: the possibility that antioxidants need to be taken more than five years to have a significant effect on atherosclerotic plaque formation cannot be dismissed34

aRefers to the mortality data of CHAOS, which, contrary to the main publication of the trial on cardiovascular events, had shown no benefit from vitamin E.

Beta-Carotene for Cancer Prevention and Estrogen for Dementia Prevention

Citations to the highly cited article proposing the possibility of beta-carotene effectiveness for cancer prevention did not fall more steeply than those of the average article published in the same journal in the same year. Conversely, the decline in citation rate took a decade longer to start than for the average paper. The decline was heralded by the publication of the most prominent contradicting trials (Figure 2). Citations to the highly cited epidemiological study on estrogen use for dementia prevention largely followed the pattern of the citations in the same journal for articles published in the same year. The contradiction is recent and there was only a modestly steeper decline in 2006 (Figure 2).

Figure 2. Standardized Citation Counts for the Most Highly Cited Article on Effectiveness of Beta-Carotene in Cancer Prevention and Effectiveness of Estrogen in Alzheimer Dementia Prevention
Graphic Jump Location

Annual citation counts are standardized against the calendar year with the highest citation count for the highly cited article. For beta-carotene, the highest citation count was in 1985. The articles published in the same journal in 1981 as the highly cited article on beta-carotene received a total of 141 586 citations during 1981-2006. For estrogen, the highest citation count was in 1999. The articles published in the same journal in 1996 as the highly cited article on estrogen received a total of 65 300 citations during 1996-2006.

The highly cited studies received 17citations for beta-carotene and 48 citations for estrogen in 2006 (Table 4) and 1 citing article could not be retrieved for each. For beta-carotene, 10 citing articles (62.5%) were favorable, 3 (18.8%) equivocal, and 3 (18.8%) unfavorable. For estrogen, 29 citing articles (61.7%) were favorable, 14 (29.8%) equivocal, and 4 (8.5%) unfavorable. All beta-carotene citations and all but 2 estrogen citations appeared in specialty journals. The overall stance of the citing articles was significantly more favorable when the contradicting Alpha-Tocopherol, Beta Carotene Cancer Prevention Study Group and Women's Health Initiative Memory Study were not cited (P = .001 and P = .009, respectively).

Table Graphic Jump LocationTable 4. Association of Characteristics of the Citing Articles With Their Overall Stance Toward Beta-Carotene for Cancer Prevention and Estrogen for Dementia Prevention

Of the 10 favorable and the 3 equivocal beta-carotene citations, only 1 raised counterarguments against the contradicting evidence, claiming35 that “the effectiveness of these carotenoids as antioxidants depends upon a number of factors (eg, concentration, cell type, cell status, timing of insult exposure, location in the cell, interaction with other antioxidants, etc)”. The other 9 favorable citing articles (2 reviews and 7 experimental articles on human tissue and animals) simply did not cite any trials that had contradicted beta-carotene effectiveness.

For Alzheimer disease, counterarguments (Box 2)3643 to support estrogen pertained to various selection biases; issues related to the participants, including differences in age, menopausal symptoms, prior hormonal treatment, and stage of disease (before vs after the onset of dementia process); differences in the intervention scheme, including estrogen form, route of administration and regimen, and cointerventions; and choice of outcome definitions (Box 2).

Box 2. Qualitative List of Counterarguments Made to Defend Estrogen Effectiveness Despite Contradictory Evidence From Randomized Trials

Biases

Selection bias—general: the dichotomy between the observational and recent prospective studies may be due to selection bias36

Different baseline risk despite randomization: there were baseline differences on the global cognitive test—low scorers (more frequent in the treatment than placebo arm) were at much greater risk for developing dementia37

Different risk factors at baseline despite randomization: significantly more women with a history of hypertension were randomized to active treatment than to placebo (41% vs 38%), but more women with a history of stroke were allocated to the placebo group (2.0% vs 1.3%)37

Genuine Diversity

Participant Characteristics

Age: women were an average age of 68 when entering the study. These issues may limit the generalizability38

Background disease: the majority of the dementia diagnoses in the WHI study seemed to be related to vascular disease38

Prior treatment: many variables may contribute to the discrepancy; these include prior hormone replacement history39

Concomitant symptoms: women with vasomotor symptoms were excluded from the Women's Health Initiative clinical trial if it was anticipated that symptoms would affect treatment compliance37

Stage of disease: estrogen replacement therapy may be applied to delay the progression of AD pathogenesis but not to recover the lost functions40

Intervention: Estrogen Form, Route of Administration, Dosage

Estrogen source and mode of delivery: other factors have also been identified for consideration in interpretation of the WHI study including the source of hormone (equine estrogens as compared to synthetic human forms of these hormones) and mode of delivery (cyclic vs continuous)41

Estrogen preparation, route and mode of delivery: although informative, the interpretation of the WHI studies is limited by the hormone preparations used, their route of administration, the regimen of hormone administration (ie, continuous daily therapy vs cyclic affect concentrations and localization of antiapoptotic proteins, which appear to exert their antiapoptotic effects through maintenance of mitochondrial membrane potential in the face of cellular stresses42

Estrogen dose: in contrast, in vitro exposure of neurons to estrogen if the dose is high enough, can exacerbate degeneration43

Co-interventions

Progestin (harmful co-intervention): progestin included in the estrogen replacement therapy could compromise estrogen's effect40a

Outcomes

Type of endpoints: WHI did not consider Alzheimer disease as a specific endpoint, whereas most observational studies looked specifically at Alzheimer disease risk37

aWHI generated randomized data both for estrogen and for estrogen plus progestin regimens

Citations to the 2 highly cited observational studies proposing an association of vitamin E with reduced cardiovascular events became less favorable over time, as contradicting data from randomized trials accumulated. Nevertheless, despite the eventual accumulation of strongly refuting evidence, even in 2005, half of the articles citing these epidemiological studies were still favorable to the vitamin E claim. Even among articles that cited the contradicting HOPE trial rather than the positive epidemiological studies, the majority in 2005 still could not conclude that vitamin E was ineffective. Many counterarguments were raised to defend vitamin E in the face of contradictory evidence from RCTs. In a similar fashion, in 2006 more than half of the articles citing the highly cited epidemiologic articles on beta-carotene for cancer prevention and estrogen for dementia prevention remained favorable for these interventions. For beta-carotene, after a decade had passed from the contradiction of its effectiveness, counterarguments were uncommon: citing articles simply did not mention the contradicting trials. Conversely, for estrogen, a claim for which contradiction has been more recent, many counterarguments (of similar breadth as for vitamin E) were raised to defend its effectiveness.

We observed an apparent split of stance in the scientific literature. The persistent favorable stance toward the contradicted interventions was particularly prominent in articles published in specialty journals of both clinical and basic science disciplines. Specialist articles apparently continued to use references to the highly cited observational studies to support their own lines of research. The presence of refuting data were not mentioned in many articles. Other articles did report data with contrary results, but they raised also a wide array of counterarguments to support the observational claim. Most nonrandomized studies published in specialty journals show positive results.18,44,45 Apparently, there is also a citation bias selecting positive citations.46,47 Conversely, for journals with a more general medical audience, apparently the contradicting randomized data carried more weight than the observational data. For beta-carotene and estrogen, almost all analyzed citations appeared in specialty journals.

Our citation content analysis also highlights another aspect of the existing antithesis between randomized and observational research.17,4851 Apparently, the same data are used and interpreted entirely differently by different investigators depending on whether they supported findings from the randomized trials or observational studies. However, when randomized and observational studies disagree, it is incorrect to assume that nonrandomized studies are always wrong. Disagreements and contradictions appear also between randomized trials, even large ones, and also in many other research fields in which other designs are used.

In the evaluation of counterarguments, we encountered almost any source of bias, genuine diversity, and biological reasoning invoked to defend the original observations. While some or even many of these counterarguments may be valid, this is also consistent with a belief that is defended at all cost. The defense of the observational associations was persistent, despite the availability of very strong contradicting randomized evidence on the same topic. Thus, one wonders whether any contradicted associations may ever be entirely abandoned, if such strong randomized evidence is not considered as much stronger evidence on the topic. For most associations and questions of medical interest, either no randomized data exist, or the randomized evidence is minimal or of poor quality.52,53

Our data also suggest that contradiction through randomized trials may lead eventually to a decrease in the absolute frequency of citations to the epidemiological studies. However, this may occur with considerable delay and a considerable segment of the literature continues to cite the contradicted articles long after the contradiction. The articles that cited these observational studies continued to be predominantly favorable. Moreover, even when we considered articles that referenced the most prominent contradicting trial against vitamin E, clearly unfavorable citations for vitamin E were still the minority. Beta-carotene, in particular, offers the opportunity to examine what happens when many years have passed after the contradiction: a citation rate of decreasing (but still substantial) volume continues to support the contradicted claims without even mentioning the contradicting evidence or raising counterarguments.

Sometimes investigator beliefs in scientific circles may have similar psychological characteristics as the nonscientific beliefs observed in other areas of society. The wish bias of individuals, irrespective of topic, can be large and may also influence the interpretation of scientific results. Such bias has been discussed and demonstrated in the past for several other societal and scientific efforts.5459 Wish bias does not necessarily mean that the defended beliefs are wrong. Moreover, it can be difficult to discern whether perpetuated beliefs are based on careful consideration of all evidence and differential interpretation, inappropriate entrenchment of old information, lack of dissemination of newer data, or purposeful silencing of their existence. Regardless of the reasons, better communication between research specialists and evidence-based clinical science60 may improve this situation and may lead to more rational and concerted translational efforts in basic, preclinical, and clinical research.

Corresponding Author: John P. A. Ioannidis, MD, Department of Hygiene and Epidemiology, University of Ioannina School of Medicine, University Campus, Ioannina, 45110 Greece (jioannid@cc.uoi.gr).

Author Contributions: Dr Ioannidis had full access to all of the data in the study and takes responsibility for the integrity of the data and the accuracy of the data analysis.

Study concept and design: Tatsioni, Bonitsis, Ioannidis.

Acquisition of data: Tatsioni, Bonitsis, Ioannidis.

Analysis and interpretation of data: Tatsioni, Bonitsis, Ioannidis.

Drafting of the manuscript: Tatsioni, Ioannidis.

Critical revision of the manuscript for important intellectual content: Bonitsis.

Statistical analysis: Tatsioni, Ioannidis.

Study supervision: Ioannidis.

Financial Disclosures: None reported.

Funding/Support: None.

Ioannidis JP. Contradicted and initially stronger effects in highly cited clinical research.  JAMA. 2005;294(2):218-228
PubMed   |  Link to Article
Steinberg D, Parthasarathy S, Carew TE, Khoo JC, Witztum JL. Beyond cholesterol: modifications of low-density lipoprotein that increase its atherogenicity.  N Engl J Med. 1989;320(14):915-924
PubMed   |  Link to Article
Steinberg D, Witztum JL. Lipoproteins and atherogenesis: current concepts.  JAMA. 1990;264(23):3047-3052
PubMed   |  Link to Article
Steinberg D. Antioxidants and atherosclerosis: a current assessment.  Circulation. 1991;84(3):1420-1425
PubMed   |  Link to Article
Rimm EB, Stampfer MJ, Ascherio A.  et al.  Vitamin E consumption and the risk of coronary heart disease in men.  N Engl J Med. 1993;328(20):1450-1456
PubMed   |  Link to Article
Stampfer MJ, Hennekens CH, Manson JE.  et al.  Vitamin E consumption and the risk of coronary disease in women.  N Engl J Med. 1993;328(20):1444-1449
PubMed   |  Link to Article
Stephens NG, Parsons A, Schofield PM.  et al.  Randomised controlled trial of vitamin E in patients with coronary disease: Cambridge Heart Antioxidant Study (CHAOS).  Lancet. 1996;347(9004):781-786
PubMed   |  Link to Article
Yusuf S, Dagenais G, Pogue J, Bosch J, Sleight P. Vitamin E supplementation and cardiovascular events in high-risk patients: the Heart Outcomes Prevention Evaluation Study Investigators.  N Engl J Med. 2000;342(3):154-160
PubMed   |  Link to Article
Peto R, Doll R, Buckley JD, Sporn MB. Can dietary beta-carotene materially reduce human cancer rates?  Nature. 1981;290(5803):201-208
PubMed   |  Link to Article
 The effect of vitamin E and beta carotene on the incidence of lung cancer and other cancers in male smokers: the Alpha-Tocopherol, Beta Carotene Cancer Prevention Study Group.  N Engl J Med. 1994;330(15):1029-1035
PubMed   |  Link to Article
Omenn GS, Goodman GE, Thornquist MD.  et al.  Effects of a combination of beta carotene and vitamin A on lung cancer and cardiovascular disease.  N Engl J Med. 1996;334(18):1150-1155
PubMed   |  Link to Article
Hennekens CH, Buring JE, Manson JE.  et al.  Lack of effect of long-term supplementation with beta carotene on the incidence of malignant neoplasms and cardiovascular disease.  N Engl J Med. 1996;334(18):1145-1149
PubMed   |  Link to Article
Tang MX, Jacobs D, Stern Y.  et al.  Effect of oestrogen during menopause on risk and age at onset of Alzheimer's disease.  Lancet. 1996;348(9025):429-432
PubMed   |  Link to Article
Shumaker SA, Legault C, Kuller L.  et al. Women's Health Initiative Memory Study.  Conjugated equine estrogens and incidence of probable dementia and mild cognitive impairment in postmenopausal women: Women's Health Initiative Memory Study.  JAMA. 2004;291(24):2947-2958
PubMed   |  Link to Article
Espeland MA, Rapp SR, Shumaker SA.  et al. Women's Health Initiative Memory Study.  Conjugated equine estrogens and global cognitive function in postmenopausal women: Women's Health Initiative Memory Study.  JAMA. 2004;291(24):2959-2968
PubMed   |  Link to Article
Ioannidis JP, Haidich AB, Lau J. Any casualties in the clash of randomised and observational evidence?  BMJ. 2001;322(7291):879-880
PubMed   |  Link to Article
Patsopoulos NA, Analatos AA, Ioannidis JPA. Relative citation impact of various study designs in the health sciences.  JAMA. 2005;293(19):2362-2366
PubMed   |  Link to Article
Kavvoura FK, Liberopoulos G, Ioannidis JP. Selection in reported epidemiological risks: an empirical assessment.  PLoS Med. 2007;4(3):e79
PubMed   |  Link to Article
Buchanan AV, Weiss KM, Fullerton SM. Dissecting complex disease: the quest for the Philosopher's Stone?  Int J Epidemiol. 2006;35(3):562-571
PubMed   |  Link to Article
Prentice RL, Langer R, Stefanick ML.  et al.  Combined postmenopausal hormone therapy and cardiovascular disease: toward resolving the discrepancy between observational studies and the Women's Health Initiative clinical trial.  Am J Epidemiol. 2005;162(5):404-414
PubMed   |  Link to Article
Petitti DB, Freedman DA. Invited commentary: how far can epidemiologists get with statistical adjustment?  Am J Epidemiol. 2005;162(5):415-418
PubMed   |  Link to Article
Miller ER III, Pastor-Barriuso R, Dalal D.  et al.  Meta-analysis: high-dosage vitamin E supplementation may increase all-cause mortality.  Ann Intern Med. 2005;142(1):37-46
PubMed   |  Link to Article
Bjelakovic G, Nikolova D, Gluud LL, Simonetti RG, Gluud C. Mortality in randomized trials of antioxidant supplements for primary and secondary prevention: systematic review and meta-analysis.  JAMA. 2007;297(8):842-857
PubMed   |  Link to Article
Mosca L, Appel LJ, Benjamin EJ.  et al. American Heart Association.  Evidence-based guidelines for cardiovascular disease prevention in women.  Circulation. 2004;109(5):672-693
PubMed   |  Link to Article
Kris-Etherton PM, Lichtenstein AH, Howard BV, Steinberg D, Witztum JL. Antioxidant vitamin supplements and cardiovascular disease.  Circulation. 2004;110(5):637-641
PubMed   |  Link to Article
Thomson Scientific Institute for Scientific Information.  ISI Web of Knowledge. http://portal.isiknowledge.com/. Accessed October 7, 2007
Counsell C. Formulating questions and locating primary studies for inclusion in systematic reviews.  Ann Intern Med. 1997;127(5):380-387
PubMed   |  Link to Article
Hyman M, Pizzorno J, Weil A. A rational approach to antioxidant therapy and vitamin E.  Altern Ther Health Med. 2005;11(1):14-17
PubMed
Meydani M. Vitamin E and atherosclerosis: beyond prevention of LDL oxidation.  J Nutr. 2001;131(2):366S-368S
PubMed
Alissa EM, Bahjri SM, Al-Ama N, Ahmed WH, Starkey B, Ferns GA. Dietary vitamin A may be a cardiovascular risk factor in a Saudi population.  Asia Pac J Clin Nutr. 2005;14(2):137-144
PubMed
Cherubini A, Vigna GB, Zuliani G, Ruggiero C, Senin U, Fellin R. Role of antioxidants in atherosclerosis: epidemiological and clinical update.  Curr Pharm Des. 2005;11(16):2017-2032
PubMed   |  Link to Article
Eichholzer M, Luthy J, Gutzwiller F, Stahelin HB. The role of folate, antioxidant vitamins and other constituents in fruit and vegetables in the prevention of cardiovascular disease: the epidemiological evidence.  Int J Vitam Nutr Res. 2001;71(1):5-17
PubMed   |  Link to Article
Vogel JH, Bolling SF, Costello RB.  et al. American College of Cardiology Foundation Task Force on Clinical Expert Consensus Documents (Writing Committee to Develop an Expert Consensus Document on Complementary and Integrative Medicine).  Integrating complementary medicine into cardiovascular medicine: a report of the American College of Cardiology Foundation Task Force on Clinical Expert Consensus Documents (Writing Committee to Develop an Expert Consensus Document on Complementary and Integrative Medicine).  J Am Coll Cardiol. 2005;46(1):184-221
PubMed   |  Link to Article
Spinler SA, Hilleman DE, Cheng JW.  et al.  New recommendations from the 1999 American College of Cardiology/American Heart Association acute myocardial infarction guidelines.  Ann Pharmacother. 2001;35(5):589-617
PubMed   |  Link to Article
Santocono M, Zurria M, Berrettini M, Fedeli D, Falcioni G. Influence of astaxanthin, zeaxanthin and lutein on DNA damage and repair in UVA-irradiated cells.  J Photochem Photobiol B. 2006;85(3):205-215
PubMed   |  Link to Article
McAsey ME, Cady C, Jackson LM.  et al.  Time course of response to estradiol replacement in ovariectomized mice: brain apolipoprotein E and synaptophysin transiently increase and glial fibrillary acidic protein is suppressed.  Exp Neurol. 2006;197(1):197-205
PubMed   |  Link to Article
Henderson VW. Estrogen-containing hormone therapy and Alzheimer's disease risk: understanding discrepant inferences from observational and experimental research.  Neuroscience. 2006;138(3):1031-1039
PubMed   |  Link to Article
Dumas J, Hancur-Bucci C, Naylor M, Sites C, Newhouse P. Estrogen treatment effects on anticholinergic-induced cognitive dysfunction in normal postmenopausal women.  Neuropsychopharmacology. 2006;31(9):2065-2078
PubMed   |  Link to Article
Bimonte-Nelson HA, Francis KR, Umphlet CD, Granholm AC. Progesterone reverses the spatial memory enhancements initiated by tonic and cyclic oestrogen therapy in middle-aged ovariectomized female rats.  Eur J Neurosci. 2006;24(1):229-242
PubMed   |  Link to Article
Xu H, Wang R, Zhang YW, Zhang X. Estrogen, beta-amyloid metabolism/trafficking, and Alzheimer's disease.  Ann N Y Acad Sci. 2006;1089:324-342
PubMed   |  Link to Article
Barron AM, Fuller SJ, Verdile G, Martins RN. Reproductive hormones modulate oxidative stress in Alzheimer's disease.  Antioxid Redox Signal. 2006;8(11-12):2047-2059
PubMed   |  Link to Article
Singh M, Dykens JA, Simpkins JW. Novel mechanisms for estrogen-induced neuroprotection.  Exp Biol Med (Maywood). 2006;231(5):514-521
PubMed
Morrison JH, Brinton RD, Schmidt PJ, Gore AC. Estrogen, menopause, and the aging brain: how basic neuroscience can inform hormone therapy in women.  J Neurosci. 2006;26(41):10332-10348
PubMed   |  Link to Article
Easterbrook PJ, Berlin JA, Gopalan R, Matthews DR. Publication bias in clinical research.  Lancet. 1991;337(8746):867-872
PubMed   |  Link to Article
Kyzas PA, Denaxa-Kyza D, Ioannidis JP. Almost all articles on cancer prognostic markers report statistically significant results [published online ahead of print November 2, 2007]. Eur J Cancer
PubMed
Kjaergard LL, Gluud C. Citation bias of hepato-biliary randomized clinical trials.  J Clin Epidemiol. 2002;55(4):407-410
PubMed   |  Link to Article
Rosenfeld RM. Meta-analysis.  ORL J Otorhinolaryngol Relat Spec. 2004;66(4):186-195
PubMed   |  Link to Article
Ioannidis JP, Haidich AB, Pappa M.  et al.  Comparison of evidence of treatment effects in randomized and nonrandomized studies.  JAMA. 2001;286(7):821-830
PubMed   |  Link to Article
Lawlor DA, Davey Smith G, Kundu D, Bruckdorfer KR, Ebrahim S. Those confounded vitamins: what can we learn from the differences between observational vs randomised trial evidence?  Lancet. 2004;363(9422):1724-1727
PubMed   |  Link to Article
Stampfer M. Observational epidemiology is the preferred means of evaluating effects of behavioral and lifestyle modification.  Control Clin Trials. 1997;18(6):494-499
PubMed   |  Link to Article
Vandenbroucke JP. When are observational studies as credible as randomised trials?  Lancet. 2004;363(9422):1728-1731
PubMed   |  Link to Article
Gray JAM. Evidence-Based Healthcare. London, England: Churchill Livingstone; 1997
Djulbegovic B, Loughran TP Jr, Hornung CA.  et al.  The quality of medical evidence in hematology-oncology.  Am J Med. 1999;106(2):198-205
PubMed   |  Link to Article
Markovits H, Nantel G. The belief-bias effect in the production and evaluation of logical conclusions.  Mem Cognit. 1989;17(1):11-17
PubMed   |  Link to Article
Kaptchuk TJ. Effect of interpretive bias on research evidence.  BMJ. 2003;326(7404):1453-1455
PubMed   |  Link to Article
Koehler JJ. The influence of prior beliefs on scientific judgements of evidence quality.  Organ Behav Hum Decis Process. 1993;56:28-55
Link to Article
Resch KI, Ernst E, Garrow J. A randomized controlled study of reviewer bias against an unconventional therapy.  J R Soc Med. 2000;93(4):164-167
PubMed
MacCoun RJ. Biases in the interpretation and use of research results.  Annu Rev Psychol. 1998;49:259-287
PubMed   |  Link to Article
Bazerman MH, Loewenstein G, Moore DA. Why good accountants do bad audits.  Harv Bus Rev. 2002;80(11):97-102
PubMed
Ioannidis JP. Evolution and translation of research findings: from bench to where?  PLoS Clin Trials. 2006;1(7):e36
PubMed   |  Link to Article

Figures

Figure 1. Standardized Citation Counts for the 2 Highly Cited Epidemiological Studies on Vitamin E
Graphic Jump Location

Annual citation counts are standardized against the calendar year 1997 (citations received in 1997 are set at the standardized value of 100). The articles published in the same journal in 1993 received a total of 110 383 citations during 1993-2006.

Figure 2. Standardized Citation Counts for the Most Highly Cited Article on Effectiveness of Beta-Carotene in Cancer Prevention and Effectiveness of Estrogen in Alzheimer Dementia Prevention
Graphic Jump Location

Annual citation counts are standardized against the calendar year with the highest citation count for the highly cited article. For beta-carotene, the highest citation count was in 1985. The articles published in the same journal in 1981 as the highly cited article on beta-carotene received a total of 141 586 citations during 1981-2006. For estrogen, the highest citation count was in 1999. The articles published in the same journal in 1996 as the highly cited article on estrogen received a total of 65 300 citations during 1996-2006.

Tables

Table Graphic Jump LocationTable 1. Characteristics of Eligible Citing Articles for the 2 Highly Cited Vitamin E Studiesa
Table Graphic Jump LocationTable 2. Association of Characteristics of the Citing Articles With Their Overall Stance Toward Vitamin E Effectiveness in Cardiovascular Disease Prevention
Table Graphic Jump LocationTable 3. Association of Characteristics of the Citing Articles With a Favorable Stance Toward Vitamin E Effectiveness in Cardiovascular Disease Prevention
Table Graphic Jump LocationTable 4. Association of Characteristics of the Citing Articles With Their Overall Stance Toward Beta-Carotene for Cancer Prevention and Estrogen for Dementia Prevention

References

Ioannidis JP. Contradicted and initially stronger effects in highly cited clinical research.  JAMA. 2005;294(2):218-228
PubMed   |  Link to Article
Steinberg D, Parthasarathy S, Carew TE, Khoo JC, Witztum JL. Beyond cholesterol: modifications of low-density lipoprotein that increase its atherogenicity.  N Engl J Med. 1989;320(14):915-924
PubMed   |  Link to Article
Steinberg D, Witztum JL. Lipoproteins and atherogenesis: current concepts.  JAMA. 1990;264(23):3047-3052
PubMed   |  Link to Article
Steinberg D. Antioxidants and atherosclerosis: a current assessment.  Circulation. 1991;84(3):1420-1425
PubMed   |  Link to Article
Rimm EB, Stampfer MJ, Ascherio A.  et al.  Vitamin E consumption and the risk of coronary heart disease in men.  N Engl J Med. 1993;328(20):1450-1456
PubMed   |  Link to Article
Stampfer MJ, Hennekens CH, Manson JE.  et al.  Vitamin E consumption and the risk of coronary disease in women.  N Engl J Med. 1993;328(20):1444-1449
PubMed   |  Link to Article
Stephens NG, Parsons A, Schofield PM.  et al.  Randomised controlled trial of vitamin E in patients with coronary disease: Cambridge Heart Antioxidant Study (CHAOS).  Lancet. 1996;347(9004):781-786
PubMed   |  Link to Article
Yusuf S, Dagenais G, Pogue J, Bosch J, Sleight P. Vitamin E supplementation and cardiovascular events in high-risk patients: the Heart Outcomes Prevention Evaluation Study Investigators.  N Engl J Med. 2000;342(3):154-160
PubMed   |  Link to Article
Peto R, Doll R, Buckley JD, Sporn MB. Can dietary beta-carotene materially reduce human cancer rates?  Nature. 1981;290(5803):201-208
PubMed   |  Link to Article
 The effect of vitamin E and beta carotene on the incidence of lung cancer and other cancers in male smokers: the Alpha-Tocopherol, Beta Carotene Cancer Prevention Study Group.  N Engl J Med. 1994;330(15):1029-1035
PubMed   |  Link to Article
Omenn GS, Goodman GE, Thornquist MD.  et al.  Effects of a combination of beta carotene and vitamin A on lung cancer and cardiovascular disease.  N Engl J Med. 1996;334(18):1150-1155
PubMed   |  Link to Article
Hennekens CH, Buring JE, Manson JE.  et al.  Lack of effect of long-term supplementation with beta carotene on the incidence of malignant neoplasms and cardiovascular disease.  N Engl J Med. 1996;334(18):1145-1149
PubMed   |  Link to Article
Tang MX, Jacobs D, Stern Y.  et al.  Effect of oestrogen during menopause on risk and age at onset of Alzheimer's disease.  Lancet. 1996;348(9025):429-432
PubMed   |  Link to Article
Shumaker SA, Legault C, Kuller L.  et al. Women's Health Initiative Memory Study.  Conjugated equine estrogens and incidence of probable dementia and mild cognitive impairment in postmenopausal women: Women's Health Initiative Memory Study.  JAMA. 2004;291(24):2947-2958
PubMed   |  Link to Article
Espeland MA, Rapp SR, Shumaker SA.  et al. Women's Health Initiative Memory Study.  Conjugated equine estrogens and global cognitive function in postmenopausal women: Women's Health Initiative Memory Study.  JAMA. 2004;291(24):2959-2968
PubMed   |  Link to Article
Ioannidis JP, Haidich AB, Lau J. Any casualties in the clash of randomised and observational evidence?  BMJ. 2001;322(7291):879-880
PubMed   |  Link to Article
Patsopoulos NA, Analatos AA, Ioannidis JPA. Relative citation impact of various study designs in the health sciences.  JAMA. 2005;293(19):2362-2366
PubMed   |  Link to Article
Kavvoura FK, Liberopoulos G, Ioannidis JP. Selection in reported epidemiological risks: an empirical assessment.  PLoS Med. 2007;4(3):e79
PubMed   |  Link to Article
Buchanan AV, Weiss KM, Fullerton SM. Dissecting complex disease: the quest for the Philosopher's Stone?  Int J Epidemiol. 2006;35(3):562-571
PubMed   |  Link to Article
Prentice RL, Langer R, Stefanick ML.  et al.  Combined postmenopausal hormone therapy and cardiovascular disease: toward resolving the discrepancy between observational studies and the Women's Health Initiative clinical trial.  Am J Epidemiol. 2005;162(5):404-414
PubMed   |  Link to Article
Petitti DB, Freedman DA. Invited commentary: how far can epidemiologists get with statistical adjustment?  Am J Epidemiol. 2005;162(5):415-418
PubMed   |  Link to Article
Miller ER III, Pastor-Barriuso R, Dalal D.  et al.  Meta-analysis: high-dosage vitamin E supplementation may increase all-cause mortality.  Ann Intern Med. 2005;142(1):37-46
PubMed   |  Link to Article
Bjelakovic G, Nikolova D, Gluud LL, Simonetti RG, Gluud C. Mortality in randomized trials of antioxidant supplements for primary and secondary prevention: systematic review and meta-analysis.  JAMA. 2007;297(8):842-857
PubMed   |  Link to Article
Mosca L, Appel LJ, Benjamin EJ.  et al. American Heart Association.  Evidence-based guidelines for cardiovascular disease prevention in women.  Circulation. 2004;109(5):672-693
PubMed   |  Link to Article
Kris-Etherton PM, Lichtenstein AH, Howard BV, Steinberg D, Witztum JL. Antioxidant vitamin supplements and cardiovascular disease.  Circulation. 2004;110(5):637-641
PubMed   |  Link to Article
Thomson Scientific Institute for Scientific Information.  ISI Web of Knowledge. http://portal.isiknowledge.com/. Accessed October 7, 2007
Counsell C. Formulating questions and locating primary studies for inclusion in systematic reviews.  Ann Intern Med. 1997;127(5):380-387
PubMed   |  Link to Article
Hyman M, Pizzorno J, Weil A. A rational approach to antioxidant therapy and vitamin E.  Altern Ther Health Med. 2005;11(1):14-17
PubMed
Meydani M. Vitamin E and atherosclerosis: beyond prevention of LDL oxidation.  J Nutr. 2001;131(2):366S-368S
PubMed
Alissa EM, Bahjri SM, Al-Ama N, Ahmed WH, Starkey B, Ferns GA. Dietary vitamin A may be a cardiovascular risk factor in a Saudi population.  Asia Pac J Clin Nutr. 2005;14(2):137-144
PubMed
Cherubini A, Vigna GB, Zuliani G, Ruggiero C, Senin U, Fellin R. Role of antioxidants in atherosclerosis: epidemiological and clinical update.  Curr Pharm Des. 2005;11(16):2017-2032
PubMed   |  Link to Article
Eichholzer M, Luthy J, Gutzwiller F, Stahelin HB. The role of folate, antioxidant vitamins and other constituents in fruit and vegetables in the prevention of cardiovascular disease: the epidemiological evidence.  Int J Vitam Nutr Res. 2001;71(1):5-17
PubMed   |  Link to Article
Vogel JH, Bolling SF, Costello RB.  et al. American College of Cardiology Foundation Task Force on Clinical Expert Consensus Documents (Writing Committee to Develop an Expert Consensus Document on Complementary and Integrative Medicine).  Integrating complementary medicine into cardiovascular medicine: a report of the American College of Cardiology Foundation Task Force on Clinical Expert Consensus Documents (Writing Committee to Develop an Expert Consensus Document on Complementary and Integrative Medicine).  J Am Coll Cardiol. 2005;46(1):184-221
PubMed   |  Link to Article
Spinler SA, Hilleman DE, Cheng JW.  et al.  New recommendations from the 1999 American College of Cardiology/American Heart Association acute myocardial infarction guidelines.  Ann Pharmacother. 2001;35(5):589-617
PubMed   |  Link to Article
Santocono M, Zurria M, Berrettini M, Fedeli D, Falcioni G. Influence of astaxanthin, zeaxanthin and lutein on DNA damage and repair in UVA-irradiated cells.  J Photochem Photobiol B. 2006;85(3):205-215
PubMed   |  Link to Article
McAsey ME, Cady C, Jackson LM.  et al.  Time course of response to estradiol replacement in ovariectomized mice: brain apolipoprotein E and synaptophysin transiently increase and glial fibrillary acidic protein is suppressed.  Exp Neurol. 2006;197(1):197-205
PubMed   |  Link to Article
Henderson VW. Estrogen-containing hormone therapy and Alzheimer's disease risk: understanding discrepant inferences from observational and experimental research.  Neuroscience. 2006;138(3):1031-1039
PubMed   |  Link to Article
Dumas J, Hancur-Bucci C, Naylor M, Sites C, Newhouse P. Estrogen treatment effects on anticholinergic-induced cognitive dysfunction in normal postmenopausal women.  Neuropsychopharmacology. 2006;31(9):2065-2078
PubMed   |  Link to Article
Bimonte-Nelson HA, Francis KR, Umphlet CD, Granholm AC. Progesterone reverses the spatial memory enhancements initiated by tonic and cyclic oestrogen therapy in middle-aged ovariectomized female rats.  Eur J Neurosci. 2006;24(1):229-242
PubMed   |  Link to Article
Xu H, Wang R, Zhang YW, Zhang X. Estrogen, beta-amyloid metabolism/trafficking, and Alzheimer's disease.  Ann N Y Acad Sci. 2006;1089:324-342
PubMed   |  Link to Article
Barron AM, Fuller SJ, Verdile G, Martins RN. Reproductive hormones modulate oxidative stress in Alzheimer's disease.  Antioxid Redox Signal. 2006;8(11-12):2047-2059
PubMed   |  Link to Article
Singh M, Dykens JA, Simpkins JW. Novel mechanisms for estrogen-induced neuroprotection.  Exp Biol Med (Maywood). 2006;231(5):514-521
PubMed
Morrison JH, Brinton RD, Schmidt PJ, Gore AC. Estrogen, menopause, and the aging brain: how basic neuroscience can inform hormone therapy in women.  J Neurosci. 2006;26(41):10332-10348
PubMed   |  Link to Article
Easterbrook PJ, Berlin JA, Gopalan R, Matthews DR. Publication bias in clinical research.  Lancet. 1991;337(8746):867-872
PubMed   |  Link to Article
Kyzas PA, Denaxa-Kyza D, Ioannidis JP. Almost all articles on cancer prognostic markers report statistically significant results [published online ahead of print November 2, 2007]. Eur J Cancer
PubMed
Kjaergard LL, Gluud C. Citation bias of hepato-biliary randomized clinical trials.  J Clin Epidemiol. 2002;55(4):407-410
PubMed   |  Link to Article
Rosenfeld RM. Meta-analysis.  ORL J Otorhinolaryngol Relat Spec. 2004;66(4):186-195
PubMed   |  Link to Article
Ioannidis JP, Haidich AB, Pappa M.  et al.  Comparison of evidence of treatment effects in randomized and nonrandomized studies.  JAMA. 2001;286(7):821-830
PubMed   |  Link to Article
Lawlor DA, Davey Smith G, Kundu D, Bruckdorfer KR, Ebrahim S. Those confounded vitamins: what can we learn from the differences between observational vs randomised trial evidence?  Lancet. 2004;363(9422):1724-1727
PubMed   |  Link to Article
Stampfer M. Observational epidemiology is the preferred means of evaluating effects of behavioral and lifestyle modification.  Control Clin Trials. 1997;18(6):494-499
PubMed   |  Link to Article
Vandenbroucke JP. When are observational studies as credible as randomised trials?  Lancet. 2004;363(9422):1728-1731
PubMed   |  Link to Article
Gray JAM. Evidence-Based Healthcare. London, England: Churchill Livingstone; 1997
Djulbegovic B, Loughran TP Jr, Hornung CA.  et al.  The quality of medical evidence in hematology-oncology.  Am J Med. 1999;106(2):198-205
PubMed   |  Link to Article
Markovits H, Nantel G. The belief-bias effect in the production and evaluation of logical conclusions.  Mem Cognit. 1989;17(1):11-17
PubMed   |  Link to Article
Kaptchuk TJ. Effect of interpretive bias on research evidence.  BMJ. 2003;326(7404):1453-1455
PubMed   |  Link to Article
Koehler JJ. The influence of prior beliefs on scientific judgements of evidence quality.  Organ Behav Hum Decis Process. 1993;56:28-55
Link to Article
Resch KI, Ernst E, Garrow J. A randomized controlled study of reviewer bias against an unconventional therapy.  J R Soc Med. 2000;93(4):164-167
PubMed
MacCoun RJ. Biases in the interpretation and use of research results.  Annu Rev Psychol. 1998;49:259-287
PubMed   |  Link to Article
Bazerman MH, Loewenstein G, Moore DA. Why good accountants do bad audits.  Harv Bus Rev. 2002;80(11):97-102
PubMed
Ioannidis JP. Evolution and translation of research findings: from bench to where?  PLoS Clin Trials. 2006;1(7):e36
PubMed   |  Link to Article
CME
Meets CME requirements for:
Browse CME for all U.S. States
Accreditation Information
The American Medical Association is accredited by the Accreditation Council for Continuing Medical Education to provide continuing medical education for physicians. The AMA designates this journal-based CME activity for a maximum of 1 AMA PRA Category 1 CreditTM per course. Physicians should claim only the credit commensurate with the extent of their participation in the activity. Physicians who complete the CME course and score at least 80% correct on the quiz are eligible for AMA PRA Category 1 CreditTM.
Note: You must get at least of the answers correct to pass this quiz.
You have not filled in all the answers to complete this quiz
The following questions were not answered:
Sorry, you have unsuccessfully completed this CME quiz with a score of
The following questions were not answered correctly:
Commitment to Change (optional):
Indicate what change(s) you will implement in your practice, if any, based on this CME course.
Your quiz results:
The filled radio buttons indicate your responses. The preferred responses are highlighted
For CME Course: A Proposed Model for Initial Assessment and Management of Acute Heart Failure Syndromes
Indicate what changes(s) you will implement in your practice, if any, based on this CME course.

Multimedia

Some tools below are only available to our subscribers or users with an online account.

Web of Science® Times Cited: 81

Related Content

Customize your page view by dragging & repositioning the boxes below.

Articles Related By Topic
Related Collections
PubMed Articles