0
We're unable to sign you in at this time. Please try again in a few minutes.
Retry
We were able to sign you in, but your subscription(s) could not be found. Please try again in a few minutes.
Retry
There may be a problem with your account. Please contact the AMA Service Center to resolve this issue.
Contact the AMA Service Center:
Telephone: 1 (800) 262-2350 or 1 (312) 670-7827  *   Email: subscriptions@jamanetwork.com
Error Message ......
Peer Review Congress |

Positive-Outcome Bias and Other Limitations in the Outcome of Research Abstracts Submitted to a Scientific Meeting FREE

Michael L. Callaham, MD; Robert L. Wears, MD; Ellen J. Weber, MD; Christopher Barton, MD; Gary Young, MD
[+] Author Affiliations

From the Division of Emergency Medicine (Drs Weber, Callaham, and Young), University of California, San Francisco; the Division of Emergency Medicine, University of Florida Health Center, Jacksonville (Dr Wears); the Department of Emergency Medicine, University of North Carolina, Chapel Hill (Dr Barton); and the Department of Emergency Medicine, Highland Hospital, Alameda County Medical Center, Oakland, Calif (Dr Young). Dr Young is now with Sacred Heart Medical Center, Eugene, Ore.


JAMA. 1998;280(3):254-257. doi:10.1001/jama.280.3.254.
Text Size: A A A
Published online

Context.— Studies with positive results are more likely to be published in biomedical journals than are studies with negative results. However, many studies submitted for consideration at scientific meetings are never published in full; bias in this setting is poorly studied.

Objective.— To identify features associated with the fate of research abstracts submitted to a scientific meeting.

Design and Setting.— Prospective observational cohort, with 5-year follow-up of all research submitted for consideration to the major annual 1991 US research meeting in the specialty of emergency medicine.

Participants.— All research abstracts submitted for consideration at the meeting for possible presentation.

Main Outcome Measures.— Characteristics associated with acceptance for presentation at the meeting and subsequent publication as a full manuscript.

Results.— A total of 492 research abstracts were submitted from programs in emergency medicine and other specialies affiliated with 103 US medical schools. A total of 179 (36%) were accepted for presentation and 214 (43%) were published in 44 journals. Of the 179 abstracts accepted for presentation, 111 studies were published. Scientific quality of abstracts or prestige of the journal in which the study was eventually published did not predict either of these outcomes. The best predictors (by logistic regression) of meeting acceptance were a subjective "originality" factor (odds ratio [OR], 2.07; 95% confidence interval [CI], 1.13-3.89) and positive results (OR, 1.99; 95% CI, 1.07-3.84), and, for publication, meeting acceptance (OR, 2.49; 95% CI, 1.49-4.35) and large sample size (OR, 2.26; 95% CI, 1.23-4.31). Forty-nine percent (241) of abstracts did not report on blinding, and 24% (118) did not report on randomization. Acceptance and publication were both more likely for positive outcomes (P=.03). Funnel plots showed the classic distribution of positive-outcome ("publication") bias at each of the submission, acceptance, and publication phases. Meeting acceptance predicted publication with a sensitivity of only 51%, specificity of 71%, positive predictive value of 57%, and negative predictive value of 66%.

Conclusions.— Positive-outcome bias was evident when studies were submitted for consideration and was amplified in the selection of abstracts for both presentation and publication, neither of which was strongly related to study design or quality.

Figures in this Article

POSITIVE-OUTCOME (also known as "publication") bias refers to the fact that research with positive outcomes is much more likely to be published than that with negative outcomes.14 Presentation of results in abstracts at scientific meetings is the first and often only publication for most biomedical research studies.5 However, the abstract selection process for meetings rarely has been studied. We, therefore, examined all research submitted for presentation at a national meeting to determine if positive-outcome bias was present in this process and what characteristics determined successful subsequent publication in a peer review journal.

The Society for Academic Emergency Medicine (SAEM) meeting is comparable to the meetings of 31 other societies of the Council of Academic Societies.6 Abstracts with mandatory structured formats were submitted and, independent of our study (and similar to other specialty meetings), each submission was evaluated by 5 to 7 blinded members of the SAEM screening committee (selected for their relevant expertise) and ranked on a 5-point Likert scale. Selection was based on an average score, and no journal had right of first refusal.

Four years after the 21st annual meeting of SAEM in 1991, all SAEM data were obtained by the authors and each submitted study was categorized according to design by a blinded Delphi panel. Since no established system exists for abstract classification, we modified a previously published approach.7 In addition, the review panel ranked each study for scientific quality and "originality" ("newsworthiness") on a Likert scale like the one previously validated.8 Institutional review board approval was obtained, and a detailed description of the methods used is available from the authors.

All authors' names were searched in MEDLINE in late 1995 to determine if the study had been published in any listed journal.9 For papers not found, the search was repeated in early 1996; if still not found, a questionnaire was sent to the authors and EMBASE and the Cochrane Collaboration databases were also searched.9 Journal impact factor was derived from the Science Citation Index for the year of publication.1013 The authors' institutions were ranked according to a system14,15 based on total dollars of National Institutes of Health (NIH) grant support.

There is no standardized definition of positive results.16 We used one of the more common definitions—that results were positive if the studied variable produced positive (beneficial) results.1,3,17,18 Some authors have defined positive results as those reporting statistically significant results (regardless of direction),4,5,16,1921 so we also used this definition.

We performed the major logistic regression analysis on those studies in which the subjects were either humans or animals, and the design was a prospective interventional trial, a prospective observational study, or a retrospective observational study. We also separately examined prospective studies with controls, excluding retrospective studies. We used a general iterative model-building strategy, as suggested by Hosmer and Lemeshow22 and Harrell,23 assessed for goodness of fit and subjected to bootstrap validation using S-Plus version 4.0, release 3 (Mathsoft Inc, Seattle, Wash) and Harrell extensions (http://www.lib.stat.cmu.edu). Correlation coefficients were calculated using JMP software 3.2.2 for Macintosh (SAS Institute, Inc, Cary, NC). We calculated the effect size of all interventions for the subgroup of prospective studies with controls in the usual fashion, except that SDs were not available.

Five hundred research abstracts were submitted to the SAEM selection committee from a total of 144 institutions, 103 of which had formal US medical school affiliations. Eight duplicates were deleted, leaving 492 abstracts as the basis of our study. The submitting schools averaged an NIH funding rank of 55 (of all US medical schools) (95% confidence interval [CI], 49-62).

Thirty percent of studies did not state a hypothesis, and 49% did not report on blinding, 24% on randomization, and 74% on exclusion criteria. Seventy-six percent of the studies were conducted on humans, 10% on animals, and the remainder on other models. Twenty-nine percent of the studies were retrospective, 27% prospective observational with control groups, and 26% prospective interventional trials. Sixty-six percent of all submitted interventional trials had positive outcome by our initial definition, 83% by the positive P value definition, and 80% by effect size. Respective figures for all submitted observational prospective studies were 70%, 92%, and 80%.

Three hundred eighty studies met the criteria for logistic regression (see "Methods"). Most measures of scientific merit did not predict the decision to accept an abstract for presentation. Instead, this decision was most strongly related to positive results and the reviewers' subjective originality score, while controlling for institutional funding, study design, randomization, blinding, controls, exclusion criteria, and sample size (Table 1). Results were similar for the subgroup of 166 prospective studies with control groups except that sample size greater than 50 was also predictive (odds ratio [OR], 2.4; 95% CI, 1.0-6.5) and similarly controlled. Full regression results are available from the authors.

Table Graphic Jump LocationTable 1.—Characteristics Predicting Acceptance for Presentation in Main Group of 380 Submitted Studies*

One hundred seventy-nine (36%) of the submitted abstracts were accepted for presentation at the meeting (11 reports on teaching methods were excluded). The SAEM committee scores determining acceptance correlated best with our subjective quality scale (R=0.57) and originality factor (R=0.49). Two hundred fourteen (43%) of the 492 studies submitted were published, an average of 18 months after presentation, in 44 journals with impact factors ranging from 0.23 to 24.5. A follow-up questionnaire to authors of unpublished papers was returned by 226 authors and identified 21 publications not found in MEDLINE.9 One hundred four (49%) of the 214 studies ultimately published were rejected for presentation at the meeting. The mean impact factor of the publishing journal did not differ for those papers rejected for the meeting vs those accepted (1.48 vs 1.19; P =.47), nor did time to publication.

One hundred forty-seven studies (70%) were published in emergency medicine specialty journals. The remaining studies were published in 39 other journals, including American Journal of Public Health, Annals of Internal Medicine, JAMA , The New England Journal of Medicine, Pediatrics, and Stroke. Of all the studies published after the meeting, 38% had been published 1 year later, 68% in 2 years, 88% in 3 years, and 95% in 4 years.

Publication of a full manuscript in the main group of 380 studies was related most strongly to abstract acceptance and sample size, controlling for the previous variables by logistic regression (Table 2). Results in the 166 prospective studies with control groups were identical. Results of these analyses did not differ using either of the 2 definitions of positive outcome ("Methods").

Table Graphic Jump LocationTable 2.—Characteristics Predicting Peer-Reviewed Publication of a Manuscript in Main Group of 380 Submitted Studies*

Data allowing calculation of effect size was reported in only 122 (66%) of 186 prospective studies. Positive-outcome bias was evident for all studies submitted before any screening (Figure 1). Submissions with less positive effect size were then disproportionately rejected for presentation. At the level of publication of full manuscripts, the same bias again appeared. A funnel plot (Figure 1) shows the absence of expected negative effect sizes at low sample size, which is the hallmark of positive-outcome bias.

Graphic Jump Location
Funnel plot of all 122 prospective studies submitted to the meeting that reported effect sizes (see text). Studies subsequently accepted for presentation at the meeting are plotted separately from those rejected. Inset graph similarly shows results at the journal publication level. The absence of the left (negative) base of the inverted funnel suggests the presence of selection bias in favor of positive outcomes.

Numerical testing confirms the funnel plots. The mean effect size of all submitted papers was 0.71 (95% CI, 0.40-1.01). The mean effect size of papers accepted for the meeting was 0.92 vs 0.45 for those rejected. The mean effect size of papers eventually published was 0.96 vs 0.45 for those never published (P=.03, Kruskal-Wallis analysis of variance). Effect size contributed much more to acceptance or publication than study sample size.

Our study was limited to 1 specialty, but 103 medical schools contributed and the publication rate was comparable with 31 other academic society meetings.6 Research from this meeting was published in 44 journals, 39 of them outside this specialty. The emergency medicine literature, the abstracts we studied, and the general medicine literature are identical in the proportion of studies with positive outcomes.1 Effect size data were available for only a minority of studies, and most studies were not randomized controlled trials. These subgroups might not be representative of all 492 abstracts, but the results between groups were similar and consistent.

Presentation of scientific studies at meetings is an important part of the dissemination of knowledge, but half of these studies appear only as abstracts and never undergo any other peer review.5 Whether the abbreviated peer review used to select abstracts for meetings actually identifies scientific merit is unknown, yet abstracts are cited as often as fully published papers.24

We reviewed all submitted research, not just studies accepted for presentation, assessing those characteristics previously suggested to predict publication.25,14,15 Our results show that acceptance of an abstract for presentation at the meeting was not strongly related to study design, methods, sample size, or even a subjective quality score. Instead, a subjective "originality" factor and presence of positive results best predicted acceptance (ORs, 2.07 and 1.99, respectively), regardless of study design.

Publication as a full manuscript was best predicted by whether the abstract had been accepted at the meeting (OR, 2.49) and large sample size (OR, 2.26), again independent of study design or scientific quality. Positive studies were preferentially accepted during both the acceptance and publication decisions (P=.03), which is illustrated in the funnel plots (Figure 1).

Positive-outcome bias has been documented previously in publication of full journal articles, but not in detail at the meeting acceptance level.3,4,18 Four studies examined, in limited ways, the publication of studies after acceptance for presentation at meetings.5,2527 Abstracts submitted on the single subject of gestational exposure to cocaine demonstrated positive-outcome bias in acceptance.28 The impact of sample size and positive outcome on acceptance and publication was reported for cancer abstracts.24

Our study examined all submitted research from a broad cross-section of institutions, with subsequent publication in a broad variety of journals. Positive-effect (or publication) bias was already present when studies were first submitted for consideration (Figure 1). Presumably this was due to authors who did not complete or submit smaller studies with negative effects, perhaps after experiencing a tradition of publication bias by meeting selection committees and scientific journals.

The selection process for presentation at the meeting further increased this positive-outcome bias (Figure 1). Logistic regression showed that an intangible "originality" ("newsworthiness") factor and positive outcome were more strongly associated with acceptance than traditional measures of scientific quality, such as study design, randomization, sample size, and blinding (Table 1 and Table 2).

Positive-outcome bias appeared again in the selection process for publication in a journal. Full publication was best predicted by acceptance at the meeting and study size, rather than study methods or quality. Our results confirm a smaller study of the cancer literature, which did not control for scientific quality.24

A number of potential solutions, such as trials registries, have been proposed to remedy positive-outcome bias.21,29,30 We offer one more solution: that all studies submitted to scientific meetings be published as abstracts, indicating whether or not they were chosen to be presented. This might encourage researchers to submit studies with negative findings, and readers and researchers could more easily identify the entire spectrum of research. Journals might adopt a similar practice, publishing the abstracts of all submitted manuscripts.

Despite the mandatory structured format, 49% of SAEM abstracts failed to report adequately about blinding, 74% about exclusion criteria, 24% about randomization, and 14% about sample size. Perhaps because these deficiencies made the merit of the research difficult to evaluate, acceptance for presentation at the meeting predicted publication as a full manuscript with a sensitivity of only 51%, a specificity of 71%, a positive predictive value of 57%, and a negative predictive value of 66%.

Moscati R, Jehle D, Ellis D, Fiorello A, Landi M. Positive-outcome bias: comparison of emergency medicine and general medicine literatures.  Acad Emerg Med.1994;1:267-271.
Dickersin K, Chan S, Chalmers TC, Sacks HS, Smith Jr H. Publication bias and clinical trials.  Control Clin Trials.1987;8:343-353.
Dickersin K, Min Y-I, Meinert CL. Factors influencing publication of research results: follow-up of applications submitted to two institutional review boards.  JAMA.1992;267:374-378.
Easterbrook PJ, Berlin JA, Gopalan R, Matthews DR. Publication bias in clinical research.  Lancet.1991;337:867-872.
Scherer RW, Dickersin K, Langenberg P. Full publication of results initially presented in abstracts: a meta-analysis.  JAMA.1994;272:158-162.
Wuerz R, Holliman J. Attendance and research abstract activity at the 1993 annual meetings of the academic medical societies.  Acad Emerg Med.1994;1:A59.
Chalmers T, Smith H, Blackburn B.  et al.  A method for assessing the quality of a randomized controlled trial.  Control Clin Trials.1981;2:31-49.
Oxman AD, Guyatt GH. Validation of an index of the quality of review articles.  J Clin Epidemiol.1991;44:1271-1278.
Weber EJ, Callaham ML, Wears RL, Barton C, Young G. Unpublished research from a medical specialty meeting: why investigators fail to publish.  JAMA.1998;280:257-259.
Garfield E. SCI Journal Citation Reports: A Bibliometric Analysis of Science Journals in the ISI Database . Philadelphia, Pa: Institute for Science Information Inc; 1991.
Garfield E. SCI Journal Citation Reports: A Bibliometric Analysis of Science Journals in the ISI Database . Philadelphia, Pa: Institute for Science Information Inc; 1992.
Garfield E. SCI Journal Citation Reports: A Bibliometric Analysis of Science Journals in the ISI Database . Philadelphia, Pa: Institute for Science Information Inc; 1993.
Garfield E. SCI Journal Citation Reports: A Bibliometric Analysis of Science Journals in the ISI Database . Philadelphia, Pa: Institute for Science Information Inc; 1994.
Gallagher E, Goldfrank L, Anderson G.  et al.  Current status of academic emergency medicine within academic medicine in the United States.  Acad Emerg Med.1994;1:41-46.
Garfunkel JM, Ulshen MH, Hamrick HJ, Lawson EE. Effect of institutional prestige on reviewers' recommendations and editorial decisions.  JAMA.1994;272:137-138.
Olson CM. Publication bias.  Acad Emerg Med.1994;1:207-209.
Simes RJ. Publication bias: the case for an international registry of clinical trials.  J Clin Oncol.1986;4:1529-1541.
Dickersin K, Min YI. NIH clinical trials and publication bias.  Online J Curr Clin Trials [serial online].1993;doc 50.
Rennie D, Flanagin A. Publication bias: the triumph of hope over experience.  JAMA.1992;267:411-412.
Berlin JA. Will publication bias vanish in the age of online journals?  Online J Curr Clin Trials [serial online].1992;doc 12.
Begg CB, Berlin JA. Publication bias and dissemination of clinical research.  J Natl Cancer Inst.1989;81:107-115.
Hosmer D, Lemeshow S. Applied Logistic Regression . New York, NY: John Wiley & Sons Inc; 1989.
Harrell FE. Predicting Outcomes: Applied Survival Analysis and Logistic Regression . Charlottesville: Dept of Health Evaluation Sciences, School of Medicine, University of Virginia; 1996.
de Bellefeuille C, Morrison C, Tannock I. The fate of abstracts submitted to a cancer meeting: factors which influence presentation and subsequent publication.  Ann Oncol.1992;3:187-191.
Goldman L, Loscalzo A. Fate of cardiology research originally published in abstract form.  N Engl J Med.1980;303:255-259.
Meranze J, Ellison N, Greenhow D. Publications reseulting from anesthesia meeting abstracts.  Anesth Analg.1982;61:445-448.
McCormick M, Holmes J. Publication of research presented at the pediatric meetings.  AJDC.1985;139:122-126.
Koren G, Graham K, Shear H, Einarson T. Bias against the null hypothesis: the reproductive hazards of cocaine.  Lancet.1989;2:1440-1444.
Newcombe RG. Towards a reduction in publication bias.  BMJ.1987;295:656-659.
Dickersin K. The existence of publication bias and risk factors for its occurrence.  JAMA.1990;263:1385-1389.
Weintraub WH. Are published manuscripts representative of the surgical meeting abstracts? an objective appraisal.  J Pediatr Surg.1987;22:11-13.

Figures

Graphic Jump Location
Funnel plot of all 122 prospective studies submitted to the meeting that reported effect sizes (see text). Studies subsequently accepted for presentation at the meeting are plotted separately from those rejected. Inset graph similarly shows results at the journal publication level. The absence of the left (negative) base of the inverted funnel suggests the presence of selection bias in favor of positive outcomes.

Tables

Table Graphic Jump LocationTable 1.—Characteristics Predicting Acceptance for Presentation in Main Group of 380 Submitted Studies*
Table Graphic Jump LocationTable 2.—Characteristics Predicting Peer-Reviewed Publication of a Manuscript in Main Group of 380 Submitted Studies*

References

Moscati R, Jehle D, Ellis D, Fiorello A, Landi M. Positive-outcome bias: comparison of emergency medicine and general medicine literatures.  Acad Emerg Med.1994;1:267-271.
Dickersin K, Chan S, Chalmers TC, Sacks HS, Smith Jr H. Publication bias and clinical trials.  Control Clin Trials.1987;8:343-353.
Dickersin K, Min Y-I, Meinert CL. Factors influencing publication of research results: follow-up of applications submitted to two institutional review boards.  JAMA.1992;267:374-378.
Easterbrook PJ, Berlin JA, Gopalan R, Matthews DR. Publication bias in clinical research.  Lancet.1991;337:867-872.
Scherer RW, Dickersin K, Langenberg P. Full publication of results initially presented in abstracts: a meta-analysis.  JAMA.1994;272:158-162.
Wuerz R, Holliman J. Attendance and research abstract activity at the 1993 annual meetings of the academic medical societies.  Acad Emerg Med.1994;1:A59.
Chalmers T, Smith H, Blackburn B.  et al.  A method for assessing the quality of a randomized controlled trial.  Control Clin Trials.1981;2:31-49.
Oxman AD, Guyatt GH. Validation of an index of the quality of review articles.  J Clin Epidemiol.1991;44:1271-1278.
Weber EJ, Callaham ML, Wears RL, Barton C, Young G. Unpublished research from a medical specialty meeting: why investigators fail to publish.  JAMA.1998;280:257-259.
Garfield E. SCI Journal Citation Reports: A Bibliometric Analysis of Science Journals in the ISI Database . Philadelphia, Pa: Institute for Science Information Inc; 1991.
Garfield E. SCI Journal Citation Reports: A Bibliometric Analysis of Science Journals in the ISI Database . Philadelphia, Pa: Institute for Science Information Inc; 1992.
Garfield E. SCI Journal Citation Reports: A Bibliometric Analysis of Science Journals in the ISI Database . Philadelphia, Pa: Institute for Science Information Inc; 1993.
Garfield E. SCI Journal Citation Reports: A Bibliometric Analysis of Science Journals in the ISI Database . Philadelphia, Pa: Institute for Science Information Inc; 1994.
Gallagher E, Goldfrank L, Anderson G.  et al.  Current status of academic emergency medicine within academic medicine in the United States.  Acad Emerg Med.1994;1:41-46.
Garfunkel JM, Ulshen MH, Hamrick HJ, Lawson EE. Effect of institutional prestige on reviewers' recommendations and editorial decisions.  JAMA.1994;272:137-138.
Olson CM. Publication bias.  Acad Emerg Med.1994;1:207-209.
Simes RJ. Publication bias: the case for an international registry of clinical trials.  J Clin Oncol.1986;4:1529-1541.
Dickersin K, Min YI. NIH clinical trials and publication bias.  Online J Curr Clin Trials [serial online].1993;doc 50.
Rennie D, Flanagin A. Publication bias: the triumph of hope over experience.  JAMA.1992;267:411-412.
Berlin JA. Will publication bias vanish in the age of online journals?  Online J Curr Clin Trials [serial online].1992;doc 12.
Begg CB, Berlin JA. Publication bias and dissemination of clinical research.  J Natl Cancer Inst.1989;81:107-115.
Hosmer D, Lemeshow S. Applied Logistic Regression . New York, NY: John Wiley & Sons Inc; 1989.
Harrell FE. Predicting Outcomes: Applied Survival Analysis and Logistic Regression . Charlottesville: Dept of Health Evaluation Sciences, School of Medicine, University of Virginia; 1996.
de Bellefeuille C, Morrison C, Tannock I. The fate of abstracts submitted to a cancer meeting: factors which influence presentation and subsequent publication.  Ann Oncol.1992;3:187-191.
Goldman L, Loscalzo A. Fate of cardiology research originally published in abstract form.  N Engl J Med.1980;303:255-259.
Meranze J, Ellison N, Greenhow D. Publications reseulting from anesthesia meeting abstracts.  Anesth Analg.1982;61:445-448.
McCormick M, Holmes J. Publication of research presented at the pediatric meetings.  AJDC.1985;139:122-126.
Koren G, Graham K, Shear H, Einarson T. Bias against the null hypothesis: the reproductive hazards of cocaine.  Lancet.1989;2:1440-1444.
Newcombe RG. Towards a reduction in publication bias.  BMJ.1987;295:656-659.
Dickersin K. The existence of publication bias and risk factors for its occurrence.  JAMA.1990;263:1385-1389.
Weintraub WH. Are published manuscripts representative of the surgical meeting abstracts? an objective appraisal.  J Pediatr Surg.1987;22:11-13.
CME
Also Meets CME requirements for:
Browse CME for all U.S. States
Accreditation Information
The American Medical Association is accredited by the Accreditation Council for Continuing Medical Education to provide continuing medical education for physicians. The AMA designates this journal-based CME activity for a maximum of 1 AMA PRA Category 1 CreditTM per course. Physicians should claim only the credit commensurate with the extent of their participation in the activity. Physicians who complete the CME course and score at least 80% correct on the quiz are eligible for AMA PRA Category 1 CreditTM.
Note: You must get at least of the answers correct to pass this quiz.
Your answers have been saved for later.
You have not filled in all the answers to complete this quiz
The following questions were not answered:
Sorry, you have unsuccessfully completed this CME quiz with a score of
The following questions were not answered correctly:
Commitment to Change (optional):
Indicate what change(s) you will implement in your practice, if any, based on this CME course.
Your quiz results:
The filled radio buttons indicate your responses. The preferred responses are highlighted
For CME Course: A Proposed Model for Initial Assessment and Management of Acute Heart Failure Syndromes
Indicate what changes(s) you will implement in your practice, if any, based on this CME course.

Multimedia

Some tools below are only available to our subscribers or users with an online account.

Web of Science® Times Cited: 153

Related Content

Customize your page view by dragging & repositioning the boxes below.