0
We're unable to sign you in at this time. Please try again in a few minutes.
Retry
We were able to sign you in, but your subscription(s) could not be found. Please try again in a few minutes.
Retry
There may be a problem with your account. Please contact the AMA Service Center to resolve this issue.
Contact the AMA Service Center:
Telephone: 1 (800) 262-2350 or 1 (312) 670-7827  *   Email: subscriptions@jamanetwork.com
Error Message ......
Brief Report |

Accuracy of Data in Abstracts of Published Research Articles FREE

Roy M. Pitkin, MD; Mary Ann Branagan; Leon F. Burmeister, PhD
[+] Author Affiliations

Author Affiliations:Obstetrics & Gynecology, Los Angeles, Calif (Dr Pitkin); Chest, Northbrook, Ill (Ms Branagan); and Department of Preventive Medicine, University of Iowa, Iowa City (Dr Burmeister).


JAMA. 1999;281(12):1110-1111. doi:10.1001/jama.281.12.1110.
Text Size: A A A
Published online

Context The section of a research article most likely to be read is the abstract, and therefore it is particularly important that the abstract reflect the article faithfully.

Objective To assess abstracts accompanying research articles published in 6 medical journals with respect to whether data in the abstract could be verified in the article itself.

Design Analysis of simple random samples of 44 articles and their accompanying abstracts published during 1 year (July 1, 1996-June 30, 1997) in each of 5 major general medical journals (Annals of Internal Medicine, BMJ, JAMA, Lancet, and New England Journal of Medicine) and a consecutive sample of 44 articles published during 15 months (July 1, 1996-August 15, 1997) in the CMAJ.

Main Outcome Measure Abstracts were considered deficient if they contained data that were either inconsistent with corresponding data in the article's body (including tables and figures) or not found in the body at all.

Results The proportion of deficient abstracts varied widely (18%-68%) and to a statistically significant degree (P<.001) among the 6 journals studied.

Conclusions Data in the abstract that are inconsistent with or absent from the article's body are common, even in large-circulation general medical journals.

The abstract accompanying a research article, because it is often the only part of the article that will be read, should reflect fully and accurately the work reported. We observed in 1 medical specialty journal that a quarter or more of manuscripts returned after revision contained data in the abstract that could not be verified in the body of the paper.1 If this problem were to persist in published articles, then a potential for misinterpretation would exist. In the present study, we surveyed research articles and their accompanying abstracts published recently in 6 medical journals to verify data in the abstract by relating them to corresponding data in the body of the report.

Articles studied included simple random samples of reports of original research (including meta-analyses but not other types of reviews) appearing in 5 medical journals between July 1, 1996, and June 30, 1997 (Annals of Internal Medicine, BMJ, JAMA, Lancet, and New England Journal of Medicine); all articles appearing in a sixth journal CMAJ (Canadian Medical Association Journal), between July 1, 1996, and August 15, 1997, were also studied. Additional inclusion criteria were (1) the article was accompanied by an abstract and (2) the article occupied at least 2 full journal pages.

To estimate the sample sizes, we used some preliminary observations1 that 25% to 50% of articles published in 2 of the journals studied contained 1 or more deficiencies in abstracts. We assumed this rate would range from 10% to 40% across the 6 journals studied and that α was .05 and power was 0.8, yielding a projected sample size of 44 from each journal. From each of the 5 journals that published more than 44 research articles in the 2 volumes studied (July 1, 1996-June 30, 1997), we selected a computer-generated simple random sample of 44. From the CMAJ, we analyzed a consecutive cohort of all 44 articles published from July 1, 1996, through August 15, 1997.

For each selected article, the abstract was scrutinized by 1 of 3 examiners who identified each datum or other piece of information in the abstract and then sought to relate it to its source in the body of the article, including tables and figures. Two types of discrepancies were sought: (1) data given differently in the abstract and the body and (2) data given in the abstract but not in the body. If either was identified, the abstract was considered deficient. Discrepancies attributable to rounding were not considered to be deficiencies as long as the rounding was done appropriately, and the rounded value appeared in the abstract and the more detailed value in the body.

The proportions of articles containing deficiencies were compared across journals by χ2 analysis. On the basis of normal approximation, 95% confidence intervals (CIs) were calculated for each proportion. We also performed a validation study by randomly selecting (using another computer-generated random number sequence) 7 of each set of 44 articles and having these examined by a second (and different) examiner.

Table 1 contains the proportions of deficient abstracts and 95% CIs for each journal, tabulated considering the abstract as the unit, as well as the types of deficiencies found in the 6 journals. The proportion of deficient abstracts ranged from a low of 18% to a high of 68%. Inconsistency between abstract and body was generally more common than omitted data (ie, data in the abstract not found in the body). A substantial proportion of deficient abstracts contained both kinds of defects (25/104; 24%).

In the validation study, 38 of the 42 paired comparisons were concordant with respect to identification of deficiencies. The κ value for agreement between the 2 evaluators was 0.81 (z = 5.22; P<.001).

The frequency with which we found abstracts to be inaccurate, in the sense of containing information not verifiable in the article's main body (including tables and figures, as well as text) was surprisingly large, ranging from 18% to 68% in the 6 journals surveyed. The more common type of the 2 deficiencies was inconsistency between data in the abstract and those in the body. Giving data or other information in the abstract but not in the body was somewhat less common. These findings are all the more surprising considering that the journals studied are all prominent and highly regarded general medical publications whose editors were founding members of the International Committee of Medical Journal Editors, a respected standard-setting body. These journals have full-time professional staffs who can be presumed to devote a good deal of time and energy to editorial and production processes.

Many of the discrepancies identified were quite minor and not likely to cause serious misinterpretation. For example, 1 abstract2 reported the population to consist of "42 consecutive patients," whereas the body indicated it to be "44 consecutive patients of which 42 agreed to participate." Sometimes, however, discrepancies were more serious; for example, 1 abstract3 gave the estimated 15-year survival as 48%, whereas the body of the text indicated it to be 58%.

The specific question we asked in this study—Can the data and other information in the abstract be verified in the body of the article?—does not seem to have been examined before. Previous studies4,5 of abstract quality generally involved overall or global assessment. Most of the recent literature on abstracts has concerned structured abstracts, introduced in 19876 with the goal of making abstracts more informative. Several investigations79 indicated that structured abstracts are actually better in quality, more informative, more readable, and a more efficient use of readers' time. Structured abstracts may well offer all of these advantages, but there is little reason to expect them to reduce the types of deficiencies assessed in this study. Indeed, if structured abstracts are more informative (ie, if they provide more information), they might be more likely to be subject to deficiencies we assessed. In the present study, we could not discern any relationship between various structured formats and the deficiencies assessed.

It is important to acknowledge that we addressed only 1 aspect of abstract accuracy in asking if what is in the abstract is consistent with the body of the article. There is another, at least equally important question: Is the important information in the article found in the abstract? Our study was not designed to address this question.

We found previously1 that providing authors with specific instructions about abstract accuracy when they are revising manuscripts is ineffective in preventing the types of defects assessed in this study. If it is important that abstracts be as accurate as possible—and it can hardly be argued otherwise—and if authors cannot be counted on to provide this level of accuracy, the responsibility must be taken by journals' editorial staffs. As part of the copyediting process, the abstract needs to be scrutinized painstakingly on a line-by-line or even word-by-word basis and each bit of information verified individually and specifically.

Pitkin RM, Branagan MA. Can the accuracy of abstracts be improved by providing specific instructions? a randomized controlled trial.  JAMA.1998;280:267-269.
Rothwell PM, McDowell Z, Wong CK, Dorman PJ. Doctors and patients don't agree: cross-sectional study of patients' and doctors' perceptions and assessments of disability in multiple sclerosis.  BMJ.1997;314:1580-1583.
Socié G, Mary J-Y, de Gramont A.  et al. , for the French Society of Haematology.  Paroxysmal nocturnal haemoglobinuria: long-term follow-up and prognostic factors.  Lancet.1996;348:573-577.
Goodman SN, Berlin J, Fletcher SW, Fletcher RH. Manuscript quality before and after peer review and editing at Annals of Internal Medicine Ann Intern Med.1994;121:11-21.
Froom P, Froom J. Deficiencies in structured medical abstracts.  J Clin Epidemiol.1993;46:591-594.
Ad Hoc Working Group for Critical Appraisal of the Medical Literature.  A proposal for more informative abstracts of clinical articles.  Ann Intern Med.1987;106:598-604.
Comans ML, Overbeke AJ. The structural summary: a tool for reader and author.  Ned Tijdschr Geneeskd.1990;134:2338-2343.
Narine L, Yee DX, Einarson TR, Ilersich AL. The quality of journal abstracts for research articles.  CMAJ.1991;144:449-453.
Taddio A, Pain T, Fassos FF, Boon H, Ilersich AL, Einarson TR. Quality of nonstructured and structured abstracts of original research articles in the British Medical Journal, the Canadian Medical Association Journal and the Journal of the American Medical Association CMAJ.1994;150:1611-1615.

Figures

References

Pitkin RM, Branagan MA. Can the accuracy of abstracts be improved by providing specific instructions? a randomized controlled trial.  JAMA.1998;280:267-269.
Rothwell PM, McDowell Z, Wong CK, Dorman PJ. Doctors and patients don't agree: cross-sectional study of patients' and doctors' perceptions and assessments of disability in multiple sclerosis.  BMJ.1997;314:1580-1583.
Socié G, Mary J-Y, de Gramont A.  et al. , for the French Society of Haematology.  Paroxysmal nocturnal haemoglobinuria: long-term follow-up and prognostic factors.  Lancet.1996;348:573-577.
Goodman SN, Berlin J, Fletcher SW, Fletcher RH. Manuscript quality before and after peer review and editing at Annals of Internal Medicine Ann Intern Med.1994;121:11-21.
Froom P, Froom J. Deficiencies in structured medical abstracts.  J Clin Epidemiol.1993;46:591-594.
Ad Hoc Working Group for Critical Appraisal of the Medical Literature.  A proposal for more informative abstracts of clinical articles.  Ann Intern Med.1987;106:598-604.
Comans ML, Overbeke AJ. The structural summary: a tool for reader and author.  Ned Tijdschr Geneeskd.1990;134:2338-2343.
Narine L, Yee DX, Einarson TR, Ilersich AL. The quality of journal abstracts for research articles.  CMAJ.1991;144:449-453.
Taddio A, Pain T, Fassos FF, Boon H, Ilersich AL, Einarson TR. Quality of nonstructured and structured abstracts of original research articles in the British Medical Journal, the Canadian Medical Association Journal and the Journal of the American Medical Association CMAJ.1994;150:1611-1615.

Letters

CME
Meets CME requirements for:
Browse CME for all U.S. States
Accreditation Information
The American Medical Association is accredited by the Accreditation Council for Continuing Medical Education to provide continuing medical education for physicians. The AMA designates this journal-based CME activity for a maximum of 1 AMA PRA Category 1 CreditTM per course. Physicians should claim only the credit commensurate with the extent of their participation in the activity. Physicians who complete the CME course and score at least 80% correct on the quiz are eligible for AMA PRA Category 1 CreditTM.
Note: You must get at least of the answers correct to pass this quiz.
You have not filled in all the answers to complete this quiz
The following questions were not answered:
Sorry, you have unsuccessfully completed this CME quiz with a score of
The following questions were not answered correctly:
Commitment to Change (optional):
Indicate what change(s) you will implement in your practice, if any, based on this CME course.
Your quiz results:
The filled radio buttons indicate your responses. The preferred responses are highlighted
For CME Course: A Proposed Model for Initial Assessment and Management of Acute Heart Failure Syndromes
Indicate what changes(s) you will implement in your practice, if any, based on this CME course.

Multimedia

Some tools below are only available to our subscribers or users with an online account.

Related Content

Customize your page view by dragging & repositioning the boxes below.

Articles Related By Topic