0
We're unable to sign you in at this time. Please try again in a few minutes.
Retry
We were able to sign you in, but your subscription(s) could not be found. Please try again in a few minutes.
Retry
There may be a problem with your account. Please contact the AMA Service Center to resolve this issue.
Contact the AMA Service Center:
Telephone: 1 (800) 262-2350 or 1 (312) 670-7827  *   Email: subscriptions@jamanetwork.com
Error Message ......
Review |

Data Extraction Errors in Meta-analyses That Use Standardized Mean Differences

Peter C. Gøtzsche, MD, DrMedSci; Asbjørn Hróbjartsson, MD, PhD; Katja Marić, MSc; Britta Tendal, MSc
JAMA. 2007;298(4):430-437. doi:10.1001/jama.298.4.430.
Text Size: A A A
Published online

Context Meta-analysis of trials that have used different continuous or rating scales to record outcomes of a similar nature requires sophisticated data handling and data transformation to a uniform scale, the standardized mean difference (SMD). It is not known how reliable such meta-analyses are.

Objective To study whether SMDs in meta-analyses are accurate.

Data Sources Systematic review of meta-analyses published in 2004 that reported a result as an SMD, with no language restrictions. Two trials were randomly selected from each meta-analysis. We attempted to replicate the results in each meta-analysis by independently calculating SMD using Hedges adjusted g.

Data Extraction Our primary outcome was the proportion of meta-analyses for which our result differed from that of the authors by 0.1 or more, either for the point estimate or for its confidence interval, for at least 1 of the 2 selected trials. We chose 0.1 as cut point because many commonly used treatments have an effect of 0.1 to 0.5, compared with placebo.

Results Of the 27 meta-analyses included in this study, we could not replicate the result for at least 1 of the 2 trials within 0.1 in 10 of the meta-analyses (37%), and in 4 cases, the discrepancy was 0.6 or more for the point estimate. Common problems were erroneous number of patients, means, standard deviations, and sign for the effect estimate. In total, 17 meta-analyses (63%) had errors for at least 1 of the 2 trials examined. For the 10 meta-analyses with errors of at least 0.1, we checked the data from all the trials and conducted our own meta-analysis, using the authors' methods. Seven of these 10 meta-analyses were erroneous (70%); 1 was subsequently retracted, and in 2 a significant difference disappeared or appeared.

Conclusions The high proportion of meta-analyses based on SMDs that show errors indicates that although the statistical process is ostensibly simple, data extraction is particularly liable to errors that can negate or even reverse the findings of the study. This has implications for researchers and implies that all readers, including journal reviewers and policy makers, should approach such meta-analyses with caution.

Figures in this Article

Sign in

Purchase Options

• Buy this article
• Subscribe to the journal
• Rent this article ?

Figures

Figure 1. Flowchart for Selection of Meta-analyses
Graphic Jump Location

RCT indicates randomized controlled trial; SMD, standardized mean differences.

Figure 2. Cases for Which Our Calculated Standardized Mean Differences (SMDs) Differed From That of the Authors
Graphic Jump Location

The meta-analyses differed by 0.1 or more for the point estimate or its 95% confidence interval (CI) for at least 1 of 2 selected trials. (For den Boer et al,19 we used the original signs for differences.) The size of the data markers indicates the relative weight of the data. IQR indicates interquartile range.

Figure 3. Standardized Mean Differences and 95% Confidence Intervals (CIs)
Graphic Jump Location

A trial of depression using the Beck Depression Inventory score and comparing self-directed bibliotherapy with cognitive behavioral therapy.45 There were 40 patients at baseline and 28 remained after 20 weeks. (We changed the sign for the effect as the author did.) The size of the data markers indicates the relative weight of the data.

Figure 4. Replication of the 10 Meta-analyses for Which Our Calculated Standardized Mean Differences for at Least 1 of 2 Selected Trials Differed From That of the Authors
Graphic Jump Location

Seven meta-analyses differed by 0.1 or more for the point estimate or its 95% confidence interval (CI). (For den Boer et al,19 we used the original signs for differences.) The size of the data markers indicates the relative weight of the data.

Tables

References

CME
Also Meets CME requirements for:
Browse CME for all U.S. States
Accreditation Information
The American Medical Association is accredited by the Accreditation Council for Continuing Medical Education to provide continuing medical education for physicians. The AMA designates this journal-based CME activity for a maximum of 1 AMA PRA Category 1 CreditTM per course. Physicians should claim only the credit commensurate with the extent of their participation in the activity. Physicians who complete the CME course and score at least 80% correct on the quiz are eligible for AMA PRA Category 1 CreditTM.
Note: You must get at least of the answers correct to pass this quiz.
Please click the checkbox indicating that you have read the full article in order to submit your answers.
Your answers have been saved for later.
You have not filled in all the answers to complete this quiz
The following questions were not answered:
Sorry, you have unsuccessfully completed this CME quiz with a score of
The following questions were not answered correctly:
Commitment to Change (optional):
Indicate what change(s) you will implement in your practice, if any, based on this CME course.
Your quiz results:
The filled radio buttons indicate your responses. The preferred responses are highlighted
For CME Course: A Proposed Model for Initial Assessment and Management of Acute Heart Failure Syndromes
Indicate what changes(s) you will implement in your practice, if any, based on this CME course.

Multimedia

Some tools below are only available to our subscribers or users with an online account.

1,506 Views
61 Citations

Sign in

Purchase Options

• Buy this article
• Subscribe to the journal
• Rent this article ?

Related Content

Customize your page view by dragging & repositioning the boxes below.

Articles Related By Topic
Related Collections
Jobs
JAMAevidence.com

The Rational Clinical Examination: Evidence-Based Clinical Diagnosis
Evidence to Support the Update

The Rational Clinical Examination: Evidence-Based Clinical Diagnosis
Evidence Summary and Review 2

×
brightcove.createExperiences();