Poor-Quality Medical Research: Title and subTitle BreakWhat Can Journals Do?

There is considerable evidence that many published reports of randomized controlled trials (RCTs) are poor or even wrong, despite their clear importance.¹ The results of several reviews of published trials are briefly summarized in Table 1. Poor methodology and reporting are widespread.

Similar problems afflict other study types. A review of 308 phase 2 trials in cancer (295 of which were single-arm studies) found that 250 (81%) did not report an identifiable statistical design. Further, positive findings were reported in 48% of designed studies but 70% of studies with no reported design (P = .003).³ Of 40 molecular genetics articles published in leading general medical journals, 15 (38%) failed to meet at least 2 of 7 methodological standards. The authors wrote: "Without suitable attention to fundamental methodological standards, the expected benefits of molecular genetic testing may not be achieved."⁴

In recent years, systematic reviews have become common. In these, all reliable evidence relating to a clinical question is sought, systematically appraised, and, if suitable, combined statistically in a meta-analysis.⁵ A key component is an assessment of the methodological quality of the individual (primary) studies.⁶ Reviewers often conclude that the available evidence is of poor scientific quality,^{7 - 8} sometimes leading to heated debate about interpretation.⁹

General reviews also find much to be concerned about. Serious statistical errors were found in 40% of 164 articles published in a psychiatry journal¹⁰ and in 19% of 145 articles published in an obstetrics and gynecology journal.¹¹ I suspect that many basic errors have become less common, but statistics has become more complex, and there is evidence of frequent misapplication of newer advanced techniques.¹²

Also, when interpreting a study, readers need to know how it relates to existing knowledge. Many authors interpret their findings narrowly, failing to either identify previous studies or place their findings in the context of those previous studies.¹³

Errors in published research articles indicate poor research that has survived the peer-review process. But the problems arise earlier, so a more important question is, Why are submitted articles poor?

Much research is done without the benefit of anyone with adequate training in quantitative methods.¹⁴ Many investigators are not professional researchers; they are primarily clinicians. ". . . [I]f they had any training in research methods it was usually a single course in statistics in the first or second year of their degree, before they really appreciated how important rigorous research methods are in order to do good science."¹⁵ Also, training in statistics often focuses on data analysis, an emphasis reinforced by several statistics textbooks, often by nonstatisticians, in which design issues are not addressed.¹⁶ Yet study design is a crucial element of education in research methods and appropriately forms a key aspect of training in critical appraisal generally and evidence-based medicine in particular.^{17 - 18}

A contributory reason is inadequate review by research ethics committees (institutional review boards). Such review should detect studies with important flaws in design but clearly often fails to do so. Unfortunately, committees tend to use a narrow interpretation of ethics that downplays scientific quality, despite the clear ethical implications of allowing research that is not scientifically valid.¹⁹

A further issue is the copying of incorrect or inappropriate methods. Once incorrect procedures become common, it can be hard to stop them from spreading through the medical literature like a genetic mutation. Many editors have wrestled with the problem of authors objecting to a reviewer's criticism on the grounds that the same methods have appeared in previous articles, quite possibly by the same authors in the same journal. Examples of incorrect practices that persist despite published warnings include using the correlation to compare 2 methods of measurement,²⁰ using significance tests to compare baseline characteristics in randomized trials,²¹ conducting multiple tests of data recorded at multiple times,²² and ignoring the clustering in the design and analysis of cluster randomized trials.²³

Peer review can and should weed out serious methodological errors. However, expert methodological input is in short supply. Only a third of high-impact journals reported statistical review of all published manuscripts.²⁴ The vast majority of research is published in low-impact journals where peer review is undoubtedly less thorough.

Many readers seem to assume that articles published in peer-reviewed journals are scientifically sound, despite much evidence to the contrary. It is important, therefore, that misleading work be identified after publication.

As Gehlbach²⁵ noted, "[t]he ultimate interpretation and decision about the value of an article rests with the reader." Recent draft recommendations from the World Association of Medical Editors say that "[e]ditors should promote self-correction in science and participate in efforts to improve the practice of scientific investigation by . . . publishing corrections, retractions, and letters critical of articles published in their own journal."²⁶ Although journals do publish correspondence, there are weaknesses in the way they do so. Most obviously, editors select which letters to publish.

Editors should give special attention to letters making criticisms of methodology. They should do one of the following: satisfy themselves (perhaps by having the letter peer reviewed) that the criticisms are unfounded or unimportant, agree to publish the letter and invite the authors to respond, or invite a response from the authors and then decide whether to publish. Letters should not be rejected because of previously published correspondence (making different points) or lack of space.

Time limitation on correspondence denies readers the opportunity to draw attention to methodological deficiencies. Table 2 shows the current rules of 6 general medical journals. In effect, there is a statute of limitations by which authors of articles in these journals are immune to disclosure of methodological weaknesses once some arbitrary (short) period has elapsed, which cannot be right.

None of these journals suggests that there are exceptions, but from personal experience, at least 3 of them have occasionally published letters received beyond the stated time limit. The BMJ recently published a letter pointing out errors in an article published 6 years earlier. In it, Bland commented: "Potentially incorrect conclusions, based on faulty analysis, should not be allowed to remain in the literature to be cited uncritically by others."²⁷ A time limit discourages potential postpublication peer review; potential correspondents will surely be deterred by the unambiguous cutoff. Journals with such a policy should reconsider.

A few journals (eg, BMJ and the CMAJ) have rapid publication of correspondence on their Web pages. All (or most) letters are published, and there is no apparent time limit. Nor is there the same limit on length as in print journals (Table 2). Electronic letters are linked to the original publication and are relatively easily accessed. It is remarkable and disappointing that as yet so few journals have such a capability. Restricting the facility to current subscribers, as currently done by Neurology and Pediatrics, is inadequate. A weakness yet to be resolved is the absence of pressure on authors to respond to criticisms.²⁸ For such journals there is uncertainty about which version is definitive. Although the BMJ considers bmj.com to be the definitive version,²⁹ only the letters that appear in the print journal are indexed on MEDLINE.

Authors and editors should have the same goals: the advancement of scientific understanding and improvement in the treatment and prevention of disease. Poor research is the fault of authors, not journals. Poor research methods, unnecessary research, redundant or duplicate publication, thinly sliced study results, selective reporting, and scientific fraud, as well as a general tendency to inflate the importance of the results, should all be resisted vigorously. All could be less likely if research were not a career necessity for physicians.

Rather than abandon peer review, as some have suggested, journals should work to strengthen it. In particular, methodological review should be implemented much more widely. It will never be possible to eliminate misleading studies, but our imperfect peer-review system is a safeguard without which the quality of published research would be lower.

Journals can also help improve the literature by requiring the full and transparent reporting of research. Guidelines have been developed for RCTs,³⁰ systematic reviews and meta-analyses of RCTs³¹ and observational studies,³² and studies of diagnostic tests,³³ and other initiatives are under way. Editors should continue to be involved in the development of reporting recommendations and explicitly require authors to follow them.

Journals can enable and encourage the publication of research protocols.^{34 - 36} They can use their Web pages to publish extended versions of articles. They should also enable and encourage publication of the raw data used in medical research articles (eg, Clinical Chemistry and Neurology). If journals are willing to publish data, they should explicitly suggest this possibility to authors.