Evaluating the BMJ Guidelines for Economic Submissions:  Prospective Audit of Economic Submissions to BMJ and The Lancet FREE

HEALTH ECONOMICS literature has increased in the last 10 years,¹ but many of the published articles are of low quality.^2- 5 Poor economic evaluations are not only wasteful of scarce resources but also misleading. Although guidelines for economic evaluations have been promulgated by governments,^6- 8 the pharmaceutical industry, and groups of researchers,⁹ little is known about editorial practices relating to economic studies.¹⁰ Furthermore, few medical journals have written policies, and even fewer have a clear idea of what a "good" economic evaluation is.¹¹ In January 1995, BMJ formed a working party with the aim of improving the quality of submitted and published economic articles by clarifying acceptable methods and their systematic application throughout the peer review process. The guidelines (published in August 1996) provided referees and editors with checklists discussing how manuscripts should be approached and algorithms of the editorial process with an explanatory text.¹² We are reporting the results of our study to assess whether publication of the BMJ guidelines made any difference to (1) the quality of submitted manuscripts, (2) the quality of published manuscripts, and (3) the editorial and peer review process.

All submissions with an economic content (those making explicit comments about resource allocation and/or costs of interventions) made during the periods July 1 to September 30, 1994, to BMJ and October 1 to December 31, 1995, to BMJ and The Lancet were included in a "before" phase of the study. Submissions to BMJ in the period July 1 and July 31, 1997, and to The Lancet from April 1 to July 1, 1997, were included in the "after" phase of the study. Study periods were chosen on the basis of high manuscript volume. The Lancet was chosen as a control journal because of its similar scope and size to BMJ and because no concerted effort was made to promulgate the guidelines.

We recorded the editorial fate of submissions (eg, rejected without external review, rejected after review, or published) and undertook an unblinded assessment of the quality of submitted and published economic evaluations using the referees' checklist included in the guidelines. This checklist contains 35 items important for reporting the results of economic evaluations.¹² For each item the 4 possible outcomes were (1) yes, the authors dealt with the issue; (2) no, they did not; (3) not clear; and (4) the issue was not applicable to that particular submission. Therefore, the sum of the "no" and "not clear" columns gives an indication of the extent issues were not dealt with. A score of 35 would represent a paper failing on all the items.

To answer the second study question, we sent questionnaires to editorial staff of both journals. This contained questions about whether editorial staff were aware of the guidelines, had used the editors' or referees' checklists, and had found the guidelines useful in the editorial process.

Submitted manuscripts were classified as economic evaluations (EEs), in which 2 or more alternatives were compared with their costs and consequences, and other economic submissions (OESs), including partially analytical, descriptive, or methodological papers. The full classification has been described elsewhere.¹³ In the "before" period, 1642 and 1340 manuscripts were submitted to BMJ and The Lancet, respectively (433 and 1644 for the "after" period). Most manuscripts were traced and assessed to determine whether they constituted economic submissions. However, a number of manuscripts could not be traced for different reasons, including incorrect sequential submission numbering, editorial delay, and withdrawals.

Figure 1 outlines the number of submissions to both journals in the "before" and "after" periods and their editorial fate. Full EEs constitute 23% of all economic submissions. Although more were submitted to The Lancet, fewer were sent out for external peer review and none were published. The main change is in the treatment of OESs by BMJ. In the "before" period, 52 submissions were received, 20 (38%) were sent for peer review, and 2 were accepted. In the "after" period, only 5 (13%) of 38 were sent for peer review, and 1 was accepted, suggesting that the guidelines may have improved the efficiency of peer review with no discernible impact on the editorial fate of EEs in both journals.

The overall results of the questionnaire (Table 1) show that the guidelines were widely promulgated among editors of BMJ and were perceived to be useful. Few of The Lancet editors were aware of the guidelines. In BMJ the usefulness of the guidelines seemed to mostly help the editors with the internal peer review process, as there was little evidence that editors had sent the referees' checklists to those refereeing papers.

The guidelines had no apparent impact on the quality of the 43 EEs submitted to the journals (Table 2). A conclusion on whether the guidelines have contributed to any improvements observed between submission and final publication cannot be drawn yet due to small numbers. The most frequent defects in the economic evaluations submitted were (1) failure to state the research question (36 studies or 84%); (2) failure to justify the choice of form of economic evaluation in relation to the question being asked (40 studies or 93%); (3) failure to give details of price adjustments for inflation or currency conversion (34 studies or 79%); (4) failure to justify the choice of discount rate (36 studies or 84%); and (5) failure to state the approach to sensitivity analysis (42 studies or 98%).

Because this may be the first attempt to evaluate the impact of economic methodological guidelines, the study had several important shortcomings. We collected few manuscripts relating to the "after" period from BMJ, as manuscripts submitted prior to June 1997 had been shredded and manuscripts submitted from August 1997 onward were still undergoing editorial assessment. Missing manuscripts and exclusive focus on management of economic submissions in only 2 journals may introduce a bias into the study results, hence the absence of statistical analysis of the data.

There was some evidence that the editorial process had improved at BMJ. Editors found the guidelines helpful and potentially wasteful peer review of OEs was reduced. In the "before" period, BMJ put 20 manuscripts (38%) in this category through to peer review but published only 2. In the "after" period, only 5 (13.1%) were sent for peer review and 1 paper was published, suggesting a greater in-house confidence in dealing with submissions. In The Lancet, where knowledge of the guidelines was limited, 35% of OEs were sent for peer review in the "before" period, rising to 45% in the "after" period, with 4 and 5 papers being published, respectively.

There is no evidence that the guidelines had an impact on improving the quality of submitted and published articles. Application of the referees' checklist confirmed that the quality of the economic evaluation literature is disappointing. However, there was no change in the quality scores "before" and "after" probably because, although BMJ editors were made aware of the guidelines, they were not asked to distribute the guidelines or referees' checklist when sending papers out for peer review. Only a minority of referees used the checklist as a basis for their review, and authors took little notice of the guidelines when preparing their manuscripts, possibly because of ignorance, disagreement, or the impression that BMJ was unlikely to enforce them. This suggests that a more sustained educational effort is required to change authors' and editors' practice. A more rigorous prospective study of the impact of the guidelines could be conducted, whereby editors and/or referees could be randomly assigned to use the checklists.