Pharmacoeconomic Analyses: Title and subTitle BreakMaking Them Transparent, Making Them Credible

The issue of prescription drug benefits is being hotly debated in the United States. We can be sure that cost-conscious insurance organizations, both governmental and commercial, will demand an ever stronger role in deciding which drugs they pay for and include in managed care formularies. Pharmaceutical companies can be expected to continue to fund analyses of the cost-effectiveness of their products, and, as legal and political maneuvering in the United Kingdom, Canada, and Australia has shown, to continue to bring great political and legal pressure on the organizations responsible for deciding the relative merits of their products.^{1 - 3} Much of the battle will revolve around measurements of the cost-effectiveness of new drugs.

Recently, Friedberg et al⁴ found that studies funded by pharmaceutical companies were one eighth as likely to reach unfavorable qualitative conclusions and 1.4 times as likely to reach favorable qualitative conclusions as nonprofit-funded studies. In response to Friedberg et al, Gagnon,⁵ writing for the Health Outcomes Committee of the Pharmaceutical Research and Manufacturers of America, has pointed out that commercial organizations naturally sponsored those activities that promised the best return. Any bias in the outcomes of commercially sponsored economic analyses was inevitable and due to the fact that published economic analyses concentrate only on those drugs remaining after the weeding-out process of development. These drugs would already have been found by internal studies to have both favorable clinical and positive economic findings.⁵ But this does not explain why a higher proportion of nonprofit-sponsored cost-effectiveness analyses of drugs already on the market are negative and why the evidence from other studies, cited below, confirms that the process is skewed.

Although the rules for cost-effectiveness analysis have been widely disseminated in the medical literature since 1977,⁶ Udvarhelyi et al⁷ in 1992, showed that these rules were widely flouted and the fundamental principles usually ignored. Hillman et al,⁸ based on their experience conducting 33 economic analyses for 15 pharmaceutical companies over 13 years, were the first to come to grips with the problems of bias posed by cost-effectiveness analyses funded by makers of the drugs being analyzed. They noted that pharmaceutical companies sponsored most such research, which was unregulated, unstandardized, and prone to subjective bias. The researcher often had the design of the analysis, choice of the comparison drug, and selection of what confidential proprietary data they were allowed to see decided in advance by the sponsoring company. The companies, which considered such analyses to be marketing tools to promote sales, had studies supervised by marketing divisions that had no trouble with the idea of selecting ineffectual drugs for comparison, making investigators accept favorable assumptions, terminating unfavorable studies early, blocking publication, and putting pressure on investigators to produce favorable results.

Even if they were done well, the quality of cost-effectiveness analyses can only be as good as that of the trials on which they are based. Bero and Rennie⁹ have listed the many influences that can distort drug studies, influences that operate at every stage from design to publication. O'Brien¹⁰ noted that although randomized controlled trials (RCTs) are considered the highest level of evidence, they are designed to measure efficacy in a deliberately restricted population. Thus, their results may be of limited applicability and their use in assessing effectiveness in the real world of practice, which is what should most concern the pharmacoeconomic analyst, may be problematic. O'Brien¹⁰ summarized other problems that may occur when pharmacoeconomic studies use data from RCTs. Placebo-controlled trials may not apply, and there may be no head-to-head comparison of relevant drugs; the gold standard of outcomes for measuring results of therapy in the RCT may be impractical outside the trial; short-term, surrogate measures may be used and their relation to final outcomes may be obscure; sample sizes may be too small and follow-up inadequate; costs and outcomes may not reflect real life; and the results may not be transferable. Simply tacking on an economic analysis to a premarketing RCT may be inadequate for competent economic studies.

In short, such pharmacoeconomic analyses were likely to be poorly done and to be based on studies that, even if done well, were likely to be inappropriate for economic analysis. They were usually undertaken for commercial reasons and often biased by commercial influence.

To improve standards, task forces have published guidelines on the reporting of economic analyses,^{11 - 12} the first of which also addresses the issues of researcher independence and disclosure of conflicts of interest.¹¹ But Evans¹³ has poured scorn on such efforts, calling pharmacoeconomics a "pseudodiscipline . . . conjured into existence by the magic of money." He further observes: "There are a lot of drugs, and there is a lot of money, so the `field' is booming." Editors, who look at manuscripts that are often barely concealed advertisements for drugs, would tend to agree with this harsh assessment. Task force recommendations boil down, in Evans' view, to doing good and refraining from evil, but they cannot get around the basic structural problem that both sponsor and researcher have strong economic interests in the same outcomes.

In this issue of THE JOURNAL, a report by Hill and colleagues¹⁴ from the University of Newcastle, New South Wales, Australia, and the Pharmaceutical Evaluation Section of the Australian Pharmaceutical Benefits Scheme, adds to this gloomy picture. Under this scheme, new drugs are listed, and so paid for by the government, only when their pharmacoeconomic analyses indicate they are cost-effective in some patients. Such analyses are supposed to relate "any improved health outcomes with new drugs (compared with established treatments) to the net costs associated with their use."¹⁴ These authors, provided by law with all available information, describe all sorts of difficulties in their evaluation and interpretation of 326 submissions between 1994 and 1997. Of these submissions, 218 (67%) were considered to present "serious problems of interpretation." Sometimes no RCTs were available, or the RCTs were of poor quality or low power; the analysis was defective or made ill-justified claims; models were defective; assumptions were unsubstantiated; and calculations of costs and outcomes were not transparent.

Given all this, is it surprising that journal editors, concerned to diminish bias in what they publish, look with suspicion at such analyses? Bias in reporting cost-effectiveness analyses is prevalent and related to sponsorship, so the questions are What is causing this? and What can we do to prevent it? One way for journals to address the problem would be to give studies directly funded by pharmaceutical companies extra scrutiny or refuse to publish them at all. But bias need not lie on one side only. Governments or health plans that pay for pharmaceuticals may have a financial interest in not approving new drugs. Rather than assume that potential bias results in analytic bias, we prefer to improve the standards of research and its reporting.

Like them or not, there is no real alternative to cost-effectiveness analyses, and indeed, Hill et al¹⁴ in the end give them qualified support. What is needed are creative ways of improving the design, conduct, and reporting of the trials on which they are based as well as the analyses themselves. Legislation to increase US Food and Drug Administration oversight to standardize methods and reduce bias might be useful,^{8 ,15} but many analyses are undertaken after a drug has been approved. We would put the onus squarely on researchers, their institutions, and the journals that publish their results.

What can editors do to help the process? For the trials underlying the economic analyses, they can insist that a requirement for publication be that the trial has been registered at inception to reduce publication bias¹⁶ and require use of the CONSORT system for reporting RCTs¹⁷ and the QUOROM recommendations on the reporting of meta-analyses.¹⁸

For cost-effectiveness analyses, however, the problem is more complex. Since retrospective looks allow pharmaceutical companies to select the trials most likely to support favorable conclusions and since even the results of RCTs are often inadequate, editors should favor prospective, naturalistic, real-world RCTs in which physicians and patients are freer to pursue their usual practices and in which the goals of pharmacoeconomic studies can be achieved.^{4 ,10}

Kassirer and Angell¹⁹ have pointed out that cost-effectiveness analyses share many of the features of a review article. Both allow numerous judgment calls by the analyst, all of which may be biased. The conduct of reviews of therapy has been revolutionized by the systematic approach advocated by Mulrow.²⁰ The editor and reader must now demand that the author of a review article formally describe the search methods used and the criteria for inclusion and exclusion of all the studies. By analogy, the same standards must be adopted by authors of economic reports concerning drugs, to prevent selective quotation, especially where, unlike a review, the readers may have no access to the sources. However, the key requirement for any cost-effectiveness report is that the assumptions, models, and possible biases are well described, transparent, and fully supported by evidence, the strength of which is made easily available to any critical reader. At the very least, one should be able, when reading a cost-effectiveness study, to know whether all the relevant clinical trials were used in the analysis, and, if not, why.

In addition, editors should require that authors disclose all commercial ties (employment, consultancies, paid travel, stock ownership, and the like) to the editors for disclosure to the readers. Sponsorship money must be directed to the institution and must not depend on initial results being favorable to the sponsor's product.⁶ Researchers do not have to cede their rights to design, carry out, and publish their research, and their institutions must support them in this.²¹ Journals can help this by insisting that authors reveal whether there were any restrictions on their access to material or their ability to publish.⁸ Rakatansky²² has even proposed that where a sponsor had the right to review a manuscript before submittal to a journal for publication, disclosure to the reader also should include information about the results of that review. Editors also should insist that those who performed all major functions such as design, be included among the co-authors.²³ Having said this, editors have to recognize that, as happened recently with the New England Journal of Medicine's admirably strict policies to avoid conflicts in their drug therapy series, journal policies can break down embarrassingly.^{24 - 25}

There is room for improvement in editorial assessment of clinical economics manuscripts,^{26 - 27} and readers must be more skeptical.²⁸ The problem is that when the reader has access to such a small and selected proportion of analyses, no amount of skepticism is helpful. As the study by Hill et al¹⁴ shows, decrees from journals that articles include all the details necessary for review of cost-effectiveness analyses—including the data from the trials cited, the assumptions and their justifications, the models and methods—are well-intentioned but unworkable. This is partly for reasons of space and partly because the study by Hill et al shows that paid experts, with standardized spreadsheets already made up, are unable to complete the review in less than 2 weeks full time. How, then, can journals possibly expect their reviewers to validate all the trial results and key assumptions and confirm the resource costs? We agree with Hill et al when they say, "We doubt whether any conventional peer-review process can be expected to identify" the problems they discovered.¹⁴ So what is the answer?

We believe that changes in electronic publication offer a way around the difficulty, although most journals, including JAMA, are not yet set up to accomplish the changes we suggest. Electronic submission of data provides an easy way to give a journal's prepublication peer reviewers the chance to see all the data from all the trials and to get into the workings of the cost-effectiveness analysis. Realistically, few reviewers will be able to take the time to examine the data thoroughly. So, after review, if the paper is accepted, all these data and assumptions could be placed on a journal's Web site so that readers can examine the primary data at their leisure and, by the process of postpublication peer review, draw attention to problems they detect.

By furnishing on Web site data banks all the necessary information to reproduce the study findings, journals will give readers the true transparency that we all require and that is the only way by which cost-effectiveness analyses can become credible. Because groups with contrary financial interests, such as competitors and payers, will have access to the data and the incentive to determine if there are crucial flaws, the original work is likely to be of high quality to avoid the embarrassment of retraction. In essence, our proposal would be similar to the discovery process in lawsuits, whereby each side has access to the underlying data that may be presented.

Under the Australian scheme, everything the evaluators see is completely confidential. There is some rationale for keeping confidential data for drugs that are not approved, but after approval for use (and all drugs seen by the Australian panel have already been licensed by a separate authority), the paying public would have a great deal more confidence in decisions made on the basis of publicly available information. So, too, in cost-effectiveness studies, the equalization that ready access to the basic information provides would allow the reader to feel comfortable in the knowledge that those with the greatest interest in critiquing the studies had all the material to do so.

We are not arguing that all pharmacoeconomic analyses be done the same way. We are not arguing that the results of such sponsored studies reflect nothing more than commercial spin. We are saying that methodological choices determine the output, the results, and we need to be able to examine those choices by setting up a much more open process. Is it possible to publish credible cost-effectiveness analyses sponsored by drug companies? We'll see, but we will see only if we can all see the data.