A Comparison of the Opinions of Experts and Readers as to What Topics a General Medical Journal (JAMA ) Should Address FREE

SCIENTISTS, physicians, and editors of their journals often confront issues of great importance to their profession and the people it serves. Many of these problems can be studied by using scientific methods, posing questions, creating hypotheses, and testing such by scientific study. But when the question is one for which no apparent scientific study can be devised to answer, and when decisions are necessary for daily work, the people involved often seek opinion to guide them. Opinion can be from 1 individual, a few individuals, or a group.¹

Editors of medical and science journals must decide what articles to publish. They usually simply respond to what is sent to them by authors, using peer review and editorial judgment, and publishing the best of what they receive. We at JAMA decided many years ago that, although this reactive approach has worked well for many medical journals, we wanted to be proactive as well.

Since 1990 we at JAMA and the 10 American Medical Association (AMA) Archives (specialty) Journals have used a multipass, modified Delphi voting approach to ascertain the subjects of greatest importance for JAMA to deal with in the next year.^2- 3 In addition, we have consulted the editors of the Archives Journals to determine if there is 1 subject near the top of each year's list that all 11 journals should publish about concurrently with theme issues during that year. The purposes of these theme issues and the resulting annual publication topics have been reported elsewhere.⁴

The persons voting in the annual Delphi process were senior editorial staff of JAMA and JAMA Editorial Board members, which include the chief editors of the 10 AMA Archives Journals. All are considered experts in their fields and represent a wide range of areas of interest. This entire process is seen by us as a great success and has spawned the notion of global theme issues, which have now occurred twice and are highly regarded by editors and readers worldwide.^5- 6

In 1996 we decided that we should test the ability of these expert opinions to predict what ordinary readers might wish to read about, by assessing the level of congruence of subject interest among these groups. Thus, we performed our usual criterion standard design survey using the Delphi process involving 55 recognized experts who were asked to propose the topics most important for JAMA to deal with in 1997. All voters were informed of the process, consented to participate, and were provided the results.

A total of 40 (73%) of the 55 experts surveyed responded with 178 ideas for topics. Editing for crossovers and groupings left 73 topics. The same 55 persons were asked to stratify all 73 topics, which had been alphabetically arranged, on a scale of 1 to 5. The response rate was 85% (47/55). The same 55 persons were given the results of this Delphi pass and asked to vote again. The response rate was 76% (42/55). After all results had been provided to 40 attendees (of the original 55) at the regular 1996 JAMA Editorial Board meeting, the topics were discussed and debated. Thirty-nine voted on the final ballot—a response rate of 98% of those present.

To assess the topic interest of JAMA readers, in late 1996 and early 1997 a direct mail survey from an independent (not AMA) company was sent to a stratified sample of 500 practicing physicians (not basically academic) who are regular JAMA readers. A random draw from the AMA Masterfile of confirmed JAMA readers was taken to compile a list for survey. All who participated were informed of the process in the cover letter and by virtue of responding consented. They were not called in advance. They were not provided results.

A response rate of 41.6% (208 readers) was realized for the single-pass ballot of the same 73 topics. Nonresponders were roughly equivalent in available parameters to the responders (age, reading habits, specialties, sex, type of practice, number of patients seen per week). We compared and contrasted the results across all 73 topics.

The final top 10 topics determined by the experts were (in order) managed care, death and dying, genetics, quality of care, violence, aging, caring for the uninsured and underinsured, outcomes research, human immunodeficiency virus/acquired immunodeficiency virus, and cancer. Readers agreed with the experts on only 3 of the top 10 subjects: managed care, cancer, and aging (Table 1).

The data are substantial in amount and detail and are reported here only as the full comparison of the 2 total groups for the 73 topics, since this is considered a pilot project.

It is clear that according to the methods described here, there is substantial variation between expert opinion and the opinion of readers as to what JAMA should emphasize. We recognize weaknessess in this pilot study. For example, the methods for the 2 groups—experts and readers—were different; one was a single-pass mail survey, while the other was a 4-pass, sequential modified Delphi. The sample sizes were small: 55 in the expert group and 500 in the reader group. Response rates were less than ideal, from 73% to 98% in the expert group but only 46% in the reader group. But the differences in results were startling and we believe important. For example, the experts ranked alternative medicine 68th of 73 topics JAMA should publish about, while the readers ranked it seventh of 73. Similarly, the experts ranked tobacco 17th of 73, while the readers ranked tobacco 55th of 73. We were so impressed that we have altered our manuscript acceptance process to become even more reader-friendly.

Among the potential explanations for the marked differences is the viewpoint: "Where one stands often depends upon where one sits." Perhaps the experts told us what they thought physicians needed to know, but also the experts tended to know what research was being performed and in what subjects the journal was likely to get good papers. The readers, however, mostly practicing physicians, perceived what they needed to know, and may not know, in order to care best for the patients walking into their offices that day. They also want journals to be useful for their practices.

Editors of primary-source, peer-reviewed medical journals have trust relationships with many publics. Key among these are readers, authors, and owners. Having hard data on which to make publishing choices can be useful, since the principal interests of these diverse publics frequently come into conflict.

It is the editor's job to keep all of these publics happy at all times. That, of course, is impossible. We should strive to serve the perceived needs of all these publics but recognize that we may not be able to serve all their wants as well. We have embarked on substantial new research aimed at balancing these inputs in a more positive way.