0
We're unable to sign you in at this time. Please try again in a few minutes.
Retry
We were able to sign you in, but your subscription(s) could not be found. Please try again in a few minutes.
Retry
There may be a problem with your account. Please contact the AMA Service Center to resolve this issue.
Contact the AMA Service Center:
Telephone: 1 (800) 262-2350 or 1 (312) 670-7827  *   Email: subscriptions@jamanetwork.com
Error Message ......
Review |

Rating Health Information on the Internet:  Navigating to Knowledge or to Babel? FREE

Alejandro R. Jadad, MD, DPhil; Anna Gagliardi, MSc, MLS
[+] Author Affiliations

From the Health Information Research Unit (Dr Jadad and Ms Gagliardi), Canadian Cochrane Network and Centre (Dr Jadad), Supportive Cancer Care Research Unit (Dr Jadad), and the McMaster Evidence-Based Practice Center (Dr Jadad), Department of Clinical Epidemiology and Biostatistics, McMaster University, Hamilton, Ontario.


JAMA. 1998;279(8):611-614. doi:10.1001/jama.279.8.611.
Text Size: A A A
Published online

Context.— The rapid growth of the Internet has triggered an information revolution of unprecedented magnitude. Despite its obvious benefits, the increase in the availability of information could also result in many potentially harmful effects on both consumers and health professionals who do not use it appropriately.

Objectives.— To identify instruments used to rate Web sites providing health information on the Internet, rate criteria used by them, establish the degree of validation of the instruments, and provide future directions for research in this area.

Data Sources.— MEDLINE (1966-1997), CINHAL (1982-1997), HEALTH (1975-1997), Information Science Abstracts (1966 to September 1995), Library and Information Science Abstracts (1969-1995), and Library Literature (1984-1996); the search engines Lycos, Excite, Open Text, Yahoo, HotBot, Infoseek, and Magellan; Internet discussion lists; meeting proceedings; multiple Web pages; and reference lists.

Instrument Selection.— Instruments used at least once to rate the quality of Web sites providing health information with their rating criteria available on the Internet.

Data Extraction.— The name of the developing organization, Internet address, rating criteria, information on the development of the instrument, number and background of people generating the assessments, and data on the validity and reliability of the measurements.

Data Synthesis.— A total of 47 rating instruments were identified. Fourteen provided a description of the criteria used to produce the ratings, and 5 of these provided instructions for their use. None of the instruments identified provided information on the interobserver reliability and construct validity of the measurements.

Conclusions.— Many incompletely developed instruments to evaluate health information exist on the Internet. It is unclear, however, whether they should exist in the first place, whether they measure what they claim to measure, or whether they lead to more good than harm.

THE INTERNET has given rise to an information revolution of unprecedented magnitude. Using the Internet, increasing numbers of health care providers and consumers gain free access to an expanding volume of information that previously was inaccessible.

Seeking useful and valid information on the Internet can be difficult because of the speed and lack of control with which the information is accumulating. Filtering through information on the Internet may also be very time-consuming. Tools such as Internet directories, indexes, and search engines assist health care providers and consumers in their search for health information, particularly on the World Wide Web. However, searching for and locating information are only starting points, after which the Internet users themselves must choose appropriate resources to guide their decisions.1

Judging whether the information is applicable and credible may present a greater challenge than just searching for information. To make this process more time efficient, Internet users may rely on a number of Internet resources that review and rate Web sites that provide health information. Theoretically, by relying on these ratings, users could more easily identify valuable information on the Internet.2,3 However, if the instruments used to produce the ratings are flawed (eg, if they are produced to sell specific products or if they do not have any discriminative power), they may mislead or misinform health care providers or consumers.

The objectives of this review are to identify rating instruments that evaluate Web sites providing health information, to describe their criteria, to establish the degree of validation of these rating instruments, and to recommend future directions for research in this area. In this unique study, the Internet was both the focus of a systematic review and the main source of data.

Inclusion and Exclusion Criteria

Rating instruments were included for review if there was a description of the rating criteria available on a Web site and if it had been used at least once to evaluate Web sites providing health information. We focused on instruments whose evaluations had been used to produce awards, quality ratings, or seals of approval; to feature sites such as "best of the Web" or "best" in a given category; or to declare a site as meeting quality standards or receiving top ratings.

Instruments were excluded if they were used to rate only the quality of Web sites in areas other than health or did not provide rating criteria.

Search Strategy

Rating instruments were identified using the following strategies:

  • A search to November 31, 1997, of MEDLINE (from 1966), CINHAL (from 1982), and HEALTH (from 1975) with the terms Internet or World Wide Web;

  • A search of the databases Information Science Abstracts (1966 to September 1995), Library and Information Science Abstracts (1969 to fall 1995), and Library Literature (1984 to March 1996) with the strategy rate or rating or rank or ranking or top or best and Internet or Web or site and healthcare or health care or information;

  • A search to November 1997 using the search engines Lycos (http://www.lycos.com), Excite (http://www.excite.com), Open Text (http://www.opentext.com), Yahoo (http://www.yahoo.com), HotBot (http://www.hotbot.com), Infoseek (http://www.infoseek.com), and Magellan (http://www.magellan.com) with the strategy rate or rating or rank or ranking or top or best and health;

  • A review of messages posted to the Internet discussion list "Public Communication of Science and Technology" (PCST-L@cornell.edu) from 1996 to November 1997;

  • A review of messages about rating instruments posted to the Medical Library Association listserv medlib-l (http://mlanet.org/discussion/medlibl.html) and the Canadian Health Libraries listserv canmedlib-l (http://www.mun.ca/lists/canmedlib/);

  • A search of the American Medical Informatics Association's 1997 Spring Congress table of contents (http://www.amia.org/search/fsearch.html) for mention of Internet rating instruments;

    A hand search of the journal Internet World (1995 to December 1997);

  • A connection to all relevant links provided by sites identified with any of the above methods; and

  • A search of all the references of any relevant printed articles.

We stopped searching for rating instruments on November 21, 1997.

Data Extraction

The Web site, group, or organization that developed each of the eligible instruments was identified in all cases, and information was extracted from the Web site by both of us by consensus. We extracted the following data:

  • General characteristics of the instruments: name of the rating instrument, the Internet address where the instrument was described, and the scope of the instrument (health specific or generic);

  • Information about the extent to which each instrument had been validated. This was based on criteria used previously to evaluate instruments to assess the quality of randomized controlled trials.4 These criteria included the number of items in the instruments, the availability of rating instructions, information on the development of the rating criteria (eg, the process for item generation and reduction and assessments of construct validity), and evaluation of interobserver reliability; and

  • Specific criteria to evaluate Internet sites. This was achieved by establishing whether the instruments considered authorship, attribution, and disclosure as part of their rating criteria. These criteria were selected because of the objectivity with which they could be rated and because they were the 3 common criteria proposed in recent articles published in the journals of the American Medical Association and the British Medical Association.2,3 We decided that an instrument considered authorship if it mentioned or requested, in any way, information on the authors, their contributions, affiliations, and relevant credentials; attribution if the instruments mentioned that they required, in any way, listing of references and sources for all content or information provided; and disclosure if they required, in any way, a description of Web site ownership, sponsorship, advertising, underwriting, commercial funding arrangements or support, or potential conflicts of interest. These definitions are less stringent variations of the original descriptions.2

Our search yielded 47 different rating instruments used to evaluate Web sites with health information on the Internet. Thirteen of the instruments published their rating criteria. One additional instrument included in our evaluation did not provide criteria on their Web site but invited interested people to contact them for additional information. We obtained the criteria from the editors of this site (OncoLink's Editors' Choice Awards), who informed us that the criteria are currently undergoing revision. A summary of the characteristics of the 14 instruments is presented in Table 1. A more detailed description of the rating criteria for each of the 14 instruments is available at http://hiru.mcmaster.ca/ebm/rating/table_3.htm.

Table Graphic Jump LocationTable 1.—Summary of Criteria for Rating Instruments*

Rating criteria were not available for 33 of the 47 instruments and were, therefore, not included in the review (Table 2). During the last 6-month period of our search, 9 instruments whose criteria were previously available ceased to make them available, and 3 groups decided to stop rating Web sites providing health information (Table 2). In addition, we know of 1 group that is developing a new instrument (http://hitiweb.mitretek.org/docs/criteria.html).

Table Graphic Jump LocationTable 2.—Rating Insturments Not Included for Review

Of the 14 instruments with criteria available, 8 appeared to be designed specifically to rate health information. Eight of the instruments are presented as scales, 4 of which use graphic analog scales (eg, stars, apples, medals, or thumbs) and 4 use numerical scales. In 6 cases, the type of instrument was unclear. Five of the instruments provided instructions for their use (Argus Clearinghouse Seal of Approval, Health on the Net Foundation Code of Conduct, Physician's Choice, Six Senses Seal of Approval, and Sympatico Health); 2 included authorship, attribution, and disclosure as criteria (American Medical Association Library Choices and Health on the Net Foundation Code of Conduct); 2 provided information on how the items had been generated (Physician's Choice and Six Senses Seal of Approval); and only 1 instrument stated that the evaluation of sites is conducted in duplicate and that the final rating is obtained by consensus (Six Senses Seal of Approval). There was no information available on formal evaluation of interobserver reliability or construct validity for any of the instruments included in our review.

The potential benefits of the use of the Internet are obvious. For health professionals, it can be a valuable clinical tool, another medium by which information can be exchanged with colleagues and patients, and a constantly growing source of new information.5 The Internet can also be a valuable source of information for consumers.68 For patients and their relatives or close friends, particularly in cases of newly diagnosed life-threatening diseases, electronic mailing lists, online support groups, and Web sites devoted to their particular disease can provide valuable information and emotional support.811 The access to information provided by the Internet is likely to improve consumers' sense of control as well as their ability to participate actively in health care decisions, with potentially better psychological outcomes.810 Some have suggested that it could also improve consumers' skills for self-help and reduce the financial burden of their care.7

Given the rapid, uncontrolled growth of health information on the Internet and its potential influence in health care decisions, it could be argued that it is desirable to have user-friendly, valid, and useful tools to evaluate any relevant information available on the Internet. Others, however, could view any attempts to evaluate the information on the Internet as yet another attempt by the academic community or regulatory agencies to control the production of information on the Internet, threatening the new level of freedom of expression and communication that the Internet has generated. Formal scholarly discussion, with participation of people with as many divergent views as possible, is required to inform any existing controversy.

Even if desirable, however, the next question is: is it possible to evaluate the information on the Internet? The successful development of instruments to evaluate health information on the Internet is not an easy task. Evaluation of just the content, for instance, presents the same challenges faced by those evaluating the quality of randomized controlled trials published in paper-based journals, including the lack of a "gold standard" for quality and the controversy around its definition.12 In addition, the information available on the Internet is very different from that found in traditional journals. For instance, the information on the Internet is produced and exchanged by groups of people (eg, health professionals, consumers, vendors, etc), presented using multiple formats (eg, text, video, sound), modified at a fast and unpredictable rate, and linked within a highly elaborate and complex network of Internet sites. Because of the dynamic and complex nature of the Internet, it has been suggested that a comprehensive evaluation instrument should consider issues beyond the content itself, such as criteria related to the structure, functions, and likely impact of the Web sites that provide the content.3 Evaluating subjective, context-dependent criteria like these in a valid and consistent way, however, will be even harder than evaluating the content itself. Given the number of instruments already available, it could be concluded that it is in fact possible to evaluate health information on the Internet. However, given the incomplete development of the tools and the complexity, variability, and dynamism of the information on the Internet, it could also be argued that any attempt to develop instruments to evaluate such information has been and will be futile. We welcome more discussion on this issue and invite those who believe in the possibility of adequate evaluation of information on the Internet to provide rigorous empirical evidence to support their views.

Establishing whether it is desirable or even possible to evaluate health information on the Internet will not be sufficient to justify systematic and widespread rating of Web sites. Ideally, any such efforts should be supported, or at least complemented, by studies designed to show that the evaluations have a beneficial effect on health care decisions, health outcomes, or resource utilization. The design of studies to generate such evidence, however, will require highly innovative methodological strategies.

In summary, a large number of incompletely developed instruments to evaluate health information on the Internet exist. It is unclear, however, whether they should exist in the first place, whether they measure what they claim to measure, or whether they lead to more good than harm. We also need to acknowledge that users may never notice or, if they notice, they may choose to ignore any evidence in support of or against the desirability, feasibility, and benefits of formal evaluations of health information on the Internet, as may happen frequently with more traditional paper-based information.13,14

With the Internet and the rapid development of other information technology, we are facing a sea change in health care. The Internet is creating new opportunities to improve decisions and communication in health care, but it can also generate many unprecedented problems. In the midst of an unparalleled information revolution, good communication, scholarly discussions, and rigorous evaluations are more crucial than ever. If we fail to meet the current challenges, we may miss an extraordinary opportunity to make health care more efficient and equitable, moving instead into a health care environment ruled by confusion, battles of opinion, anxiety, and unnecessary conflicts.

Hayward RSA, Gagliardi A, Jadad AR. Healthcare on the Internet.  Health Measures.September 1997:28-36.
Silberg WM, Lundberg GD, Musacchio RA. Assessing, controlling, and assuring the quality of medical information on the Internet.  JAMA.1997;277:1244-1245.
Wyatt JC. Commentary: measuring quality and impact of the World Wide Web.  BMJ.1997;314:1879-1881.
Moher D, Jadad AR, Nichol G, Penman M, Tugwell P, Walsh S. Assessing the quality of randomized controlled trials: an annotated bibliography.  Control Clin Trials.1995;16:62-73.
Peters R, Sikorski R. Digital dialogue: sharing information and interests on the Internet.  JAMA.1997;277:1258-1260.
Widman L, Tong D. Requests for medical advice from patients and families to health care providers who publish on the World Wide Web.  Arch Intern Med.1997;157:209-212.
Bennett-Clark J. Where health information is byte-size: online advice can help you manage health costs.  Kiplinger's Personal Finance.June 1996:96-97.
Ferguson T. Consumer health informatics.  Health Forum J.January/February 1996:28-33.
Vaitones V. Protocol ‘shopping' on the Internet.  Cancer Practice.1995;3:274-278.
Fernsler J. Commentary to the case study ‘Protocol "shopping" on the Internet.'.  Cancer Practice.1995;3:277-278.
Degner L. Commentary to the case study ‘Protocol "shopping" on the Internet.'.  Cancer Practice.1995;3:275-276.
Moher D, Jadad AR, Tugwell P. Assessing the quality of clinical trials: current issues and future directions.  Int J Technol Assess Health Care.1996;12:195-208.
Bero LA, Jadad AR. How patients and policy makers can use systematic reviews for decision-making.  Ann Intern Med.1997;127:37-42.
Redelmeier DA, Rozin P, Kahneman D. Understanding patients' decisions: cognitive emotional perspectives.  JAMA.1993;270:72-76.

Figures

Tables

Table Graphic Jump LocationTable 1.—Summary of Criteria for Rating Instruments*
Table Graphic Jump LocationTable 2.—Rating Insturments Not Included for Review

References

Hayward RSA, Gagliardi A, Jadad AR. Healthcare on the Internet.  Health Measures.September 1997:28-36.
Silberg WM, Lundberg GD, Musacchio RA. Assessing, controlling, and assuring the quality of medical information on the Internet.  JAMA.1997;277:1244-1245.
Wyatt JC. Commentary: measuring quality and impact of the World Wide Web.  BMJ.1997;314:1879-1881.
Moher D, Jadad AR, Nichol G, Penman M, Tugwell P, Walsh S. Assessing the quality of randomized controlled trials: an annotated bibliography.  Control Clin Trials.1995;16:62-73.
Peters R, Sikorski R. Digital dialogue: sharing information and interests on the Internet.  JAMA.1997;277:1258-1260.
Widman L, Tong D. Requests for medical advice from patients and families to health care providers who publish on the World Wide Web.  Arch Intern Med.1997;157:209-212.
Bennett-Clark J. Where health information is byte-size: online advice can help you manage health costs.  Kiplinger's Personal Finance.June 1996:96-97.
Ferguson T. Consumer health informatics.  Health Forum J.January/February 1996:28-33.
Vaitones V. Protocol ‘shopping' on the Internet.  Cancer Practice.1995;3:274-278.
Fernsler J. Commentary to the case study ‘Protocol "shopping" on the Internet.'.  Cancer Practice.1995;3:277-278.
Degner L. Commentary to the case study ‘Protocol "shopping" on the Internet.'.  Cancer Practice.1995;3:275-276.
Moher D, Jadad AR, Tugwell P. Assessing the quality of clinical trials: current issues and future directions.  Int J Technol Assess Health Care.1996;12:195-208.
Bero LA, Jadad AR. How patients and policy makers can use systematic reviews for decision-making.  Ann Intern Med.1997;127:37-42.
Redelmeier DA, Rozin P, Kahneman D. Understanding patients' decisions: cognitive emotional perspectives.  JAMA.1993;270:72-76.

Letters

CME
Meets CME requirements for:
Browse CME for all U.S. States
Accreditation Information
The American Medical Association is accredited by the Accreditation Council for Continuing Medical Education to provide continuing medical education for physicians. The AMA designates this journal-based CME activity for a maximum of 1 AMA PRA Category 1 CreditTM per course. Physicians should claim only the credit commensurate with the extent of their participation in the activity. Physicians who complete the CME course and score at least 80% correct on the quiz are eligible for AMA PRA Category 1 CreditTM.
Note: You must get at least of the answers correct to pass this quiz.
You have not filled in all the answers to complete this quiz
The following questions were not answered:
Sorry, you have unsuccessfully completed this CME quiz with a score of
The following questions were not answered correctly:
Commitment to Change (optional):
Indicate what change(s) you will implement in your practice, if any, based on this CME course.
Your quiz results:
The filled radio buttons indicate your responses. The preferred responses are highlighted
For CME Course: A Proposed Model for Initial Assessment and Management of Acute Heart Failure Syndromes
Indicate what changes(s) you will implement in your practice, if any, based on this CME course.

Multimedia

Some tools below are only available to our subscribers or users with an online account.

Web of Science® Times Cited: 394

Related Content

Customize your page view by dragging & repositioning the boxes below.

Articles Related By Topic
Related Collections
PubMed Articles