Empirical Studies Assessing the Quality of Health Information for Consumers on the World Wide Web:  A Systematic Review FREE

The Internet has become an important mass medium for consumers seeking health information and health care services online.¹ A recent concern and public health issue has been the quality of health information on the World Wide Web. However, the scale of the problem and the "epidemiology" (distribution and determinants) of poor health information on the Web are still unclear, as is their impact on public health and the question of whether poor health information on the Web is a problem at all.² Many studies have been conducted to describe, critically appraise, and analyze consumer health information on the Web. These typically report proportions of inaccurate or imperfect information as estimates of the prevalence of flawed information or the risk of encountering misinformation on the Web.

However, to date no systematic and comprehensive synthesis of the methodology and evidence has been attempted. Two previous systematic reviews focused on compiling quality criteria and rating instruments, but did not synthesize evaluation results. Jadad and Gagliari³ reviewed non–research-based rating systems (eg, criteria used for "awards," "seals of approval," or gateways) published on the Internet but did not include rating systems published in the peer-reviewed literature. Kim et al⁴ compiled instruments or criteria proposed to evaluate Web sites, but it is unclear to what extent and with what results these criteria can be and have been applied in practice.

To our knowledge, this is the first systematic review conducted to compile criteria actually used and to synthesize evaluation results from studies containing quantitative data on structure and process measures of the quality of health information on the Web. The objectives of this study were to establish a methodological framework on how "quality" on the Web is evaluated in practice, to determine the heterogeneity of the results and conclusions, to compare the methodological rigor of these studies, to determine to what extent the conclusions depend on the methodology used, and to suggest future directions for research.

Inclusion and exclusion criteria were established in advance in a written protocol. Studies were included when their authors: (1) searched the World Wide Web systematically for health information (eg, to identify all consumer Web sites for a given topic) or clearly defined a set of specific health information services to be included; (2) evaluated information likely to be accessed by consumers; (3) evaluated the quality of health information or services against certain criteria—for example, by judging the authority of source; assessing the accuracy of information, readability, or comprehensiveness; or by comparing Internet information and services against those outside of the Internet; and (4) provided quantitative results, such as proportions of Web sites complying with the quality criteria, or the distribution of quality scores, or information on the readability level(s) in the sample.

Studies were excluded if authors simply listed "quality" Web sites without indicating that they had performed a comprehensive, systematic search, if they did not list quality criteria, or if they did not quantitate how many Web sites were assessed in total or how many did not comply with quality criteria. We also excluded studies assessing tools that enable or facilitate access to health information, such as search engines, directories, health portals, rating systems, and review services, or studies comparing different methods of information retrieval. Theoretical papers that only described quality criteria alone were also excluded, as were qualitative descriptions of Internet contents. We did not include studies that dealt with content other than health information, or studies evaluating information not intended for consumers (eg, those providing continuing medical education), but we did include studies evaluating services and information likely to be used by both professionals and consumers (eg, those providing drug information). Studies evaluating a single site or application were excluded, as were studies focusing exclusively on privacy and security issues, or on interactions via newsgroups or e-mail.

We sought all relevant studies and unpublished reports, regardless of language or peer-review/publication status. We searched MEDLINE and PREMEDLINE (1966-September 2001) by entering the following query into the PubMed interface on September 20, 2001 (quality OR reliability OR accuracy OR readability OR evaluation OR assessment) AND (information OR education OR advice) AND (internet OR web OR ehealth OR "e-health" OR cyber* OR www). We retrieved 1545 citations and their titles and abstracts were screened for potential relevance by 2 independent reviewers. In addition, 1 reviewer used analogous search terms to screen citations in LISA (Library and Information Science Abstracts Database, 1969-July 2001) (1269 hits), CINAHL Nursing and Allied Health (1982-July 2001) (2312 hits), PsychINFO (American Psychological Association, 1988-September 2001 [week 2]) (321 hits), EMBASE (Elsevier Science BV, Amsterdam/NL, 1988-June 2001) (647 hits), and SIGLE (Grey Literature in Europe database, 1980-June 2001) (83 hits). The Web of Science database (Institute for Scientific Information, Philadelphia, Pa), which includes the Science Citation Index (1997-September 2001), the Social Sciences Citation Index (1997-September 2001), and the Arts and Humanities Citation Index (1997-September 2001) was searched using both a traditional backward search strategy (1120 hits) as well as using a forward citation-search strategy, using 5 seminal papers in this field^3- 7 as "seed publications" to identify all publications that subsequently cited 1 of these papers (533 hits). We also checked the references of all identified studies, hand-searched the Journal of Medical Internet Research (because it contains pertinent studies and because at that time it was not yet indexed on MEDLINE), conducted general Internet searches using combinations of the search terms at the Google and Northern Light search engines, reviewed 3 bibliographies of consumer health information studies collated by academic institutions, and used private literature databases.

In total, we screened 7830 citations and retrieved 165 potentially eligible full articles to determine whether they met the inclusion criteria. Articles in Hungarian, Japanese, French, Spanish, Dutch, and Italian were translated by professional translators into English or German.

Two reviewers (G.E., J.P.) independently extracted study characteristics using electronic data extraction forms, maintaining all data in an Access database (Microsoft Corp, Redmond, Wash). Each extraction form contained 85 items, of which 43 were closed questions, mostly to be answered on a scale (yes/no/partially/not applicable/to be discussed). We extracted information on search strategies used, quality criteria applied, and methodology of quality assessment used by authors, as well as quality and rigor of study methodology and reporting (see below and Table 1).

We rated the overall tone of the authors' conclusions as "negative" if authors were wary and pessimistic about the Web as a source for health information, "positive" if authors did not express concern or recommended the Web as a source for health information, or "neutral" if authors discussed both risks and benefits, or if the reviewers disagreed.

To extract evaluation results of each study, one reviewer tabulated data on how many Web sites or Web pages were evaluated for each quality criterion and how many of those evaluated did not comply with the respective criterion.

To determine interobserver reliability for all coded items, we calculated standard κ values for each extracted item reported on a binary scale (eg, include study: yes/no), or weighted κ values for ordinal outcomes (eg, study characteristics graded on a scale [yes/partially/no]). We decided in advance not to include items with κ values less than 0.6.⁸ For the final consensus coding used for analysis, reviewers examined all disagreements or items coded "to be discussed" and resolved them by extensive discussion.

We used meta-analysis of proportions⁹ to test for homogeneity of results for quality criteria that have been used by 3 or more authors to assess Web sites. We pooled study results for a given quality criterion if the assumption of homogeneity could not be rejected on a significance level of less than .05.

To test whether the different conclusions of studies could be explained by their different quality scores we compared study assessment scores of study groups using the Wilcoxon test and logistic regression. All calculations were performed using SAS versions 6.12 and 8.2 (SAS Institute Inc, Cary, NC).

We summarized the quality of reporting and the rigor of the studies by checking whether authors reported certain items deemed important for a well-reported, systematic, and comprehensive Internet search strategy (ie, S-score for search quality; range, 0-8 [transformed to percentage score]), and items thought to indicate a rigorous evaluation (ie, E-score for evaluation quality; [range, 0-10 [transformed to percentage score]). To ascertain face validity, an initial list of candidate items was compiled by 2 reviewers (G.E., J.P.) independently and later combined by consensus. As studies systematically evaluating Web information resemble in some ways systematic research overviews, the candidate criteria for the S-score partly derived from corresponding checklists.¹⁰ To ascertain score reliability, we eliminated items for which κ was less than 0.6; the remaining items are shown in Table 1. One point was given per item reported in order to calculate a raw summary score for search quality and evaluation quality. The final reported S-score and E-score in Online Table A (available in PDF format) represent the percentages of the maximum score achievable for each study (because some items are not applicable for some studies).

To test construct validity we compared the scores of 5 letters to the editor with the scores of the 74 full publications. As expected, the letters had, on average, lower S-scores (28.8 vs 44.2, P = .06), but similar E-scores (22.4 vs 33.5, P = .45). The 7 full articles published in peer-reviewed non–English-language journals had lower average E-scores than articles published in English-language journals (mean E-score: 18.6 vs 34.8, P = .09), while their S-scores were similar (mean S-score: 42.3 vs 44.5, P = .85). While these differences are not significant at a 5% level, perhaps due to the small study number, the trend goes in the expected direction.

Of 170 articles reviewed, 85 met the inclusion criteria^{11- 5960- 95} (Online Table A available in PDF format). These constitute 79 distinct studies, as 6 studies were published either in duplicate,^{26- 27,50- 51,92- 93} in a fragmented manner,^87- 88 or were reported both in a peer-reviewed publication and in an unpublished form.^{14- 15,68- 69} In these cases we pooled the results from the different articles and referred to these as a single study. Table 1 summarizes and Online Table A (available in PDF format) lists, all included studies.

Technical Criteria (T). "Technical" quality criteria were defined as general, domain-independent criteria, ie, criteria referring to the question of how the information was presented or what meta-information was provided. The 24 technical quality criteria most frequently used (Table 2) are variations of what could be called "transparency criteria" from the print world, mentioned by Silberg et al⁷: authorship, attribution, disclosure, and currency. While the latter is strictly speaking a dimension of accuracy, almost all studies sought for provision of a date of creation or last update (rather than actual currency of the content), which is a technical criterion.

Design (D). Fifteen studies also evaluated subjective design features such as the visual aspect of the site or layout, but only a few studies reported results (Online Table B available in PDF format), presumably because of their subjectivity and low reliability. For example, Gillois et al³⁹ used a visual analog scale to assess the quality of visual presentation and of the interface, but noted high interobserver variability. Stausberg et al⁸¹ report a κ of 0.08 when evaluating navigation, and 0.23 when evaluating layout. Speed, browser compatibility, and presence of a search engine were considered technical criteria.

Readability (R). Eleven studies used readability formulas to establish the reading level of a document based on the complexity and length of words and sentences (Online Table B available in PDF format). Nine studies used the Flesch-Kincaid (FK) Grade Level Index. Other formulas used include the SMOG Readability Formula, the Fry Readability Graph, the Gunning-Fog formula, and the Lexile Framework. As different formulas yield different reading levels for the same document, and as authors use different cutoffs, the results cannot be pooled, but studies suggest that reading levels are frequently too high (Online Table B available in PDF format).⁶⁵

Using reading formulas has limitations, as readability scores do not reflect other factors that affect comprehension such as frequency and explanation of medical jargon, writing style (use of active voice, nonpatronizing language, motivational messages, tone/mood, how it relates to the audience), or use of culturally specific information. Few studies analyzed these important but subjective aspects—Oermann and Wilson⁶⁶ discussed some of these parameters, Fitzmaurice and Adams³⁵ scored the writing style, and Wilson et al⁹⁴ examined the cultural sensitivity of the documents to various ethnic groups using Bloch's Ethnic/Cultural Assessment Tool. None of the studies conducted comprehension tests with actual consumers or used judgments of literacy experts.

Accuracy (A). Accuracy can be defined as the degree of concordance of the information provided with the best evidence or with generally accepted medical practice. Alternative terms used for the concept of "accuracy" include "reliability"⁵¹ or "conventionality of information."⁷⁹ We coded 2 studies as assessing accuracy "partially," because authors claimed to have evaluated "value and amount of text" without further specifying how they define "value."^36,90

Authors who appraised the accuracy and/or comprehensiveness could either define clear criteria beforehand (a priori) or extract information from the Web first and then check these claims against the literature (a posteriori). Thirty studies assessed accuracy a posteriori, and only 19 a priori (Online Table A available in PDF format). Studies defining elements a priori varied considerably in the granularity and specificity of the items defined in advance: for example, 1 study stated that "paracetamol should be given in a dose of 10 to 15 mg/kg every 4 hours,"⁵¹ while others just predefined broad keywords that should be mentioned, eg, "oestrogen."⁷⁵

Accuracy ideally should be defined using the best available evidence.⁵ Fourteen studies used evidence-based guidelines or systematic reviews to define elements a priori and 1 study used a guideline as the criterion standard a posteriori. Eleven studies used primary literature for a posteriori comparison; however, with 1 exception⁸⁹ these studies did not provide details on how and what literature was screened, leaving open the possibility that some authors did not actually search the literature, but just compared information against their own knowledge of the literature. Textbooks or expert consensus were used as the criterion standard in 3 a priori and in 3 a posteriori studies.^{29,54- 55,64,70,84} The remaining studies used unclear sources or the personal opinion of the author as the a priori (2 studies) or a posteriori (15 studies) criterion standard.

Completeness/Comprehensiveness/Coverage/Scope (C). Several methods were used to evaluate "completeness," "comprehensiveness," "coverage," or "scope." Most authors^{12- 15,22,29,35,37,42,44- 46,50- 51,54- 56,58- 59,70,75,78- 79,85,92- 93,95} calculated a proportion of a priori–defined elements covered by a Web site or reported the proportion of Web sites that mentioned all key elements (eg, from a clinical guideline). Willems and Bouvy^92- 93 used a 5-point scale to evaluate completeness. One study¹¹ evaluated "balance," eg, whether the adverse effects and contraindications as well as the advantages of a drug are presented, but detailed subcriteria were not made explicit. The Soot score^13,46,58,79 addresses completeness by measuring coverage of topic areas defined a priori, weighting some areas as being more important than others.

While 19 studies evaluated completeness as a distinct entity, 21 other studies evaluated completeness as an integral part of accuracy (marked "AC" in Online Table A available in PDF format). For example, Impicciatore et al⁵¹ reported that only 4 of 44 Web sites "adhered closely to the guidelines," implying that these sites were both accurate and complete, whereas Davison²⁷ only assessed the accuracy of statements made on the site against the respective statement in the guideline without requiring that all recommendations from nutritional guidelines appear on the Web site. While it is possible to evaluate accuracy without demanding completeness, studies in which authors just prepared checklists for completeness sometimes raise the question as to how and to what extent accuracy has been evaluated.^{29,35,54- 55,75} For example, the Soot⁷⁹ score is a weighted checklist giving points for each broad topic covered by the site (such as "treatment options"), but does not specify whether topics were merely present or absent or if they were evaluated for accuracy.

Of the 19 studies that evaluated completeness as a distinct entity, 5 studies used an external source a priori to define elements: 2 used evidence-based guidelines, 2 used primary literature, and 1 used a textbook. The remaining studies used unclear sources or the personal opinion of the author, either a priori (7 studies) or a posteriori (7 studies).

Twenty-two studies^{11,13- 15,23,31,35- 36,38,42- 44,46,58- 59,70,72,74- 75,78- 79,87,90} used a composite score system, most frequently the Soot score,^13,46,58,79 for adherence to quality criteria. Two studies^38,87 used DISCERN⁹⁶ and another used Suitability Assessment of Materials (SAM),⁶⁵ both described by their developers as "validated" instruments to assess printed patient education material; however, none of these have been validated in a way such that a higher score would predict better health outcomes or consumer satisfaction.

The quality scores for each study are shown in Online Table A (available in PDF format). Out of the 79 studies, 49 received an S-score and an E-score of 50 or less, indicating that both search strategy and evaluation were not well reported or rigorous. However, of the 31 studies using more than 1 rater, 20 made efforts to ascertain the within-study reliability of their instruments, which is in contrast to what has been criticized in a previous study about Web-based instruments.⁹⁷

Most studies (55 [70%]) concluded that quality is a problem on the Internet (κ for coding conclusions, 0.88). Seventeen (21.5%) were neutral. Only 7 (9%) came to a more positive conclusion, none of which used evidence-based guidelines as a criterion standard. The mean S-score of the positive studies was significantly lower than that of the negative studies (29.4 vs 45.0; P = .02), as was the E-score (15.6 vs 37.9; P = .04), indicating that the more enthusiastic studies used a less rigorous search and evaluation strategy than did negative studies. In a logistic regression model with both scores as predictive variables and the study conclusion "positive/negative" as the dependent variable, the odds ratio (OR) for S-score is 0.929 (95% confidence interval [CI], 0.871-0.991; P = .02), ie, for each additional S-score percentage point the odds to reach a positive conclusion decreases by 7.1%; the OR for E-score is 0.938 (95% CI, 0.884-0.996; P = .04), ie, for each additional E-score percentage point the odds for a positive conclusion decreases by 6.2%.

When studies evaluated accuracy, the proportions of inaccurate Web sites depended on the level of evidence used as a criterion standard: studies that didn't report the criterion standard or that used personal opinion found an average of 15.4% of Web sites inaccurate; those using literature, textbooks, or expert consensus, 35.3%; and those using clinical guidelines, 38.3%. In an ordinal logistic regression model, the reported proportions of inaccurate Web sites were significantly associated with the 3 levels of evidence used by authors as criterion standard (P<.001).

Even if authors had similar results, they sometimes interpreted these differently. For example, 3 studies found that about 5% of cancer Web sites provided inaccurate information, but these results were interpreted as being "of concern,"¹⁶ "encouraging,"⁷⁶ or even "reassuring."⁴⁶ In at least 1 case investigators reviewed the same topic area, but arrived at opposite conclusions.^28,75

We reviewed 79 studies in which authors evaluated a total of 5941 Web sites and 1329 Web pages, and reported 408 evaluation results for 86 distinct quality criteria (Online Table B available in PDF format).

In our review, most authors who evaluated content found significant problems, criticizing lack of completeness, difficulty in finding high-quality sites, and lack of accuracy, in particular if "accuracy" also implied "completeness."

Five of eight studies reporting results on completeness found that around 90% of Web sites were "incomplete" (Online Table B available in PDF format). However, completeness as a requirement has questionable validity from the perspective of the user or the public health researcher. First, too much information may overburden users. Web sites may deliberately and with good reason focus on a single topic in-depth rather than aiming for comprehensiveness. Second, in contrast to printed educational material, a single Web page or Web site is part of a universe of information: a topic not covered by one Web page or site may be covered by another (perhaps linked) Web page. Consumers will usually search across different Web sites when looking for specific health information.⁹⁸ Mechanical comparison of elements from a guideline with elements covered by a single Web site without taking into account the context and purpose of the site or exploring links to other sites is of limited use. Perhaps a better approach would be to evaluate whether materials cover the topics they claim to be discussing^96,99 and if they are balanced.

Prevalence figures of inaccurate Web sites differ across different domains, eg, diet and nutrition sites (45.5%²⁶ and 88.9%⁶⁴ inaccurate information) vs cancer sites (4% for prostate cancer,⁴⁶ 5.1% for breast cancer,⁷⁶ 9% for English-language and 4% for Spanish-language breast cancer documents,¹⁴ 6% for testicular cancer,⁴⁶ or 6.2% for Ewing Sarcoma¹⁶). While such prevalence figures may suggest that diet information on the Web is of poorer quality than cancer information, unadjusted comparisons across studies have to be made with care, as at least 3 potential confounders should be considered.

First, results heavily depend on the rigor of the methodology used: studies that used personal opinion as a criterion standard found fewer inaccurate Web sites and more often came to a positive conclusion than studies using more rigorous criteria.

Second, many studies use the terms "Web site" and "Web page" interchangeably, making comparisons difficult.

Third, different and often poorly described sampling and selection strategies were used. While in theory a truly random sample of Web sites could be identified (by choosing random Internet protocol [IP] addresses¹⁰⁰), this approach is not practical for identifying Web sites for a given topic. Thus, all studies used search engines, catalogs, or lists of popular Web sites. However, the choice of the search strategy may greatly confound the results. Most studies mimicked how consumers would search (although only 3 involved actual consumers), hand-selecting popular sites. As there is no consensus and little research on how a typical consumer searches,⁹⁸ studies used various strategies, mostly picking the top-ranked results from a search engine. As many search engines can rank the better sites first, the search tool could influence the results. Moreover, even the same search engine may give a different result if different sampling strategies (ie, which sites are picked) are used. For example, Suarez-Almazor et al⁸³ showed that, in the Webcrawler search engine, the first 20 ranked hits are more relevant, have less financial interests, contain less alternative therapies, and are more often nonprofit organizations, than sites ranked lower. Moreover, the selection of search terms may critically determine which Web sites are retrieved. For example, using the term "coronary heart disease," Eachus³¹ found only 2 sites (among 110) provided by lay people. His conclusion that "the concern that the Internet would be a major source of low-quality health information, particularly that provided by unqualified members of the lay public, is not supported by [our] findings" may be confounded by the choice of search terms, as lay people might not necessarily use terms such as "coronary heart disease." Who devises the search strategy and conducts the search can also affect the quality of the retrieved sites. For example, in a comparison of results of a search devised by doctors with a search devised by information experts, Groot et al⁴³ observed differences in the credibility and accuracy scores of the retrieved samples.

The quality of Web sites should be interpreted in the larger context of information in other media to determine whether the Web is "the beginning of an epidemic of misinformation or nothing more than a variation of what is endemic."² Many of the shortcomings detected likely are not specific to the Web and are also present in other media. For example, 2 of the 4 "erroneous information" elements found on 65 Web pages identified by Biermann et al¹⁶ were found in the online version of the Encyclopaedia Britannica, which probably has the same inaccuracies in the printed edition. This issue, and the relatively low prevalence (6.2%) of inaccurate information, was generally ignored when this study was widely quoted in the lay media as evidence that the "Internet can be a quick link to bad health information."¹⁰¹

Studies assessing information in traditional media also frequently report high prevalences of inaccurate or incomplete information. In an early study, authors found 70% of health information broadcast on television to be inaccurate, misleading, or both.¹⁰² In another study, authors rated as inaccurate 76% of the information about oral hygiene from television, 53% from magazines, and 12% from newspapers.¹⁰³ Another study of the popular press found 20% of the information on oral cancer to be a "mix of accurate and inaccurate information."¹⁰⁴ The proportion of inaccurate press reports on healthy eating was found to be 55% in free advertising newspapers, 28.9% in lifestyle magazines, 29.9% in general interest magazines, 17.5% in health magazines, and 14.1% in newspapers.¹⁰⁵ In another study, 50% of the advice in newspaper advice columns was rated inappropriate, with critical issues only partially covered or not covered at all in 76% of the articles, and 58% were rated unsafe or potentially dangerous.¹⁰⁶ Inquiries by telephone to libraries yielded a 3.6% rate of inaccurate information.¹⁰⁷

The perceived quality problem on the Internet is not restricted to the health sector: a study investigating the quality of general scientific information found that 10% to 34% was inaccurate, 20% to 35% was misleading, and 48% to 90% was unreferenced.¹⁰⁸

Very few studies in our sample directly compared Internet information with information found elsewhere. One study³⁵ evaluated both printed and Web-based patient education materials, concluding that "there was no significant difference between the ranges of scores [incorporating content, writing style, design, and readability] for Internet and non-Internet leaflets"; however, subjectively authors felt that "the overall quality of the Internet leaflets was more variable and the information less comprehensive." Two studies compared the readability of Internet information with the readability of printed information. One study³³ concluded that SMOG readability levels of Internet information were significantly higher compared with other (printed) patient information materials; the other study⁶⁵ observed no differences in the proportion of patient information written above the 9th grade level, but 87.5% of Web information materials vs only 14.3% of printed patient information, were deemed unsuitable based on the Suitability Assessment of Materials score (however, as the 8 Web-based patient education materials came from only 2 different Web sites, this result may not be representative). We identified only 1 study that compared the accuracy of advice obtained on the Internet (in a pharmacy newsgroup, hence this study was excluded from this review) with advice obtained in the real world (from drug information centers).¹⁰⁹

In summary, the prevalence figures of inaccurate or incomplete Web information reported by the studies in this review are difficult to interpret or compare, are unlikely to be representative or generalizable, and must also be considered against the background of imperfect consumer health information in other media.

The presentation criteria evaluated by most investigators are considered to be quality criteria because: (1) their presence is deemed ethical according to several codes of conduct for Web publishing^110- 114 (eg, transparency and accountability criteria, such as disclosure of authorship); (2) they help to create context and to avoid misunderstandings (eg, disclosure of sponsorship, purpose); (3) they empower users to select the information that is best for them in their individual situation (eg, disclosure of target audience); (4) they empower users to validate the information themselves (eg, references, contact addresses); or (5) they may influence the accessibility of information or effectiveness of communication (eg, search capabilities, speed, design). In addition, several authors have attempted to establish whether technical criteria or other site characteristics may be used as predictors for content quality,^{11,42,47,58,87,91} but the results have been conflicting and inconclusive. A number of studies found, perhaps unsurprisingly, that the length of a document is correlated with better content scores.^42,58,87 A few studies suggest that the source may be a predictor for content quality, with commercial sites often scoring lower than academic sites,^59,74- 75 but the way in which content is evaluated may also influence this relationship. Other technical criteria influenced by the source include references, which are more often found on Web sites targeting medical professionals,²³ on academic^46,79 or educational⁸⁹ sites, or on sites owned by organizations.⁴² Individual authors often are not disclosed on government sites¹⁴ or sites of organizations and drug companies.⁴² Given these complex relationships, it seems unlikely that a simple scoring system could be developed that predicts content quality across domains and site populations.

As with content criteria, the ability to compare or pool results across studies is impeded by wide variations in sampling strategies, methods, and operational definitions even for the same quality criteria (eg, what constitutes "attribution"). Moreover, there is no consensus on the unit of evaluation (eg, should the update cycle be published on each Web page, or is a site-specific disclosure sufficient) or granularity with which information should be provided (eg, is it sufficient to post the year of last update, or should the exact date be given).

Despite this lack of consensus, some quality criteria are consistently given (eg, "ownership disclosure," with 99% of sites providing information) or not given (eg, "credentials of physicians not disclosed," with 97.5% not complying) (Table 2).

As with content criteria, presentation criteria should be put into context by comparison with non–Web-based information. While in our study 64% of Web sites failed to provide a date of update, investigators evaluating printed patient leaflets have found that 53%,³⁵ or, in another study, one-third,¹¹⁵ did not include publication dates.

Since most authors used criteria derived from the print world,⁷ we noted important omissions concerning Web-specific criteria. For example, very few studies evaluated the privacy policy or the possibility to encrypt confidential information, and few studies checked whether the target audience or the target country (which is important in a global medium¹¹⁶) were clearly disclosed. None of the studies tested usability¹¹⁷ or accessibility (eg, compliance with guidelines of the Web Accessibility Initiative, ascertaining that the site is available to people with disabilities or with low-end technology), and only 1 study checked whether metadata were provided,⁷⁶ which could greatly enhance the consumer's ability to select and filter information.⁶

Among the public health objectives of the US Department of Health and Human Services is to "increase the proportion of health-related World Wide Web sites that disclose information that can be used to assess the quality of the site."¹¹⁸ How progress in this area might be achieved or evaluated is an open question. A meta-analysis of cross-sectional studies is one possible way, but one must take into account the different methods used (or the methods must be standardized) before meaningful conclusions can be drawn. Longitudinal studies could be used to assess changes over time, using a consistent methodology either to identify and assess Web sites for a given topic (with the caveat that observed changes may be changes in the ability of users to find better sites, ie, improvements in search engine technology to rank better sites first) or to follow up a cohort of Web sites. Two studies have attempted a longitudinal approach, with conflicting results; one observed an improvement, and the other a deterioration, in quality.^36,57,90 The third possibility is to promote among site developers the use of machine-processable disclosure statements as Web site labels (meta-data),^6,119 which would allow automatic tracking and analysis of the proportion and characteristics of Web sites making such disclosure statements. This would also facilitate the development of intelligent systems able to guide users to trustworthy Web sites.¹²⁰

The epidemiology of consumer health information on the Web is an emerging research discipline at the intersection of medical informatics and public health. Many descriptive, cross-sectional studies have attempted to draw attention to perceived "outbreaks" of misinformation on the Web by estimating the proportion or prevalence of inadequate health information. However, the individual's risk (R) of encountering an inadequate site on the Web is a function of both the proportion of inadequate information on the Web (P) and the inability (I) of the individual (or his tools) to filter the inadequate sites. Since studies usually report R, but not I, we cannot infer P, or adjust study results to make them comparable across domains or time. Even if we could know P, we would still not know how this measurement of "true" misinformation on the Web translates into health outcomes or critical incidents in a population. On an individual level, R can be reduced by improving the ability of the user to locate trustworthy sites or to filter the inadequate ones. Public e-health interventions such as MedCERTAIN¹²¹ therefore strive to reduce P and I by increasing, for example, the proportion of health information providers making disclosure statements¹¹⁹ and by empowering consumers to identify trusted sites through educational and technological innovations, including the possibilities of the semantic Web.¹²⁰

Given the difficulty in interpreting descriptive studies without control or comparison groups, future studies should use analytic rather than descriptive approaches to investigate the relationship between quality markers and other variables, including outcomes, or to compare different Web site populations or media. Such studies are urgently needed to help in the ongoing process to develop methods and instruments to guide consumers to quality information and to identify factors that can be assessed to predict favorable patient outcomes.

Finally, studies evaluating content should also harness the potential of the Web as a source for qualitative data¹²²: rather than getting bogged down by the question of how much information is inaccurate, one could analyze where and why gaps exist between evidence-based medicine and health information on the Internet, which may elicit a wealth of valuable data that may inform priorities for research, health communication, and education.