The Trials and Tribulations of Clinical Practice Guidelines

Reports critiquing the validity of randomized trials,¹ meta-analyses,² diagnostic test studies,³ and economic evaluations⁴ have challenged researchers to improve the conduct of their studies and have encouraged readers to interpret them carefully. The study by Shaneyfelt and colleagues⁵ in this issue of THE JOURNAL tackles the clinical practice guideline industry, asking fundamental questions about how well they measure up to contemporary standards. Most guidelines outlined their rationale, specified the benefits and harms of health practices, and cited their evidentiary sources. However, guidelines much less often described how the evidence was found, how its worth was judged, and how diverse sources of evidence were combined to formulate recommendations.

The metric against which clinicians might judge guidelines is complex and multidimensional. Guideline creators, consumers, and critics will recognize the challenge of developing a credible instrument for this purpose. Shaneyfelt et al have captured key features of guideline development and reporting in 25 items generated by both literature review and dialogue with representatives from organizations that develop, implement, evaluate, or use guidelines. These items were piloted at workshops and pretested by the authors, and were endorsed as comprehensive and valid by guideline experts. The final items discriminated between poorly and well-developed guidelines that were ranked by an undisclosed reference standard. Although more sensitive scaling and reliability testing would further increase our confidence in this instrument, the majority of items it encompasses are those that would be important to most guideline users.

Discerning guideline readers want to determine whether appropriate methods were used to adduce the research evidence and whether the evidence supports the recommendations. However, users cannot help but notice that guideline developers must often reckon with research that is modest in rigor, discordant, or nonexistent. Although most guidelines are an amalgam of evidence and expert opinion, methods of integrating knowledge and experience into guidelines, particularly when data are sparse, are neither as mature nor as transparent as methods of incorporating research results. Only 7.5% of 279 guidelines evaluated by Shaneyfelt et al described formal methods to combine evidence and expert opinion. One exemplary exception to this generalization is a guideline on the diagnosis and treatment of idiopathic thrombocytopenic purpura from the American Society of Hematology.⁶ This guideline describes the dearth of literature, acknowledges the limitations of opinion-based directives, then issues pragmatic recommendations based on both appropriateness scores and consensus ratings.

The broad mandate of most guidelines ensures that guideline documents tend to be longer and less formulaic than other articles. Shaneyfelt and colleagues found that longer guidelines adhered to more standards than shorter guidelines. At the same time, lengthy guidelines are often ignored in practice. Thus, developers are caught in a fix, since surveys reveal that clinicians prefer pocket cards, concise pamphlets, and journal article summaries⁷ to more discursive formats. Because guideline appraisal is invariably limited by the format in which guidelines are expressed, more attention to reporting standards is sorely needed, akin to those proposed for randomized trials,⁸ meta-analyses,⁹ and economic evaluations.¹⁰

Clinical guidelines appear to be more palatable if they have the imprimatur of a professional organization.¹¹ Nevertheless, most guidelines require adaptation to the local environment. Different policies (eg, universal health care or 2-tier medicine) also may influence guideline sanctioning (eg, predominant use of streptokinase vs tissue plasminogen activator for acute myocardial infarction). Thus, homegrown guidelines ultimately may have greater currency than guidelines that cannot be applied in different settings. Portability, or "flexibility of the clinical recommendations," was addressed by the majority of guidelines in this study. Guideline compendia that provide a menu of preventive care aids¹² may allow clinicians to choose the best and most relevant among them.

Perhaps the greatest challenge to guideline developers is making explicit the values used in formulating recommendations, which Shaneyfelt and colleagues found in 6.1% of the guidelines they evaluated. Relevant questions about values include the following: from whose perspective are the outcomes considered—patients, practitioners, administrators, society, or a combination of these? How are competing values arbitrated? Although the goal of guidelines to reduce inappropriate expenditures may be laudable, this goal may be viewed as nefarious if "inappropriate" is supplanted with "unwanted" from a nonclinical perspective. While most clinicians value optimization of individual health outcomes, delivering the most cost-effective care for groups of patients via practice guidelines may require very different management strategies. Furthermore, guidelines designed to promote cost-effectiveness at the patient level may not maximize cost-effectiveness at the population level.¹³

Given the sensitivity of cost measures to different health care environments and different economic perspectives, 2 items in this instrument are particularly ambitious. These are specification of health care costs and quantification of these costs. Only when more rigorous economic evaluations become available will financial forecasts immured in practice guidelines be more credible. Until then, inclusion of costs or cost-effectiveness is a tall order for most practice guidelines.

While it is important to incorporate cost and benefit data into guidelines when possible, it is even more important to obtain empirical data on the clinical and economic sequelae of guideline implementation in the target setting. Systematic reviews of guideline implementation have found significant improvements in the processes of care in 55 of 59 studies¹⁴ and significant improvement in patient outcomes in 12 of 17 studies¹⁵ ; however, costs have not been summarized as carefully. Thus, a key additional item that should be considered for the checklist used in this study is whether the impact of the guideline has been evaluated in practice. Few guidelines fulfill this criteria before they are disseminated, but an increasing number of publications report the influence of guidelines on clinical and economic outcomes. For example, Hay and colleagues¹⁶ developed, implemented, and evaluated an evidence-based practice guideline for patients with acute upper gastrointestinal tract hemorrhage and found a reduced hospital stay, yet no difference in rebleeding, readmission, or mortality rates.

No guideline in the study by Shaneyfelt et al met all 25 criteria. However, clinicians should heed the universal caveat that "guidelines are only guidelines," intended to inform, rather than to tell what to do with pious certitude. Contrast guidelines on (1) the pharmacologic management of myocardial infarction with (2) hormone replacement therapy for postmenopausal women and (3) withdrawal of life-support decisions. The first guideline can be evidence-based, quantitative, and directive; the second may summarize risks and benefits of alternatives with context-specific provisos; and the third document may communicate only "guiding principles," since such decisions are quintessentially personal and value-specific. Thus, assessment of whether it is legitimate for a guideline to deviate from these standards requires more intimate familiarity with each guideline, with the intentions of the developers, and with the raw data used to generate the summary conclusions.

Possible explanations for the sobering findings of the study by Shaneyfelt et al include dichotomous ratings that do not grant partial credit, the fact that guideline developers do not agree on methods perceived by others to be optimal, and that optimal methods are too onerous or costly for the real world. This instrument may not capture well other sources of wisdom besides research evidence that make guidelines innately meritorious or clinically useful. Finally, this study was not designed to measure additional influences on guideline documents such as scientific and financial motives and editorial decisions.

The study by Shaneyfelt and colleagues summarizes the trials and tribulations of modern clinical practice guidelines. It exposes the diversity of guideline methodologies, raises fundamental questions about whether the proposed "quality" criteria are necessary or equally appropriate for all guideline topics, and is a clarion call for greater transparency of guideline reporting and more rigorous peer review. Careful evaluation of the real consequences of both developing and implementing practice guidelines is a health services research priority for the new millennium.