Development and Aftercare of Clinical Guidelines: Title and subTitle BreakThe Balance Between Rigor and Pragmatism

As health care technologies, evidence-based practice, and clinical guidelines continue to evolve, so does understanding of their limitations.^{1 - 3} The emphasis of the debate regarding evidence-based guidelines recently has shifted from the what to the how.⁴ Formal evaluations of guideline quality have called for improvements in their development.⁵ Guideline development represents an alloy of evidence, expert opinion, and the views and opinions of other stakeholders.⁶

The evidence-based method for guideline development is illustrated most compellingly and concretely through the fundamental difference in credibility (face validity) between the systematic review, which is at the heart of any evidence-based guideline, and the less rigorous narrative review of a body of scientific evidence.⁷ Accessing and analyzing evidence is the most expensive and time-consuming aspect of many evidence-based guideline development processes. The rigorous demands of the systematic review often cause guideline developers to revert to consensus- or expert-based models (eg, expert-based National Comprehensive Cancer Network guidelines).⁸ Reliance on consensus as the primary strategy for guideline development has been shown to undermine scientific validity as represented by clinical research.⁹

Much of the evolving technology of guideline development concerns the identification and minimization of bias that threatens validity in clinical research. For example, based on the emerging empirical evidence of publication bias and bias within the published literature, the ideal systematic review ought to be comprehensive in its sampling of the literature, including literature in different languages and unpublished sources of evidence.^{10 - 12} In the absence of registries of ongoing trials,^{10 ,13 - 14} the solution to these suspected biases requires major efforts.

Added to the problem is new evidence that the use of individual patient data in evidence pertaining to quantitative systematic reviews (meta-analyses) might yield more valid estimates of treatment effects than the more easily accomplished pooling of aggregated data across studies.^{15 - 16} Obtaining, compiling, and analyzing such data take several years of effort, and most would probably agree that a reasonable compromise in the standard of methodological rigor is defensible to produce a guideline that is valid and timely.

As methods to protect against an ever-increasing number of lurking biases continue to improve, the clinical and scientific community may be approaching unsustainable levels of scientific purity necessary to produce and maintain the ideal evidence-based clinical guideline. Not only is the cost of current rigorous methods for producing an evidence-based guideline high, but the lag time from start to final product is severely testing the patience of those who commission and fund guideline development.

Systematic reviews are often pursued as independent research projects prompted by scientific motives, funded by peer-reviewed granting agencies, and intended primarily for publication in peer-reviewed journals. Although these reviews are expected to eventually inform clinical practice, the motives are not necessarily linked to the development of a commissioned guideline. That is, systematic reviews are often conceived as independent research projects not prompted by the need for a specific guideline. Such reviews ought to be as scientifically rigorous as current methods allow.

However, although the guideline developers committed to an evidence-based process will welcome existing systematic reviews, they will have to create them for areas in which none exist. The primary motive for such a review is to incorporate it into the clinical tool of the practice guideline, which is usually a commissioned project with tight time lines for product delivery. However, it is unclear whether the same standard of methodological rigor can be realistically applied in this situation.

Are more pragmatic approaches needed for developing and maintaining evidence-based products designed specifically for clinical use? Can pragmatic solutions be developed without unduly risking validity? What is the appropriate level of risk regarding the validity of synthesized research that is equivalent to the α and β errors consciously accepted for individual clinical trials? Should achievable and ideal processes be distinguished for producing useful documents to guide care? These questions about which shortcuts, or assumptions, are acceptable could set the agenda for empirical research on the trade-off between rigor and pragmatism that currently challenges the guideline movement.

In this issue of THE JOURNAL, Shekelle and colleagues¹⁷ describe a method for evaluating the sustained validity over time of a set of evidence-based guidelines for which development was facilitated by the US Agency for Healthcare Research and Quality (AHRQ). The authors assume that the original guidelines were rigorously developed as evidence-based products and were equally valid at their completion. The importance of this article may have more to do with the methods the authors chose than the conclusions they reached. Their approach directly addresses the practical obstacles posed to researchers with a reputation for rigor when faced with real-world constraints.

In reviewing the current validity of the AHRQ guidelines, Shekelle et al emphasized 2 methods for locating evidence: the guidance of experts about the state of the guideline topic area and the focused literature search as "a more pragmatic way to help identify . . . new evidence." To defend pragmatism, they make assumptions that challenge the current standards of rigor demanded of formal systematic reviews. For example, they assume that "new evidence that is sufficient to change practice would frequently be accompanied by an editorial or commentary." Also, they "restricted the search to key journals, ie, those most likely to have published evidence of sufficient magnitude to warrant the revision of an existing national practice guideline." The authors justify their approach by describing it as a method for determining whether update of a guideline is necessary, as opposed to completing the first step of a formal update—a fine distinction. Their use of limited literature searches as "sentinel markers" for new evidence could be improved by including journals devoted to evidence synthesis that are designed to filter relevant and rigorous research for clinical uptake, such as ACP Journal Club, Evidence-Based Medicine, or related specialty journals.

Based on their methods, and using a set of intuitively appealing criteria, the authors uncovered substantial new evidence, made important comments about the current validity of AHRQ guidelines, and generated a time-to-event curve that describes the average shelf life of AHRQ guidelines. The findings led the authors to make some specific recommendations, including that guidelines should be assessed for validity every 3 years.

The authors' methods are directly relevant to the trade-off between pragmatism and the demand for rigor associated with the usual evidence-based approach. Is it possible to find a happy medium?³ Do the authors display an acceptable balance between rigor and pragmatism? Could their approach gradually (or abruptly) corrupt a rigorously developed document that will render it less valid rather than more valid and, therefore, less credible? Do the authors take reasonable and responsible methodological shortcuts under the circumstances or too much risk that could potentially subvert validity? These questions represent the need for a pragmatic methodological research agenda.

The article by Shekelle et al prompts consideration of other approaches for effective maintenance in guideline aftercare. Maintenance strategies could be guided by analyses of the baseline characteristics of guidelines that predict the need for more or less frequent updating. For example, guidelines about interventions that are based on a substantial body of "level 1 evidence"¹⁸ may be more robust to change over time compared with those based on lesser evidence. Similarly, higher-grade recommendations (eg, grade A) should be more robust than lower-grade recommendations. Guidelines in the form of technology assessments for rapidly expanding clinical indications or that are concerned with rapidly evolving technology may be more susceptible to becoming obsolete.

Strategies for guideline maintenance also could be considered during the development phase of a guideline, such as the inclusion of a list of ongoing trials that might portend the need for scheduled updates. This approach appears to be feasible for oncology-related guidelines.¹⁹ The explicitness in the descriptions of methods also might buffer against improper inferences from pragmatic methodological compromises. For example, a systematic review could explicitly acknowledge a compromise that only English-language literature was searched. It would then be up to the guideline user to decide whether this was a reasonable approach associated with an acceptable risk to validity for the guideline topic under consideration.

Notwithstanding the practical appeal for explicit methodological compromise, adoption of such an approach should not be interpreted as justification for scientific nihilism that could promote, or even justify, sloppy methods or hinder further research on methodological improvements. In the case of guideline maintenance, an alternative exists that meets criteria for both pragmatism and rigor. The obstacles faced by Shekelle et al for keeping guidelines current relate to the time permitted to lapse before many of the AHRQ guidelines were updated. Paradoxically, more frequent rather than less frequent updating may be the pragmatic strategy that best preserves the validity of the original document.

The evidence-based approach through the systematic review creates a solid foundation for recommendations that are as up-to-date as feasible. Such a review, if properly performed and documented, should never need repeating in its entirety. If guidelines are left to languish unattended for several years, the updating process can become onerous, requiring methodological compromises. But, more frequent updating of selected guidelines using the original documented literature search strategy is likely to be much less difficult because it builds incrementally on the foundation prepared in the development phase. On balance, frequent updates using rigorous methods are likely to be easier to accomplish and more valid than infrequent updates that compromise rigor to make up for lost time.²⁰

Until more is known about optimal strategies for updating guidelines, the study by Shekelle et al should be considered primarily as an interesting and important attempt to address the issue of guideline maintenance and to deal with methodological challenges for both the development and aftercare phases of the guideline life cycle. Clinicians should be cautious about accepting their recommendations regarding frequency of updating, which are based on results from a small and heterogeneous set of guidelines that represent diverse disciplinary perspectives and variations in quality. Although more practical approaches need to be investigated, physicians and guideline developers alike should continue to be vigilant about ensuring proper use of the most rigorous methods available within the constraints and challenges involved in developing and maintaining evidence-based guidelines.