Evidence-Based Medicine in Surgery: Title and subTitle BreakThe Importance of Both Experimental and Observational Study Designs

The Institute of Medicine defines the quality of health care as the “degree to which health services for individuals and populations increase the likelihood of desired health outcomes and are consistent with current professional knowledge.”¹ Yet for much of the 20th century, the knowledge base for surgical practice was not influenced by systematic evidence, but rather by expert opinion and collective experience. Early attempts to scientifically measure and improve surgical quality, such as Codman's “end result idea”² in the early 1900s that sought to collect and analyze surgical outcomes, met intense resistance. Fortunately, evidence-based medicine in surgery has now been thoroughly accepted; sophisticated researchers are applying a variety of study designs and methodological techniques to answer an expansive number of important clinical questions. Nonetheless, despite the diversity of high-quality research being performed, many continue to contend that the randomized controlled trial (RCT) is the only valid source of evidence, whereas observational research is assumed to be only preliminary work. The need and legitimacy of both experimental and observational studies deserve to be highlighted.

Results from RCTs have revolutionized surgical practice. This is particularly evident in the treatment of breast cancer. Historically, women with carcinoma of the breast, irrespective of stage, were treated with radical surgery that completely removed the breast, pectoralis muscles, and axillary lymph nodes. This debilitating and disfiguring procedure was thought to provide better local disease control in an era when little was known about cancer biology. However, one of the first RCTs performed in breast cancer, the National Surgical Adjuvant Breast and Bowel Project (NSABP) B-04, demonstrated that radical mastectomy did not improve disease-free or overall survival in comparison with less invasive procedures.³ Soon thereafter, the NSABP B-06 RCT showed that breast conservation surgery with axillary dissection and radiation therapy did not lead to worse outcomes compared with total mastectomy.⁴ More recently, the NSABP B-32 trial⁵ demonstrated the efficacy of sentinel lymph node (SLN) biopsy compared with axillary dissection for lymph node staging, and the Z0011 trial⁶ reported that certain women with limited SLN-positive breast cancers do not require further local control with an axillary dissection. RCTs such as these have changed breast cancer care, saving countless women from undergoing more radical therapies.

In this issue of JAMA, Giuliano and colleagues⁷ address another important question in breast cancer care: the significance of occult micrometastasis in SLNs and bone marrow in 5119 women with early-stage disease. However, unlike the RCTs detailed above, this question was addressed observationally with a prospective cohort study design. Giuliano et al found that among women with hematoxylin-eosin–negative SLNs, there was no difference in overall survival associated with the 10.5% of women (n = 349) who had immunohistochemistry-positive occult SLN metastasis (adjusted hazard ratio [HR], 0.88; 95% confidence interval [CI], 0.45-1.71; P = .70). Among the 3% of women (n = 104) with occult bone marrow metastasis by immunohistochemistry, there did appear to be an associated decreased survival (unadjusted HR, 1.94; 95% CI, 1.02-3.67; P = .04). However, this association was no longer significant after multivariable analysis (adjusted HR, 1.83; 95% CI, 0.79-4.26; P = .15), perhaps due to the limited number of immunohistochemistry-positive specimens.

The importance of this study is 2-fold. First, it provides significant insights that will help guide daily treatment decisions for clinicians and their patients. Second, this study offers a clear example of how well-designed observational research can address a relevant clinical question that could not be answered with an RCT. Giuliano and colleagues used a highly standardized prospective cohort study, designed to minimize bias and confounding, and yield conclusions that may be as useful as those that might come from an RCT.

Despite such high-level observational research, not all observational studies are created equal (as is also true of RCTs). Common critiques are that observational research studies systematically overestimate treatment effect and are limited by unavoidable confounding and bias. To account for these potential limitations, investigators can use several statistical and methodological techniques, including regression, stratification, and patient matching. An example of using statistical methods to compensate for observational study limitations is the recent report by Maciejewski and colleagues.⁸ In this retrospective cohort study designed to investigate long-term outcomes after bariatric surgery among high-risk veterans, the authors compared survival after bariatric surgery with outcomes in high-risk controls, both in unmatched and propensity-matched cohorts. The survival advantage observed in the unmatched cohort (adjusted HR, 0.80; 95% CI, 0.63-0.995) disappeared in the propensity score–matched cohort (adjusted HR, 0.94; 95% CI, 0.64-1.39). Without the application of advanced statistical techniques, as well as careful cohort selection and use of a highly standardized data collection system, this study would have reached an apparent incorrect conclusion.

Surgical RCTs also have unique limitations that should be considered. For example, unlike trials comparing precise medication regimens, standardizing surgical interventions can be problematic. Surgical care is a highly technical, complex process and requires the coordination of many individuals and systems. This issue is illustrated by the recent report by Galmiche et al in the LOTUS trial⁹ that compared laparoscopic antireflux surgery with esomeprazole treatment for chronic gastroesophageal reflux disease (GERD). Each center had to participate in training sessions conducted by expert surgeons, and surgical techniques were highly standardized across all centers and surgeons.¹⁰ This study revealed nearly identical remission rates after antireflux surgery (85%; 95% CI, 81%-90%) and medical management with esomeprazole (92%; 95% CI, 89%-96%). Many of the same investigators had previously participated in a separate RCT in which surgical standardization was not performed.^{11 - 12} In this study, not only were remission rates after surgery substantially less than in the LOTUS trial with standardization (67% vs 85%), but long-term GERD symptoms such as dysphagia appeared to be qualitatively greater. If the unstandardized surgical patients were compared with the esomeprazole group from the LOTUS trial, a large and seemingly incorrect difference in treatment effect favoring medical management would have been observed.

The report by Giuliano and colleagues⁷ serves as an ideal illustration of how well-designed observational research can be conducted in surgery. RCTs should not be considered the de facto and sole source of high-level evidence. By considering RCTs and observational study designs complimentary, and recognizing the opportunities to use observational research when appropriate, it might be possible to address questions faster, cheaper, and perhaps even better than either approach alone. Above all else, it will be patients who will ultimately benefit.