Quality of Efficacy Research in Complementary and Alternative Medicine

Complementary and alternative medicine (CAM) has received increased attention in mainstream medicine since a 1993 study demonstrated that one-third of US adults used some form of “unconventional medicine.”¹ By 2002, the prevalence of CAM use by adults had increased to 62% to 68%,^{2 - 3} and it has become clear that users of CAM are not primarily dissatisfied with conventional care, but have a more holistic approach to health⁴ or simply appreciate multiple treatment options.⁵ What was once identified as “alternative medicine” has become “complementary,” “holistic,” and “integrative”; indeed, CAM therapies, such as probiotics, melatonin, massage, yoga, and acupuncture, have become part of the conventional medicine armamentarium, and the demarcation between CAM and mainstream medicine continues to shift.

Physicians have been scrambling to catch up with the exponential increase in CAM use among their patients. Most discussions of CAM are initiated by patients and families, rather than by physicians,⁶ and patients may not disclose CAM use to their physicians.⁷ Recent surveys show that most physicians are aware of their patients' interest in using CAM, believe that CAM may have beneficial effects, and are eager to seek reliable, evidence-based information about CAM for both personal and professional practice.^{6 ,8 - 9}

Limiting the indiscriminate use of costly or dangerous ineffective therapies—whether based in conventional or unconventional medicine¹⁰ —should be a high priority for all clinicians. This is especially true for children, whose unique physiology, developmental and cognitive trajectories, and position in society render them particularly vulnerable. Although often touted as “natural” and “safe,” some CAM therapies have the potential for adverse effects (eg, lead poisoning from some ayurvedic remedies),¹¹ may cause interactions with mainstream medical therapies (eg, oral contraceptives and St John's wort),¹² and involve significant time, effort, and cost burdens for families (eg, biofeedback).

In this issue of JAMA, Weber and colleagues¹³ report the results of an 8-week randomized, placebo-controlled, double-blind trial of Hypericum perforatum (St John's wort) in 54 children aged 6 through 17 years who met criteria for the diagnosis of attention-deficit/hyperactivity disorder (ADHD) and who were not receiving any other ADHD-related treatments, including pharmaceuticals, during the study. Changes in ADHD symptoms as measured by the ADHD Rating Scale-IV, changes in global functioning as rated on the Clinical Global Impression Improvement Scale, and safety as rated by the Monitoring of Side Effects Scale were the primary outcomes. Additional assessments included the parent-report Child Behavior Checklist (CBCL) and Conners' Parent Rating Scale, CBCL-Youth Self Report Form for children aged 11 years or older, and the parent- and child-report forms of the Pediatric Quality of Life Inventory 4.0 Generic Core Scales. In both intent-to-treat and per-protocol analyses, the researchers found no significant group differences on any of the efficacy measures or in frequency of adverse effects. Medication adherence was 82%, and parents, children, and the principal investigator were not able to correctly and consistently identify the child's treatment status, indicating adequate blinding.

Not only is this the first reported study of Hypericum for ADHD in children and adolescents, but it is also a noteworthy example of a high-quality randomized controlled trial (RCT) evaluating a specific CAM therapy in children. Among the strengths of the trial are a stringent research diagnosis of ADHD using a structured psychiatric diagnostic interview, rather than relying on ADHD-specific rating scales alone; a pharmaceutical washout period; a placebo run-in phase to assess likely adherence and placebo responders; blinding of both participants and investigators; a centralized, concealed allocation sequence for randomization coordinated by an independent data manager; randomization occurring after eligibility exclusions were made; explicit assessment of medication adherence; checks on the adequacy of blinding; objectively defined and measured outcome variables; intent-to-treat analysis; and full disclosure and description of exclusions prior to randomization, erroneously randomized participants, and postrandomization withdrawals. The primary study limitations are its small sample size and enrollment of only 57% of eligible participants.

The number of RCTs of CAM indexed in MEDLINE increased from fewer than 200 in 1982 to 1200 in 2002.¹⁴ In 2008, more than 7500 CAM trials were indexed in MEDLINE, with more than 1600 involving children aged 18 years or younger. In a study assessing the quality of 207 RCTs used for systematic reviews of homeopathy, acupuncture, and herbal extracts, Linde et al¹⁵ reported that less than half of the RCTs were in English or indexed in MEDLINE, and most failed to describe how the randomization sequence was generated, inadequately concealed the allocation sequence, did not catalog the number and reasons for postrandomization attrition, or did not use intent-to-treat data analysis. Trials with these deficiencies often lead to exaggerated positive treatment effects.^{2 ,16}

Most studies of CAM efficacy, both in adults and children, are generally of poor quality; this is true for many conventional medical practices as well.¹⁰ In mainstream medicine, the concepts of outcomes research, evidence-based medicine, and the need for rigorous methods and quality standards for reporting clinical trials have emerged just in the past few decades.^{14 - 18} Only very recently have these standards been extended to RCTs of nonpharmacologic treatments.¹⁹ In 2005, an Institute of Medicine report stated that “the same principles and standards of evidence of treatment effectiveness apply to all treatments, whether currently labeled as conventional medicine or CAM.”¹⁴

While the RCT should remain the gold standard for providing evidence of treatment efficacy for both CAM and conventional therapies, many interventions pose unique challenges to rigorous RCT methodology. For example, randomizing participants may be difficult or impossible when the therapy to be evaluated relies on participants' belief in the treatment or relationship with the practitioner.¹⁴ Use of placebo and blinding may be difficult in therapies such as acupuncture, yoga, psychotherapy, or surgery, although techniques such as placebo needles that do not actually enter the skin have been developed. Even when a plausible placebo can be used, placebo and expectation effects can be very large in both CAM and conventional interventions and may be part of the mechanism of treatment efficacy.

Researchers may have difficulty defining precise and measurable outcomes for therapies for which the main effect is subjective (eg, general well-being, level of energy) or dependent on the skill of the practitioner (eg, therapeutic touch, psychotherapy). Some CAM therapies (eg, homeopathic remedies, traditional Chinese medicine) are tailored for an individual's specific needs²⁰ and may not be able to be studied at a conventional “active ingredient” or “dose” level. Moreover, individuals often use a variety of CAM modalities simultaneously (eg, meditation, aromatherapy, herbs) or adjunctively with conventional therapies,^{1 ,21} and secular trends in lifestyle values such as diet, exercise, yoga, and massage may attenuate or magnify treatment effects.

Some of these challenges could be addressed through alternative study designs; for example, multiple N of 1 trials could be used to test the effectiveness of highly individualized CAM therapies, and preference RCTs can empirically test the effects of patient preferences on outcomes.¹⁴ Placebo and expectation effects can be addressed by excluding high placebo responders, as in the Hypericum trial,¹³ or by experimentally manipulating such effects as its own variable (eg, as graded dose-responses). “Attribute by treatment interaction” research methods can evaluate the extent to which interactions between individual patient and treatment characteristics, as opposed to the treatment's overall effect across a group of patients, explain variability in outcomes.²²

Ultimately, increased attention to and emphasis on deepening a rigorous evidence base for all health care practices will benefit patients and families. The time for bad science, whether in conventional or unconventional medicine, is past.