Inclusion of Comparative Effectiveness Data in High-Risk Cardiovascular Device Studies at the Time of Premarket Approval FREE

To the Editor: Use of a comparator group is not required in studies used for approval of medical devices, in contrast to drugs, which require 2 randomized controlled trials. The US Food and Drug Administration (FDA) defines valid scientific evidence as “partially controlled studies, studies and objective trials without matched controls, well-documented case histories conducted by qualified experts, and reports of significant human experience.”¹ A study of new drugs in the United States found that 51% were not supported by comparative effectiveness data at the time of FDA approval.² Unlike drugs, device trials frequently use control data from previous trials (historical controls) and performance benchmarks imputed from prior studies (objective performance criteria; OPC) in evaluating outcomes. However, the lack of active controls raises concerns about selection bias and validity.

High-risk cardiovascular devices requiring premarket approval are defined as those that support or sustain life and include many invasive or implantable devices. Minor variations in manufacture do not require approval. In a review of such devices approved from 2000-2007, 52% of primary end points had controls, 31% of which were retrospective.³ In this study, we examined the type of controls used by the FDA in evaluating comparative effectiveness during premarket approval of cardiovascular devices.

Data were extracted from publicly available FDA summaries of safety and effectiveness data (herein called summaries).⁴ Summaries representing all high-risk cardiovascular devices approved by the FDA between January 1, 2000, and December 31, 2011, were analyzed. The selection and characteristics of premarket applications have been previously described.²

All primary end points were examined to determine the type of controls with which the device was compared (Box).³ We defined comparative effectiveness research as use of active controls in evaluating primary end points.¹ We described control group characteristics by approval year and device class and examined whether number needed to treat was calculated. The Kruskal-Wallis test was used to examine differences between device classes. Statistical significance was determined at P < .05 (2-sided). Statistical analyses were performed using Stata version 10 (StataCorp).

The 121 summaries identified contained 203 supporting clinical studies. These studies used 353 primary end points, of which 40% (n = 140) were evaluated against an active control, 13% (n = 45) against a historical control, 26% (n = 90) against an OPC, and 22% (n = 78) against no control. Thirty-six percent (74/203) of studies included at least 1 primary end point evaluated against an active control; 48% (58/121) of devices had an active control in at least 1 supporting study, while 35% (42/121) of devices were approved without data from any control or OPC.

The use of active controls varied by device category (Table). The number of devices approved each year was too small to allow statistical analysis, but there did not appear to be any trends over time. Of the 90 end points defined using OPC, 28% (n = 25) relied on performance goals established in previous FDA guidance documents, 26% (n = 23) on previous trial data, and 19% (n = 17) on reviews of previously published literature. Twenty-one percent (25/121) of studies did not provide an OPC source.

Only 3 end points were identified in which an active control may not have been possible. No summary commented on the number needed to treat for any primary end point.

Most high-risk cardiovascular devices were approved without comparative effectiveness data. A minority of end points, studies, and summaries since 2000 included an active comparator group. Use of active controls varied significantly by device class.

We categorized the control group used in studies conducted for premarket approval of high-risk cardiovascular devices, but we cannot comment on internal validity or rigor of the studies included or on other key aspects of the trial designs.

It is hard to know if a new device is safe and more effective than alternative treatments unless it is compared with conventional treatment. While occasionally use of active controls may not be possible, such as ventricular assist devices, more frequent use of a comparator group may help to better inform clinical and regulatory decisions.