0
We're unable to sign you in at this time. Please try again in a few minutes.
Retry
We were able to sign you in, but your subscription(s) could not be found. Please try again in a few minutes.
Retry
There may be a problem with your account. Please contact the AMA Service Center to resolve this issue.
Contact the AMA Service Center:
Telephone: 1 (800) 262-2350 or 1 (312) 670-7827  *   Email: subscriptions@jamanetwork.com
Error Message ......
Original Contribution |

Automated Surveillance to Detect Postprocedure Safety Signals of Approved Cardiovascular Devices FREE

Frederic S. Resnic, MD, MSc; Thomas P. Gross, MD, MPH; Danica Marinac-Dabic, MD, PhD; Nilsa Loyo-Berrios, PhD; Sharon Donnelly, RN, MBA; Sharon-Lise T. Normand, PhD; Michael E. Matheny, MD, MSc, MPH
[+] Author Affiliations

Author Affiliations: Brigham and Women's Hospital and Harvard Medical School, Boston, Massachusetts (Dr Resnic and Ms Donnelly); Center for Devices and Radiological Health, US Food and Drug Administration, Silver Spring, Maryland (Drs Gross, Marinac-Dabic, and Loyo-Berrios); Department of Health Care Policy, Harvard Medical School, and Department of Biostatistics, Harvard School of Public Health, Boston (Dr Normand); and Tennessee Valley Healthcare System Veterans Administration Medical Center and Vanderbilt University Medical Center, Nashville, Tennessee (Dr Matheny).


JAMA. 2010;304(18):2019-2027. doi:10.1001/jama.2010.1633.
Text Size: A A A
Published online

Context Ensuring the safety of medical devices challenges current surveillance approaches, which rely heavily on voluntary reporting of adverse events. Automated surveillance of clinical registries may provide early warnings in the postmarket evaluation of medical device safety.

Objective To determine whether automated safety surveillance of clinical registries using a computerized tool can provide early warnings regarding the safety of new cardiovascular devices.

Design, Setting, and Patients Prospective propensity-matched cohort analysis of 7 newly introduced cardiovascular devices, using clinical data captured in the Massachusetts implementation of the National Cardiovascular Data Repository CathPCI Registry for all adult patients undergoing percutaneous coronary intervention from April 2003 through September 2007 in Massachusetts.

Main Outcome Measure Presence of any safety alert, triggered if the cumulative observed risk for a given device exceeded the upper 95% confidence interval (CI) of comparator control device. Predefined sensitivity analyses assessed robustness of alerts when triggered.

Results We evaluated 74 427 consecutive interventional coronary procedures. Three of 21 safety analyses triggered sustained alerts in 2 implantable devices. Patients receiving Taxus Express2 drug-eluting stents experienced a 1.28-fold increased risk of postprocedural myocardial infarction (2.87% vs 2.25%; absolute risk increase, 0.62% [95% CI, 0.25%-0.99%]) and a 1.21-fold increased risk of major adverse cardiac events (4.24% vs 3.50%; absolute increase, 0.74% [95% CI, 0.29%-1.19%]) compared with those receiving alternative drug-eluting stents. Patients receiving Angio-Seal STS vascular closure devices experienced a 1.51-fold increased risk of major vascular complications (1.09% vs 0.72%; absolute increased risk, 0.37% [95% CI, 0.03%-0.71%]) compared with those receiving alternative vascular closure devices. Sensitivity analyses confirmed increased risk following use of the Taxus Express2 stent but not the Angio-Seal STS device.

Conclusion Automated prospective surveillance of clinical registries is feasible and can identify low-frequency safety signals for new cardiovascular devices.

Figures in this Article

Monitoring the safety of approved medical products is of vital public health importance, given that in clinical practice such medical products are often used in numbers far greater and in patient populations more diverse than when studied in premarket evaluations and clinical trials.16 Within the broad range of medical products, implantable medical devices represent high-risk products that are uniquely challenging to monitor because there is little consensus regarding the most appropriate methods to account for the complex interactions among devices, medications, patients, and implanting physicians. In addition, the lack of unique medical device identifiers challenges the effective use of administrative claims data and electronic health records as a primary data source to evaluate manufacturer-specific device safety.7

To support clinical research and quality-improvement efforts, detailed clinical registries have been established in recent years at the state, regional, and national levels for many high-risk implantable medical devices,811 which may provide unique opportunities to prospectively monitor the safety of such devices.12 The Massachusetts statewide coronary intervention registry13 was established in 2003 to monitor the quality of care of hospitals and physicians in the state. This registry is a mandatory clinical outcomes registry based on the American College of Cardiology National Cardiovascular Data Repository (NCDR) CathPCI data set and includes manufacturer-specific device information for every adult patient undergoing angioplasty procedures in nonfederal hospitals since April 2003.14 This mandatory registry represents a high-quality data source for safety surveillance, because it is comprehensively audited and adjudicated for major adverse events and risk factors.13,15

Using the Massachusetts statewide coronary intervention registry, we examined in-hospital safety signals for recently introduced interventional cardiovascular devices using an automated safety surveillance system to assess the feasibility of such an approach to prospective surveillance of medical device safety.

Study Setting and Data Sources

The Massachusetts angioplasty registry collects detailed clinical data and inpatient outcome information for all adults (18 years or older) who undergo coronary intervention at all nonfederal Massachusetts inpatient facilities. All registry records between April 1, 2003, and September 30, 2007, were included in the analysis. Detailed clinical information obtained during the hospital admission was collected prospectively by trained data managers using variables defined in the NCDR CathPCI Registry data set16 and was subject to detailed review and audit procedures at the hospital level as well as at the state level, where all major adverse events are reviewed by a panel of trained volunteer physicians and nurse data managers. The study protocol was approved by the hospital's institutional review board and the US Food and Drug Administration Research Involving Human Subjects Committee.

Automated Prospective Safety Surveillance System

A computerized automated safety surveillance tool, the Data Extraction and Longitudinal Trend Analysis (DELTA) system, was developed and validated on outcomes and clinical trial databases and shown to efficiently identify very low-frequency events, using an array of Bayesian and frequentist inference methods.1719 The system supports multiple simultaneous device-specific analyses, tracking the accumulating experience of multiple devices while monitoring multiple independent data sets simultaneously.12,20

Tools within the system allow for joining multiple related data sets and establishing independent prospective analyses using numerous analytic options including propensity matching, risk-adjusted cumulative outcomes analysis, sequential methods, and survival methods. The system can be configured to trigger alerts at flexible levels of deviation from expected outcomes and to signal the analyst through e-mail notification when an alert is generated. The surveillance tool was implemented at a central data repository to monitor the accumulating Massachusetts interventional cardiology registry for device-specific safety signals and to trigger safety alerts when specific statistical thresholds were achieved for any monitored device.

Exposures

Four classes of high-risk interventional cardiovascular devices recently approved and introduced into clinical practice during the study period were selected for safety monitoring. These included drug-eluting coronary stents, small-vessel bare-metal coronary stents, vascular closure devices, and embolic protection devices. Potential devices were selected among all high-risk devices if they met the sample size required to achieve 80% power to detect a 50% increase in adverse event rates using a type I error rate of .05. For example, assuming an average composite adverse event rate of 2.0%, a sample size of 3826 patient exposures would be required to attain 80% power to detect a 50% increase in event rate (to 3.0%). Sample size requirements varied from 853 for vascular closure device exposures to 3536 for drug-eluting stent exposures.

Based on the evolution of the NCDR data set specification over the study period, manufacturer-specific device information was available for drug-eluting stents throughout the study period. Bare-metal stents, vascular closure devices, and embolic protection devices had similar detailed information available beginning in January 2005. The unit of inference was the procedure with patients receiving multiple studied devices included in each device-outcome analysis.

Outcomes

Each medical device was evaluated for acute adverse outcomes specific to the device group as selected by the investigator team, based on clinical relevance to the device class and incorporating the recommendations of collaborators at the US Food and Drug Administration Center for Devices and Radiological Health. All adverse events and clinical risk factors were defined in accordance with the NCDR CathPCI data set definitions.16

For each stent and embolic protection device, adverse events included in-hospital postprocedure myocardial infarction (MI), in-hospital death, and a composite end point of major adverse cardiac events (MACE) including emergent revascularization, death, and MI. For the vascular closure devices studied, adverse events included in-hospital minor vascular complications (including access-site bleeding, hematoma >5 cm, pseudoaneurysm, and arteriovenous fistula), major vascular complications (including retroperitoneal hemorrhage, vessel dissection or occlusion, or need for urgent vascular procedure), and any vascular complication.

Propensity Score Matching

For each exposure of interest, a propensity score–matched concurrent control population was developed based on published risk factors for the outcome of interest as well as on factors considered by domain experts to potentially influence the selection of one device vs another in its group (eAppendix). Propensity scores were developed from a nonparsimonious hierarchical logistic regression analysis developed with the device of interest (exposure) used as the dependent variable, adjusting for baseline covariates of the factors listed in the eAppendix, as well as between-hospital differences in device use. Initial matches were selected from the population of patients exposed to an alternative device within the same group as the exposure of interest (ie, alternative drug-eluting stents). The cohorts were matched within 6 months of device implantation date and within a fixed propensity score caliper of 0.05 using a greedy matching algorithm.21

The relative imbalance between the exposed and unexposed groups was assessed using the absolute standardized difference (percentile) in covariate means and proportions, with values greater than 10% considered severely imbalanced.22 The propensity matching was considered insufficient to examine overall safety profile of the device if less than 50% of total exposures of a device were successfully matched to control patients (typically because of high use of the exposure of interest). In this circumstance, there was poor balance of the clinical features of patients receiving the device of interest and alternative (control) devices. In these situations, the potential control population was expanded through use of less restrictive device exposure parameters (ie, all drug-eluting and bare-metal coronary stents).

Surveillance Methods

Adverse-event rates were calculated quarterly for the propensity score–matched unexposed and exposed cohorts. Safety alerts were triggered if the confidence intervals (CIs) around the difference between 2 independent proportions (as measured by the Wilson method) did not cross zero, which indicates a statistically significant difference between the exposed and unexposed groups. The CIs were established using a 95% CI corrected for multiple comparisons through use of the O’Brien-Fleming alpha-spending method.12 The χ2 test was used for comparisons of categorical data, and the 2-tailed t test was used to compare continuous variables. All prospective surveillance statistical analyses were performed within the safety monitoring system (Coping Systems, Andover, Massachusetts).12,18,20 Population summary statistics were calculated using STATA version 8.0 (StataCorp, College Station, Texas). All statistical tests were 2-sided, with P < .05 considered statistically significant for all comparisons.

To further investigate a potential safety signal and explore potential subpopulations affected, a series of prespecified sensitivity analyses were performed through the surveillance system if 3 or more safety signal alerts were generated for a device-outcome pair during the analysis. These sensitivity analyses included periodic analysis, subpopulation analyses, and alternative risk-modeling methods. The periodic analysis used periodic, rather than cumulative, safety signal evaluation to explore consistency of elevated rates and temporal trends in outcomes. To explore whether potential imbalance of specific risk factors between exposed and matched populations might be related to an alert, univariate comparisons of matched and unmatched populations were performed.

Relative device safety was assessed using logistic regression–based risk adjustment using historical nonexposed patients. The multiple logistic regression model was developed using backward stepwise selection to identify predictors of specified complications, and final models incorporated those covariates with consistent associations of P ≤ .20. The model was developed and calibrated using control patients in the 12 months prior to the study period for the particular device and applied prospectively to the entire cohort of patients exposed to the device of interest. This method supported inclusion of the entire cohort exposed to the device of interest, rather than only the subset with an adequate match to a concurrent control population.

Patient, physician, and hospital deidentified data for 74 427 consecutive coronary interventional procedures performed from April 1, 2003, to September 30, 2007, in nonfederal Massachusetts hospitals were evaluated. Seven devices met the sample size requirements for automated safety monitoring, including 2 drug-eluting coronary stent systems (Taxus Express2 [Boston Scientific, Natick, Massachusetts] and Cypher [Cordis, Bridgewater, New Jersey]), 1 bare-metal stent (Mini-Vision [Guidant/Abbott, Abbott Park, Illinois]), 1 embolic protection device (FilterWire [Boston Scientific]), and 3 vascular closure devices (Angio-Seal STS [St Jude Medical, St Paul, Minnesota], Perclose Proglide [Abbott Vascular, Santa Clara, California], and StarClose [Abbott Vascular]).

Table 1 summarizes the 7 devices, along with the 21 safety analyses performed and the matched concurrent control populations chosen for each analysis. The proportion of exposures successfully matched using the propensity-matching algorithm ranged from 51% for the Cypher drug-eluting stent to more than 99% for the Perclose Proglide and StarClose vascular closure devices. The FilterWire embolic protection device and the Angio-Seal STS vascular closure device required expansion of control patient populations because of high utilization rates (and therefore limited concurrent controls in same device group) of the devices of interest.

Table Graphic Jump LocationTable 1. Medical Devices Analyzed, Outcomes Captured, Control Populations, and Final Alert Status for Each Analysis

Of the 21 safety analyses performed, 3 (14%) generated a repeated or sustained safety signal involving 2 implanted devices, prompting detailed sensitivity analysis per study protocol (Table 1). The safety alerts included an increased risk of postprocedural MI as well as an increased risk of MACE following implantation of Taxus Express2 drug-eluting stents. In addition, an increased rate of major vascular complications following implantation of the Angio-Seal STS vascular closure device was observed. All other safety analyses resulted in outcomes within (or superior to) the 95% CI established by the propensity-matched control population.

Taxus Express2 Drug-Eluting Stent

Figure 1 illustrates the cumulative safety analysis for the Taxus Express2 drug-eluting stent. A total of 18 277 procedures involved implantation of 1 or more Taxus Express2 stents, of which 14 893 (81.5%) were successfully matched to control patients receiving alternative drug-eluting stents (predominantly the Cypher drug-eluting stent). Although proportions of use differed significantly among the institutions, both the Taxus Express2 and alternative drug-eluting stents were used in all 22 hospitals included in the analysis.

Place holder to copy figure label and caption
Figure 1. Summary Safety Analysis of the Taxus Express2 Drug-Eluting Stent
Graphic Jump Location

Longitudinal propensity-matched analysis of cumulative incidence following implantation of at least 1 stent. Solid blue lines indicate mean event rates of matched controls receiving alternative drug-eluting stents; blue shaded areas, 95% confidence intervals (CIs) corrected for multiple comparisons; dashed lines, the uncorrected 95% CI bounds. Circle sizes are proportional to the number of Taxus Express2 stents used in Massachusetts in a calendar quarter. Light blue circles indicate cumulative observed event rates within propensity-matched expectations; dark blue circles, higher than expected rates (safety alerts), ie, observed rates exceeding the upper 95% CI boundaries; solid black lines, cumulative sample sizes.

As shown in Figure 1, by the end of the study period (October 2007) the rate of postprocedural MI was 27.6% higher for Taxus Express2 drug-eluting stents compared with alternative drug-eluting stents (2.87% vs 2.25%; absolute risk increase, 0.62% [95% CI, 0.25%-0.99%]). The surveillance system issued the first alert in quarter 5 of the analysis and then demonstrated sustained alerts for increased risk with the Taxus Express2 drug-eluting stent beginning in July 2005. Similarly, the rates of MACE were increased by 21.1%, driven by the increased postprocedural MI difference, for the Taxus Express2 drug-eluting stent relative to the MACE rate for the propensity-matched control population (4.24% vs 3.50%; absolute increase, 0.74% [95% CI, 0.29%-1.19%]) (Figure 1), and a sustained safety alert for MACE was triggered beginning in July 2007. No increased risk of death was observed among the exposure cohorts (Figure 1).

Baseline disparities between patients receiving the Taxus Express2 stents and those receiving alternative stents were virtually eliminated for nearly all covariates after the propensity match was applied, with the standardized difference measure less than 10% for all covariates (Table 2). Significant differences remained, however, in age, exposure to glycoprotein IIb/IIIa antagonists, mean final stent diameter, and maximum lesion length (as a surrogate for stent length) (Table 2); however, the findings regarding safety of the stent were unchanged after controlling for these factors in multivariate analysis.

Table Graphic Jump LocationTable 2. Distribution of Clinical Covariates in Patients Receiving Taxus Express2 or Alternative Drug-Eluting Stents

Predefined sensitivity analyses were automatically performed to explore potential explanations of the positive safety signals. Rolling-quarter (period-based) analysis of the Taxus Express2 drug-eluting stent demonstrated consistent postprocedural MI and MACE rates at or above the safety-alerting threshold throughout the study, thereby confirming a temporally consistent increased hazard for use of the Taxus Express2 stent. Additionally, a multiple logistic regression predictive model for the risk of postprocedure MI based on all non-Taxus drug-eluting stents used in 2003-2004 (the period immediately preceding the study period) was applied prospectively to the entire cohort of patients receiving Taxus Express2 stents. We observed no significant difference from the alerting behavior observed in the original propensity match compared with the analysis using all patients receiving the stent.

Angio-Seal STS Vascular Closure Device

A total of 8015 case patients receiving Angio-Seal STS vascular closure devices were successfully matched from a total population of 10 801 patients receiving Angio-Seal STS devices (74.2%). Although frequency of use differed, both the Angio-Seal STS device and alternative vascular closure strategies were used at all 22 institutions included in the analysis. Patients receiving the Angio-Seal STS device were found to have a consistently higher than expected rate of major vascular complications but a lower rate of minor complications compared with alternative vascular management strategies using the propensity-matched concurrent control method (Figure 2).

Place holder to copy figure label and caption
Figure 2. Summary Safety Analysis of the Angio-Seal STS Vascular Closure Device
Graphic Jump Location

Longitudinal propensity-matched analysis of cumulative incidence following implantation of at least 1 device. Solid blue lines indicate mean event rates of matched controls receiving alternative vascular closure devices; blue shaded areas, 95% confidence intervals (CIs) corrected for multiple comparisons; dashed lines, the uncorrected 95% CI bounds. Circle sizes are proportional to the number of Angio-Seal STS devices used in Massachusetts in a calendar quarter. Light blue circles indicate cumulative observed event rates within propensity-matched expectations; dark blue circles, higher than expected rates (safety alerts), ie, observed rates exceeding the upper 95% CI boundaries; open circles, lower than expected rates, ie, observed rates less than the lower 95% CI boundaries; solid black lines, cumulative sample sizes.

By the end of the observation period, the matched subset of Angio-Seal STS case patients experienced a 51.3% increased risk of major vascular complications compared with the concurrent control population (1.09% vs 0.72%; absolute increased risk, 0.37% [95% CI, 0.03%-0.71%]). Baseline disparities between patients receiving the Angio-Seal STS devices and those receiving alternative devices were reduced for most covariates after the propensity match was applied. Some differences remained between the matched cohorts in the proportion of patients presenting with symptoms of congestive heart failure and low ejection fraction; however, the standardized difference demonstrated adequate balance of all covariates after the match (Table 3).

Table Graphic Jump LocationTable 3. Distribution of Clinical Covariates in Patients Receiving Angio-Seal STS or Alternative Vascular Closure Devices

The prespecified sensitivity analysis exploring temporal changes in outcomes for the Angio-Seal STS vascular closure device demonstrated significant heterogeneity in the outcomes observed, with an early period of increased risk of major vascular complications (March 2005 through December 2005) followed by consecutive periods of acceptable risk (January 2006 through December 2006). A detailed exploration of differences in clinical and demographic covariates was performed and demonstrated a significant increase in the use of the direct thrombin inhibitor bivalirudin as well as a reduction in the use of the Angio-Seal STS device in patients with concomitant venous access between the early and later periods. The propensity-matched rates of major vascular complications (and overall vascular complications) for patients treated with bivalirudin demonstrated that patients receiving the Angio-Seal STS device had a 38% reduced rate of major vascular complications compared with patients treated with alternative vascular management (0.49% vs 0.79%; absolute reduction, 0.30% [95% CI, 0.14%-0.46%]).

Lastly, a second propensity analysis including only patients who received alternative implantable vascular closure devices confirmed the principal findings of these analyses. However, only 48% of patients receiving Angio-Seal STS devices could be matched to this control group because of the high overall use of Angio-Seal STS devices within the registry population.

An independent multiple logistic regression predictive model for the risk of major vascular complications based on all non–Angio-Seal STS vascular closure devices used in 2004 (the period immediately preceding the propensity-matched analysis period) was developed and applied prospectively to the entire cohort of patients receiving Angio-Seal STS devices. We observed no significant difference between the alerting behavior using the logistic regression prediction method vs the original propensity match, thereby supporting the findings of the primary propensity analysis.

This study demonstrates the feasibility of automated safety surveillance of implantable devices when applied to a clinical outcomes registry through the use of computerized adverse event surveillance. The methodologies incorporated into the surveillance system were able to distinguish low-frequency medical device safety risks not highlighted in premarket approval studies.

In this study, 14% of the monitored device-outcome pairs triggered a sustained potential safety signal necessitating detailed sensitivity analysis. We found that patients receiving the Taxus Express2 drug-eluting stent experienced a significantly higher rate of postprocedural MI compared with patients receiving alternative drug-eluting stents. This finding was sustained after temporal trend analysis and alternative risk modeling approaches, although the effects of smaller stent diameters and shorter total lesion lengths in recipients of Taxus Express2 drug-eluting stents could not be fully excluded as confounders of the observed results.

Our findings regarding the increased risk of periprocedural MI with the Taxus Express2 stent are supported by trends reported in prospective randomized clinical trials, including the Taxus V trial, in which patients receiving multiple Taxus Express2 stents experienced significantly increased risk of MI within 30 days of the procedure compared with patients randomly assigned to receive bare-metal stents (8.3% vs 3.3%, respectively; P = .047).23 A potential mechanism for this increased risk has been proposed by Popma et al24 in a detailed angiographic review of the ENDEAVOR IV trial, which demonstrated an increased frequency of side-branch compromise associated with periprocedural MI when the Taxus Express2 drug-eluting stent was used as compared with the Endeavor drug-eluting stent.

Use of the Angio-Seal STS vascular closure device was associated with a significantly increased risk of major vascular complications in the early period of experience with the device. However, this risk reversed over time in association with changes in practice in antithrombotic therapy. For the Angio-Seal STS device, we conclude that case patient selection, changes in medical therapy, and potential learning curve effects likely explain a significant proportion of the increased risk observed in the early period of use.

There are few comparative data on the safety of specific vascular closure devices, although the Angio-Seal STS device has not previously been shown to pose an increased risk of major vascular complications relative to other vascular closure devices in 2 meta-analyses25,26 and a large comparative safety study.27 A learning curve effect in the use of the Angio-Seal device has been described,28 as has the significant reduction in major vascular complications through the use of bivalirudin during percutaneous coronary intervention procedures that was observed during the period of safety surveillance in this analysis.6,29,30

There was no evidence for increased risks of the analyzed outcomes in the use of the other cardiovascular devices studied, including the Cypher drug-eluting stent, the Mini-Vision bare-metal stent, the FilterWire embolic protection device, and the StarClose or Perclose Proglide vascular closure devices.

This study demonstrates the feasibility of automated surveillance of clinical device registries and provides a potential framework for temporally efficient comparative safety analysis over broad populations of real-world patients. Prospective computerized monitoring, such as that demonstrated here, can support the simultaneous monitoring of many device-outcome pairs, thereby permitting the efficient use of valuable human resources to explore specific risks identified through the automated safety screening algorithms and alerts.

While the number of data sources having features similar to those of the Massachusetts angioplasty registry are limited, such detailed clinical registries are becoming more widespread. In addition, alternative clinical data repositories, such as pooled data from increasingly available electronic health record systems as well as medical condition–specific clinical outcomes registries, may prove to be valuable resources for additional exploration of automated safety surveillance approaches. Such automated prospective surveillance of medical device safety can aid public health officials who rely on passive surveillance tools that lack denominator data (ie, comprehensive exposure information) and therefore provide accurate comparative assessments of safety risk. In addition, federally mandated postapproval studies are often of limited scope and duration, have limited control populations, and often lack statistical power to detect very low-frequency safety signals.3133 Automated safety surveillance may also complement plans for the recently announced Sentinel Initiative,34 an active surveillance program being implemented by the US Food and Drug Administration to use existing electronic health care information sources to efficiently generate, strengthen, or confirm safety signals for medical products.7 In this context, registries will achieve optimal utility when linked with longitudinal data sources. This is particularly true for implantable devices, for which outcomes of interest often extend beyond the hospital stay.

It is important to note that potential signals generated in automated surveillance systems must be interpreted with caution and that system safety alerts are intended to generate hypotheses for more in-depth exploration. All potential signals identified through such methods require further evaluation, including sensitivity analyses and more formal epidemiologic studies (which may include medical record validation of outcomes as appropriate). Also, while simple alert boundaries based on statistically significant increased risk were used in this analysis, alternative alert boundary conditions potentially incorporating the severity of the adverse outcome being studied may help inform the choice of the magnitude of the signal to be identified as well as the threshold for significance that would merit additional exploration.

Several additional limitations of the present analysis may affect the generalizability of the results. The Massachusetts angioplasty registry is an audited and adjudicated data set that provides a high-quality data source, but it may not be representative of other postmarket device clinical registries. In an effort to reduce bias in our estimates, our case-matching strategy led, in some instances, to the exclusion of unique, yet high-risk, subsets of exposed patients. In this case, because matching controls could not be identified, we were unable to make any comparative statements about safety in these patient populations. Also, potential for residual confounding (from known and unknown factors, including those influencing patient selection) may remain, despite the propensity-based adjustment methods used.

In conclusion, automated safety surveillance of medical devices is feasible using automated monitoring tools applied to detailed clinical registries and can efficiently help identify emerging potential postmarket safety risks. Automated medical product surveillance can complement existing public health strategies, providing an additional mechanism to assess the comparative safety of approved medical products and improve the quality of health care delivered.

Corresponding Author: Frederic S. Resnic, MD, MSc, Brigham and Women's Hospital, 75 Francis St, Boston, MA 02115 (fresnic@partners.org).

Author Contributions: Dr Resnic had full access to all of the data in the study and takes responsibility for the integrity of the data and the accuracy of the data analysis.

Study concept and design: Resnic, Gross, Donnelly, Normand, Matheny.

Acquisition of data: Resnic, Donnelly, Matheny.

Analysis and interpretation of data: Resnic, Gross, Marinac-Dabic, Loyo-Berrios, Donnelly, Normand, Matheny.

Drafting of the manuscript: Resnic, Normand.

Critical revision of the manuscript for important intellectual content: Resnic, Gross, Marinac-Dabic, Loyo-Berrios, Donnelly, Matheny.

Statistical analysis: Resnic, Normand, Matheny.

Obtained funding: Resnic, Gross, Marinac-Dabic.

Administrative, technical, or material support: Resnic, Gross, Donnelly, Normand.

Study supervision: Resnic, Gross, Marinac-Dabic, Donnelly.

Financial Disclosures: Dr Resnic and Dr Matheny reported filing a patent application, currently under review at the US Patent Office, for several key analytic components of the DELTA surveillance system; the intellectual property rights of Drs Resnic and Matheny (current and former employees of Brigham and Women's Hospital) related to the development of DELTA are assigned to the institution, and Partners Healthcare, the parent organization of Brigham and Women's Hospital, retains partial rights for DELTA use and licensing. Neither Dr Resnic nor Dr Matheny have previously received or anticipate any financial compensation, stock options, income, or ownership in DELTA, Coping Systems, or any other related commercial entity. Dr Resnic reported receiving modest consulting income from Boston Scientific Inc, Abbott Vascular Inc, Cordis Corp, and St Jude Medical Inc and receiving research grants from The Medicines Company. No other authors reported dislosures.

Funding/Support: This study was funded, in part, by grants from the National Library of Medicine (NIH R01-LM008142) and the US Food and Drug Administration (HHSF 223200830058C) as well as from the Veterans Administration Health Services Research and Development Service (CDP 09-387).

Role of the Sponsors: The National Institutes of Health, the Veterans Administration Health Services Research and Development Service, and the US Food and Drug Administration had no role in the design and conduct of the study; the collection, management, analysis, and interpretation of the data; or the preparation, review, or approval of the manuscript. However, the preliminary proposed study protocol was provided to the Veterans Administration Health Services Research and Development Service and the US Food and Drug Administration as part of the original grant and contract application process.

Disclaimer: The opinions and assertions presented in this article are the private views of the authors and are not to be construed as conveying either official endorsement or criticism by the US Department of Health and Human Services, the Public Health Service, or the US Food and Drug Administration.

Additional Contributions: We acknowledge the significant system development and technical contributions of Richard Cope, BS, and Susan Robbins, BS, of Coping Systems Inc, Andover, Massachusetts. Coping Systems was a subcontractor to Brigham and Women's Hospital, and neither Mr Cope nor Ms Robbins received extra compensation for their contributions; Mr Cope is part owner of Coping Systems and is also named on the patent application filed by Dr Resnic and Dr Matheny. We also acknowledge the support of the Division of Healthcare Quality and Statistics, Massachusetts Department of Public Health.

Gross TP, Kessler LG. Medical device vigilance at FDA.  Stud Health Technol Inform. 1996;28:17-24
PubMed
Samore MH, Evans RS, Lassen A,  et al.  Surveillance of medical device−related hazards and adverse events in hospitalized patients.  JAMA. 2004;291(3):325-334
PubMed   |  Link to Article
Maisel WH. Unanswered questions—drug-eluting stents and the risk of late thrombosis.  N Engl J Med. 2007;356(10):981-984
PubMed   |  Link to Article
Shah JS, Maisel WH. Recalls and safety alerts affecting automated external defibrillators.  JAMA. 2006;296(6):655-660
PubMed   |  Link to Article
Rosen CJ. The rosiglitazone story—lessons from an FDA Advisory Committee meeting.  N Engl J Med. 2007;357(9):844-846
PubMed   |  Link to Article
Garber AM. Modernizing device regulation.  N Engl J Med. 2010;362(13):1161-1163
PubMed   |  Link to Article
Platt R, Wilson M, Chan KA, Benner JS, Marchibroda J, McClellan M. The new Sentinel Network—improving the evidence of medical-product safety.  N Engl J Med. 2009;361(7):645-647
PubMed   |  Link to Article
Holman WL, Pae WE, Teutenberg JJ,  et al.   INTERMACS: interval analysis of registry data.  J Am Coll Surg. 2009;208(5):755-762
PubMed   |  Link to Article
Brindis RG, Fitzgerald S, Anderson HV, Shaw RE, Weintraub WS, Williams JF. The American College of Cardiology–National Cardiovascular Data Registry (ACC-NCDR): building a national clinical data repository.  J Am Coll Cardiol. 2001;37(8):2240-2245
PubMed   |  Link to Article
Aggarwal A, Wang Y, Rumsfeld JS, Curtis JP, Heidenreich PA.National Cardiovascular Data Registry.  Clinical characteristics and in-hospital outcome of patients with end-stage renal disease on dialysis referred for implantable cardioverter-defibrillator implantation.  Heart Rhythm. 2009;6(11):1565-1571
PubMed   |  Link to Article
Hammill S, Phurrough S, Brindis R. The National ICD Registry: now and into the future.  Heart Rhythm. 2006;3(4):470-473
PubMed   |  Link to Article
Matheny ME, Arora N, Ohno-Machado L, Resnic FS. Rare adverse event monitoring of medical devices with the use of an automated surveillance tool.  AMIA Annu Symp Proc. 2007;518-522
PubMed
Mauri L, Silbaugh TS, Wolf RE,  et al.  Long-term clinical outcomes after drug-eluting and bare-metal stenting in Massachusetts.  Circulation. 2008;118(18):1817-1827
PubMed   |  Link to Article
 Percutaeous coronary intervention in the Commonwealth of Massachusetts: fiscal year 2007 report, October 1, 2006-September 30, 2007. Massachusetts Data Analysis Center Web site. http://www.massdac.org/sites/default/files/reports/PCI%20FY2007.pdf. 2009. Accessed January 19, 2010
Mauri L, Silbaugh TS, Garg P,  et al.  Drug-eluting or bare-metal stents for acute myocardial infarction.  N Engl J Med. 2008;359(13):1330-1342
PubMed   |  Link to Article
 Description of NCDR CathPCI Registry. National Cardiovascular Data Registry Web site. https://www.ncdr.com/webncdr/DefaultCathPCI.aspx. 2010. Accessed January 4, 2010
Resnic FS, Zou KH, Do DV, Apostolakis G, Ohno-Machado L. Exploration of a Bayesian updating methodology to monitor the safety of interventional cardiovascular procedures.  Med Decis Making. 2004;24(4):399-407
PubMed   |  Link to Article
Matheny ME, Morrow DA, Ohno-Machado L, Cannon CP, Sabatine MS, Resnic FS. Validation of an automated safety surveillance system with prospective, randomized trial data.  Med Decis Making. 2009;29(2):247-256
PubMed   |  Link to Article
Matheny ME, Ohno-Machado L, Resnic FS. Risk-adjusted sequential probability ratio test control chart methods for monitoring operator and institutional mortality rates in interventional cardiology.  Am Heart J. 2008;155(1):114-120
PubMed   |  Link to Article
Matheny ME, Ohno-Machado L, Resnic FS. Monitoring device safety in interventional cardiology.  J Am Med Inform Assoc. 2006;13(2):180-187
PubMed   |  Link to Article
Austin PC. Some methods of propensity-score matching had superior performance to others: results of an empirical investigation and Monte Carlo simulations.  Biom J. 2009;51(1):171-184
PubMed   |  Link to Article
Austin PC. The relative ability of different propensity score methods to balance measured covariates between treated and untreated subjects in observational studies.  Med Decis Making. 2009;29(6):661-677
PubMed   |  Link to Article
Stone GW, Ellis SG, Cannon L,  et al.  Comparison of a polymer-based paclitaxel-eluting stent with a bare metal stent in patients with complex coronary artery disease.  JAMA. 2005;294(10):1215-1223
PubMed   |  Link to Article
Popma JJ, Mauri L, O’Shaughnessy C,  et al.  Frequency and clinical consequences associated with sidebranch occlusion during stent implantation using zotarolimus-eluting and paclitaxel-eluting coronary stents.  Circ Cardiovasc Interv. 2009;2(2):133-139
PubMed   |  Link to Article
Nikolsky E, Mehran R, Halkin A,  et al.  Vascular complications associated with arteriotomy closure devices in patients undergoing percutaneous coronary procedures: a meta-analysis.  J Am Coll Cardiol. 2004;44(6):1200-1209
PubMed
Koreny M, Riedmüller E, Nikfardjam M, Siostrzonek P, Müllner M. Arterial puncture closing devices compared with standard manual compression after cardiac catheterization.  JAMA. 2004;291(3):350-357
PubMed   |  Link to Article
Tavris DR, Dey S, Albrecht-Gallauresi B,  et al.  Risk of local adverse events following cardiac catheterization by hemostasis device use—phase II.  J Invasive Cardiol. 2005;17(12):644-650
PubMed
Warren BS, Warren SG, Miller SD. Predictors of complications and learning curve using the Angio-Seal closure device following interventional and diagnostic catheterization.  Catheter Cardiovasc Interv. 1999;48(2):162-166
PubMed   |  Link to Article
Lincoff AM, Bittl JA, Harrington RA,  et al.  Bivalirudin and provisional glycoprotein IIb/IIIa blockade compared with heparin and planned glycoprotein IIb/IIIa blockade during percutaneous coronary intervention.   JAMA. 2003;289(7):853-863
PubMed   |  Link to Article
Stone GW, Ware JH, Bertrand ME,  et al.  Antithrombotic strategies in patients with acute coronary syndromes undergoing early invasive management.  JAMA. 2007;298(21):2497-2506
PubMed   |  Link to Article
O’Shea JC, Kramer JM, Califf RM, Peterson ED. Identifying holes in the safety net.  Am Heart J. 2004;147(6):977-984
PubMed   |  Link to Article
Peterson ED, Hirshfeld JW Jr, Ferguson TB, Kramer JM, Califf RM, Kessler LG. Sealing holes in the safety net.  Am Heart J. 2004;147(6):985-990
PubMed   |  Link to Article
Rao SV, Califf RM, Kramer JM,  et al.  Postmarket evaluation of breakthrough technologies.  Am Heart J. 2008;156(2):201-208
PubMed   |  Link to Article
 FDA's Sentinel Initiative. US Food and Drug Administration Web site. http://www.fda.gov/Safety/FDAsSentinelInitiative/default.htm. 2010. Accessed May 3, 2010

Figures

Place holder to copy figure label and caption
Figure 1. Summary Safety Analysis of the Taxus Express2 Drug-Eluting Stent
Graphic Jump Location

Longitudinal propensity-matched analysis of cumulative incidence following implantation of at least 1 stent. Solid blue lines indicate mean event rates of matched controls receiving alternative drug-eluting stents; blue shaded areas, 95% confidence intervals (CIs) corrected for multiple comparisons; dashed lines, the uncorrected 95% CI bounds. Circle sizes are proportional to the number of Taxus Express2 stents used in Massachusetts in a calendar quarter. Light blue circles indicate cumulative observed event rates within propensity-matched expectations; dark blue circles, higher than expected rates (safety alerts), ie, observed rates exceeding the upper 95% CI boundaries; solid black lines, cumulative sample sizes.

Place holder to copy figure label and caption
Figure 2. Summary Safety Analysis of the Angio-Seal STS Vascular Closure Device
Graphic Jump Location

Longitudinal propensity-matched analysis of cumulative incidence following implantation of at least 1 device. Solid blue lines indicate mean event rates of matched controls receiving alternative vascular closure devices; blue shaded areas, 95% confidence intervals (CIs) corrected for multiple comparisons; dashed lines, the uncorrected 95% CI bounds. Circle sizes are proportional to the number of Angio-Seal STS devices used in Massachusetts in a calendar quarter. Light blue circles indicate cumulative observed event rates within propensity-matched expectations; dark blue circles, higher than expected rates (safety alerts), ie, observed rates exceeding the upper 95% CI boundaries; open circles, lower than expected rates, ie, observed rates less than the lower 95% CI boundaries; solid black lines, cumulative sample sizes.

Tables

Table Graphic Jump LocationTable 1. Medical Devices Analyzed, Outcomes Captured, Control Populations, and Final Alert Status for Each Analysis
Table Graphic Jump LocationTable 2. Distribution of Clinical Covariates in Patients Receiving Taxus Express2 or Alternative Drug-Eluting Stents
Table Graphic Jump LocationTable 3. Distribution of Clinical Covariates in Patients Receiving Angio-Seal STS or Alternative Vascular Closure Devices

References

Gross TP, Kessler LG. Medical device vigilance at FDA.  Stud Health Technol Inform. 1996;28:17-24
PubMed
Samore MH, Evans RS, Lassen A,  et al.  Surveillance of medical device−related hazards and adverse events in hospitalized patients.  JAMA. 2004;291(3):325-334
PubMed   |  Link to Article
Maisel WH. Unanswered questions—drug-eluting stents and the risk of late thrombosis.  N Engl J Med. 2007;356(10):981-984
PubMed   |  Link to Article
Shah JS, Maisel WH. Recalls and safety alerts affecting automated external defibrillators.  JAMA. 2006;296(6):655-660
PubMed   |  Link to Article
Rosen CJ. The rosiglitazone story—lessons from an FDA Advisory Committee meeting.  N Engl J Med. 2007;357(9):844-846
PubMed   |  Link to Article
Garber AM. Modernizing device regulation.  N Engl J Med. 2010;362(13):1161-1163
PubMed   |  Link to Article
Platt R, Wilson M, Chan KA, Benner JS, Marchibroda J, McClellan M. The new Sentinel Network—improving the evidence of medical-product safety.  N Engl J Med. 2009;361(7):645-647
PubMed   |  Link to Article
Holman WL, Pae WE, Teutenberg JJ,  et al.   INTERMACS: interval analysis of registry data.  J Am Coll Surg. 2009;208(5):755-762
PubMed   |  Link to Article
Brindis RG, Fitzgerald S, Anderson HV, Shaw RE, Weintraub WS, Williams JF. The American College of Cardiology–National Cardiovascular Data Registry (ACC-NCDR): building a national clinical data repository.  J Am Coll Cardiol. 2001;37(8):2240-2245
PubMed   |  Link to Article
Aggarwal A, Wang Y, Rumsfeld JS, Curtis JP, Heidenreich PA.National Cardiovascular Data Registry.  Clinical characteristics and in-hospital outcome of patients with end-stage renal disease on dialysis referred for implantable cardioverter-defibrillator implantation.  Heart Rhythm. 2009;6(11):1565-1571
PubMed   |  Link to Article
Hammill S, Phurrough S, Brindis R. The National ICD Registry: now and into the future.  Heart Rhythm. 2006;3(4):470-473
PubMed   |  Link to Article
Matheny ME, Arora N, Ohno-Machado L, Resnic FS. Rare adverse event monitoring of medical devices with the use of an automated surveillance tool.  AMIA Annu Symp Proc. 2007;518-522
PubMed
Mauri L, Silbaugh TS, Wolf RE,  et al.  Long-term clinical outcomes after drug-eluting and bare-metal stenting in Massachusetts.  Circulation. 2008;118(18):1817-1827
PubMed   |  Link to Article
 Percutaeous coronary intervention in the Commonwealth of Massachusetts: fiscal year 2007 report, October 1, 2006-September 30, 2007. Massachusetts Data Analysis Center Web site. http://www.massdac.org/sites/default/files/reports/PCI%20FY2007.pdf. 2009. Accessed January 19, 2010
Mauri L, Silbaugh TS, Garg P,  et al.  Drug-eluting or bare-metal stents for acute myocardial infarction.  N Engl J Med. 2008;359(13):1330-1342
PubMed   |  Link to Article
 Description of NCDR CathPCI Registry. National Cardiovascular Data Registry Web site. https://www.ncdr.com/webncdr/DefaultCathPCI.aspx. 2010. Accessed January 4, 2010
Resnic FS, Zou KH, Do DV, Apostolakis G, Ohno-Machado L. Exploration of a Bayesian updating methodology to monitor the safety of interventional cardiovascular procedures.  Med Decis Making. 2004;24(4):399-407
PubMed   |  Link to Article
Matheny ME, Morrow DA, Ohno-Machado L, Cannon CP, Sabatine MS, Resnic FS. Validation of an automated safety surveillance system with prospective, randomized trial data.  Med Decis Making. 2009;29(2):247-256
PubMed   |  Link to Article
Matheny ME, Ohno-Machado L, Resnic FS. Risk-adjusted sequential probability ratio test control chart methods for monitoring operator and institutional mortality rates in interventional cardiology.  Am Heart J. 2008;155(1):114-120
PubMed   |  Link to Article
Matheny ME, Ohno-Machado L, Resnic FS. Monitoring device safety in interventional cardiology.  J Am Med Inform Assoc. 2006;13(2):180-187
PubMed   |  Link to Article
Austin PC. Some methods of propensity-score matching had superior performance to others: results of an empirical investigation and Monte Carlo simulations.  Biom J. 2009;51(1):171-184
PubMed   |  Link to Article
Austin PC. The relative ability of different propensity score methods to balance measured covariates between treated and untreated subjects in observational studies.  Med Decis Making. 2009;29(6):661-677
PubMed   |  Link to Article
Stone GW, Ellis SG, Cannon L,  et al.  Comparison of a polymer-based paclitaxel-eluting stent with a bare metal stent in patients with complex coronary artery disease.  JAMA. 2005;294(10):1215-1223
PubMed   |  Link to Article
Popma JJ, Mauri L, O’Shaughnessy C,  et al.  Frequency and clinical consequences associated with sidebranch occlusion during stent implantation using zotarolimus-eluting and paclitaxel-eluting coronary stents.  Circ Cardiovasc Interv. 2009;2(2):133-139
PubMed   |  Link to Article
Nikolsky E, Mehran R, Halkin A,  et al.  Vascular complications associated with arteriotomy closure devices in patients undergoing percutaneous coronary procedures: a meta-analysis.  J Am Coll Cardiol. 2004;44(6):1200-1209
PubMed
Koreny M, Riedmüller E, Nikfardjam M, Siostrzonek P, Müllner M. Arterial puncture closing devices compared with standard manual compression after cardiac catheterization.  JAMA. 2004;291(3):350-357
PubMed   |  Link to Article
Tavris DR, Dey S, Albrecht-Gallauresi B,  et al.  Risk of local adverse events following cardiac catheterization by hemostasis device use—phase II.  J Invasive Cardiol. 2005;17(12):644-650
PubMed
Warren BS, Warren SG, Miller SD. Predictors of complications and learning curve using the Angio-Seal closure device following interventional and diagnostic catheterization.  Catheter Cardiovasc Interv. 1999;48(2):162-166
PubMed   |  Link to Article
Lincoff AM, Bittl JA, Harrington RA,  et al.  Bivalirudin and provisional glycoprotein IIb/IIIa blockade compared with heparin and planned glycoprotein IIb/IIIa blockade during percutaneous coronary intervention.   JAMA. 2003;289(7):853-863
PubMed   |  Link to Article
Stone GW, Ware JH, Bertrand ME,  et al.  Antithrombotic strategies in patients with acute coronary syndromes undergoing early invasive management.  JAMA. 2007;298(21):2497-2506
PubMed   |  Link to Article
O’Shea JC, Kramer JM, Califf RM, Peterson ED. Identifying holes in the safety net.  Am Heart J. 2004;147(6):977-984
PubMed   |  Link to Article
Peterson ED, Hirshfeld JW Jr, Ferguson TB, Kramer JM, Califf RM, Kessler LG. Sealing holes in the safety net.  Am Heart J. 2004;147(6):985-990
PubMed   |  Link to Article
Rao SV, Califf RM, Kramer JM,  et al.  Postmarket evaluation of breakthrough technologies.  Am Heart J. 2008;156(2):201-208
PubMed   |  Link to Article
 FDA's Sentinel Initiative. US Food and Drug Administration Web site. http://www.fda.gov/Safety/FDAsSentinelInitiative/default.htm. 2010. Accessed May 3, 2010
CME
Also Meets CME requirements for:
Browse CME for all U.S. States
Accreditation Information
The American Medical Association is accredited by the Accreditation Council for Continuing Medical Education to provide continuing medical education for physicians. The AMA designates this journal-based CME activity for a maximum of 1 AMA PRA Category 1 CreditTM per course. Physicians should claim only the credit commensurate with the extent of their participation in the activity. Physicians who complete the CME course and score at least 80% correct on the quiz are eligible for AMA PRA Category 1 CreditTM.
Note: You must get at least of the answers correct to pass this quiz.
Please click the checkbox indicating that you have read the full article in order to submit your answers.
Your answers have been saved for later.
You have not filled in all the answers to complete this quiz
The following questions were not answered:
Sorry, you have unsuccessfully completed this CME quiz with a score of
The following questions were not answered correctly:
Commitment to Change (optional):
Indicate what change(s) you will implement in your practice, if any, based on this CME course.
Your quiz results:
The filled radio buttons indicate your responses. The preferred responses are highlighted
For CME Course: A Proposed Model for Initial Assessment and Management of Acute Heart Failure Syndromes
Indicate what changes(s) you will implement in your practice, if any, based on this CME course.

Multimedia

Data Supplement
Supplemental Content

Some tools below are only available to our subscribers or users with an online account.

Related Content

Customize your page view by dragging & repositioning the boxes below.

See Also...
Articles Related By Topic
Related Collections