0
We're unable to sign you in at this time. Please try again in a few minutes.
Retry
We were able to sign you in, but your subscription(s) could not be found. Please try again in a few minutes.
Retry
There may be a problem with your account. Please contact the AMA Service Center to resolve this issue.
Contact the AMA Service Center:
Telephone: 1 (800) 262-2350 or 1 (312) 670-7827  *   Email: subscriptions@jamanetwork.com
Error Message ......
From the Centers for Disease Control and Prevention |

Assessment of ESSENCE Performance for Influenza-Like Illness Surveillance After an Influenza Outbreak—U.S. Air Force Academy, Colorado, 2009 FREE

JAMA. 2011;305(18):1851-1853. doi:.
Text Size: A A A
Published online

MMWR. 2011;60:406-409

1 table omitted

The Electronic Surveillance System for the Early Notification of Community-Based Epidemics (ESSENCE), version II, designed by the Johns Hopkins University Applied Physics Laboratory and the U.S. Department of Defense (DoD), is an Internet-based syndromic disease surveillance system used by civilian and military health departments.1 ESSENCE was designed to increase the timeliness of outbreak detection, serving as an early warning system and providing opportunities to prevent and control the spread of infection. After a 2009 pandemic influenza A (H1N1) outbreak at the U.S. Air Force (USAF) Academy in Colorado, CDC was invited to conduct an evaluation of the ESSENCE influenza-like illness (ILI) surveillance system to assess its performance during the outbreak.2,3 Medical records at the USAF Academy clinics from June 25 through July 8, 2009, the period of the outbreak, were reviewed. This report summarizes the results of the evaluation, which demonstrated strengths in data quality, flexibility, and representativeness; however, ESSENCE was not useful for detecting or monitoring the H1N1 outbreak because of its lack of timeliness (1-3 day delay), inadequate sensitivity (71.4%), and poor predictive value positive (PVP) (31.8%) for identifying ILI cases. In this localized, single-source outbreak, ESSENCE did not serve as an early warning system for an emerging infectious disease and did not detect the outbreak soon enough to institute prevention and control measures that might have slowed the spread of infection. More frequent Internet data transmissions from the clinics to the ESSENCE server could improve timeliness, and PVP could be enhanced by including measured body temperature in the ESSENCE ILI case definition.

The utility of syndromic disease surveillance for early outbreak detection and improvement of public health response remains controversial.47 A survey of U.S. health departments indicated that the most common application for syndromic surveillance was to monitor the start and stop of the annual influenza season, but that it was less useful for local outbreak detection.4 Other studies found that syndromic surveillance has been useful to identify localized respiratory, dermatologic, and gastrointestinal disease outbreaks.57

During June 25–July 24, 2009, an H1N1 outbreak occurred at the USAF Academy in Colorado, with 134 cases confirmed among a population of USAF 1,376 basic cadet trainees.3 Although ESSENCE is used at the USAF Academy, public health officials became aware of the outbreak before ESSENCE indicated the increase in ILI cases. After this large outbreak, the USAF Academy invited CDC to evaluate ESSENCE as an ILI surveillance system.

The U.S. military has used ESSENCE since 2003 to detect and monitor disease outbreaks. DoD provides an annual budget for system maintenance and development and releases updated versions of the system as surveillance needs change. ESSENCE identifies patients based on provider-assigned International Classification of Diseases, Ninth Revision, Clinical Modification (ICD-9-CM) codes that are entered into the electronic medical record.1 The ESSENCE ICD-9-CM code set for ILI* was established based on an analysis of medical records and respiratory specimens to determine the codes that most accurately represent ILI.8,9 By regular secure data transmissions via the Internet, the ESSENCE server identifies new cases and, using temporal algorithms that predict expected daily fluctuation, determines whether an increasing trend has occurred, indicating a possible outbreak.1,8,9 Raw data and aggregate reports, in the form of line graphs, are available via a password-protected ESSENCE website. An increase in syndromic cases above predicted thresholds is highlighted as a color-coded alert. Privacy and confidentiality are maintained with patient identification numbers and annual information protection training for users. In 2003, using past military and civilian data, an initial evaluation determined that ESSENCE detected eight of eight respiratory disease outbreaks within an average of 1 day after the event.1

CDC's Updated Guidelines for Evaluating Public Health Surveillance Systems was used to assess the usefulness, simplicity, flexibility, data quality, acceptability, representativeness, timeliness, stability, sensitivity, and PVP of ESSENCE for the USAF.2 To determine sensitivity and PVP, medical record data from ILI case-patient visits at the USAF Academy acute care and cadet clinics during June 25–July 8, 2009, were collected. For the medical record review, ILI was defined as measured temperature ≥100.0°F (≥37.8°C) and cough or sore throat. Medical record data collection included cough, sore throat, measured temperature, and the results of respiratory disease laboratory tests for influenza A, influenza B, H1N1, adenovirus, and group A streptococcus bacteria. Patients from the same period were identified on the ESSENCE ILI website to collect the ICD-9-CM codes. Sensitivity and PVP were calculated using (1) medical record—confirmed ILI, as defined and (2) laboratory confirmation of a respiratory infection, as criterion standards. USAF Academy, USAF School of Aerospace Medicine (USAFSAM), DoD, and CDC staff members who used ESSENCE daily were interviewed to assess the remaining evaluation criteria.

This evaluation found that the usefulness of ESSENCE varied by user. CDC, which used the aggregate USAF and DoD data, found ESSENCE useful to monitor national syndromic disease activity, and USAFSAM staff members found it useful to monitor disease activity at each base. The USAF Academy indicated ESSENCE was useful to monitor the local influenza season and determine syndromic baselines. Although users investigated ESSENCE alerts and worrying trends, most alerts and trends were time-consuming false alarms that revealed normal disease variations.

Regarding simplicity of operations and structure, the ESSENCE website's aggregate reports, line graphs, and color-coded alerts were easy to comprehend. ESSENCE's flexibility to adapt was demonstrated by the updated versions released by DoD based on user feedback and changes in surveillance needs, and the raw data query functions available to ESSENCE users. Data quality, or the completeness and validity of the data, was established by extracting demographic and medical information from official DoD systems, and scheduling automated batched data transmissions to the ESSENCE server at night, during periods of lower Internet usage to reduce transmission interruptions. For acceptability, or the willingness to participate in the surveillance system, data transmission to the ESSENCE server was automated, and personnel at the Academy and USAFSAM had passwords, although cumbersome to attain and maintain, to use ESSENCE in their daily operations. ESSENCE was determined to have a high degree of representativeness, in that it included all DoD beneficiaries visiting all USAF outpatient clinics, and thus it reported all medical events. Timeliness, or the time of the clinic visit to the time the information appeared on the ESSENCE website, was 1-3 days.10 Stability, or the reliability and availability of the system, was maintained by the annual DOD budget and the infrequent occasions when ESSENCE was unavailable to the user.

Of the 540 medical records reviewed to assess sensitivity and PVP, 189 had a laboratory test result. Compared with medical record—confirmed ILI, ESSENCE ILI sensitivity was calculated at 71.4% and PVP at 31.8%. Compared with laboratory-confirmed respiratory infections, ESSENCE ILI sensitivity and PVP were 78.6% and 49.5%, respectively. When the evaluators added a documented, measured body temperature of ≥100.0°F (≥37.8°C) to the ESSENCE ILI case definition, the ESSENCE ILI sensitivity, compared with medical records, remained the same, but PVP increased to 95.5%; however, when compared with laboratory confirmation, ILI sensitivity was 65.7%, and PVP was 66.7%.

Reported by: C Witkop, MD, USAF Academy, Colorado. M Duffy, DVM, USAF School of Aerospace Medicine, Wright-Patterson Air Force Base, Ohio. L Cohen, MD, Scientific Education and Professional Development Program Office, Office of Surveillance, Epidemiology, and Laboratory Svcs; D Fishbein, MD, Div of Global Migration and Quarantine, National Center for Emerging and Zoonotic Infectious Diseases; M Selent,* DVM, EIS Officer, CDC. *Corresponding contributor: Monica Selent, Div of Global Migration and Quarantine, National Center for Emerging and Zoonotic Infectious Disease, CDC, 404-520-2332, mselent@cdc.gov.

CDC Editorial Note: This evaluation found that the major strength of ESSENCE ILI surveillance was its usefulness for monitoring annual seasonal influenza activity. Other strengths included simplicity, flexibility, data quality, representativeness, and stability. Weaknesses included low PVP, lack of timeliness, and limited usefulness to detect and monitor an ILI outbreak.

Retrospectively, ESSENCE showed an increasing ILI trend 2-4 days before an Academy mass gathering; however, the combination of the ESSENCE time delay, occurrence of the gathering over a holiday weekend, and short incubation period of H1N1 meant the increasing trend was not detected in time to institute preventive measures. To improve timeliness, medical data transmission could be scheduled in smaller, more frequent batches throughout the day so that changing trends would appear on the ESSENCE website sooner.

After USAFSAM judged that the ILI PVP of the surveillance system was too low to distinguish actual outbreaks, the addition of a measured body temperature ≥100.0°F (≥37.8°C) to the ESSENCE case definition was evaluated to determine whether PVP, and potentially sensitivity, could be improved. Compared with medical record—confirmed ILI and laboratory-confirmed respiratory infections as criterion standards, PVP did increase with the addition. An independent study using only laboratory confirmation and ICD-9-CM—based ILI surveillance also found that PVP increased by adding measured body temperature.10 The large PVP increase with medical record confirmation was attributed to the low number of ESSENCE ILI cases with an elevated temperature at the clinic, potentially resulting from antipyretic use or actual afebrile infection. When compared with laboratory-confirmed respiratory infections, sensitivity decreased. However, with medical record confirmation, sensitivity stayed the same, because all the ILI cases still had an elevated temperature, per the medical record case definition, and an ILI ICD-9-CM code, per the ESSENCE case definition. Despite the improvement in PVP by adding a measured body temperature to the ESSENCE case definition, the potential loss in sensitivity might reduce the ability to detect actual ILI outbreaks. Users need to determine if this loss is acceptable for their purposes.

The findings in this report are subject to at least three limitations. First, as new data arrive in ESSENCE, the web page does not record the date additional case-patients appeared. Therefore, evaluators could only estimate when ESSENCE issued an alert to the ILI outbreak, based on historical documentation. Second, this evaluation collected data from only one outbreak at one USAF base. Additional outbreak analyses from other USAF bases are needed to judge the effectiveness of ESSENCE as an early-warning outbreak system for the USAF. Finally, the results of this evaluation are not generalizable to the other military services or civilian public health agencies, which might use ESSENCE differently.

This evaluation showed that, despite strengths in data quality, flexibility, and representativeness, ESSENCE did not serve as an early warning system for an emerging infectious disease during a localized, single-source outbreak, and did not detect the outbreak soon enough to allow prevention and control measures to be instituted. For enhanced outbreak detection and monitoring, more frequent Internet data transmissions would improve ESSENCE's timeliness. Additionally, the inclusion of measured body temperature in the ESSENCE ILI case definition could improve PVP, but with a possible loss in sensitivity resulting from exclusion of afebrile cases. As the strengths, weaknesses, and limitations of ILI surveillance as an early warning system for emerging infectious disease become better understood, future development should investigate how informatics and information technology can overcome ILI surveillance weaknesses.

ACKNOWLEDGMENTS

This report is based, in part, on contributions by K Cox, MD, US Army Center for Health Promotion and Preventive Medicine; J Collins, R Devine, A Cox, A Owens, M Green, US Air Force Academy; and C Hales, PhD, N Molinari, PhD, N Megateli-Das, MS, P Szymanowski, MPH, C Adams, and J Herrera, CDC.

What is already known on this topic?

The Electronic Surveillance System for the Early Notification of Community-Based Epidemics (ESSENCE), version II, is an effective syndromic disease surveillance system to determine the normal seasonal variation of influenza-like illness (ILI) on a military installation but might not be effective for early detection and monitoring of ILI outbreaks.

What is added by this report?

ESSENCE's strengths are data quality, flexibility, and representativeness, but it did not sufficiently detect or monitor the H1N1 outbreak because of its lack of timeliness (1-3 day delay), inadequate sensitivity (71.4%), and poor predictive value positive (PVP) (31.8%) for identifying ILI cases. PVP could be improved by introducing a measured body temperature to the ESSENCE case definition.

What are the implications for public health practice?

More frequent batch data transmissions to shorten the time between patient visits and generation of alerts could enhance ESSENCE's usefulness for detecting and monitoring an actual outbreak.

*The ESSENCE ICD-9-CM codes for ILI include the following: 079.99 viral infection, not otherwise specified; 382.9 otitis media, not otherwise specified; 460 acute nasopharyngitis; 461.9 acute sinusitis, not otherwise specified; 465.9 acute upper respiratory infection, not otherwise specified; 466.0 acute bronchitis; 486 pneumonia, organism not otherwise specified; 490 bronchitis, not otherwise specified; 780.6 fever; 780.60 fever, unspecified; 780.64 chills (without fever); 786.2 cough.

REFERENCES

Lombardo JS, Burkom H, Pavlin J.CDC.  ESSENCE II and the framework for evaluating syndromic surveillance systems.  MMWR Morb Mortal Wkly Rep. 2004;53:(Suppl)  159-165
PubMed
German RR, Lee LM, Horan JM, Milstein RL, Pertowski CA, Waller MN.Guidelines Working Group Centers for Disease Control and Prevention (CDC).  Updated guidelines for evaluating public health surveillance systems: recommendations from the Guidelines Working Group.  MMWR Recomm Rep. 2001;50(RR-13):1-35
PubMed
Witkop CT, Duffy MR, Macias EA,  et al.  Novel Influenza A (H1N1) outbreak at the U.S. Air Force Academy: epidemiology and viral shedding duration.  Am J Prev Med. 2010;38(2):121-126
PubMed   |  Link to Article
Buehler JW, Sonticker A, Paladini M, Soper P, Mostashari F. Syndromic surveillance practices in the United States: findings from a survey of state, territorial, and selected local health departments.  Adv Dis Surveill. 2008;6:1-20
Ang BCH, Chen MIC, Goh TLH, Ng YY, Fan SW. An assessment of electronically captured data in the patient care enhancement system (PACES) for syndromic surveillance.  Ann Acad Med Singapore. 2005;34(9):539-4
PubMed
Ritzwoller DP, Kleinman K, Palen T,  et al; CDC.  Comparison of syndromic surveillance and a sentinel provider system in detecting an influenza outbreak—Denver, Colorado, 2003.  MMWR Morb Mortal Wkly Rep. 2005;54:(Suppl)  151-156
PubMed
Lewis MD, Pavlin JA, Mansfield JL,  et al.  Disease outbreak detection system using syndromic data in the greater Washington DC area.  Am J Prev Med. 2002;23(3):180-186
PubMed   |  Link to Article
Marsden-Haug N, Foster VB, Gould PL, Elbert E, Wang H, Pavlin JA. Code-based syndromic surveillance for influenzalike illness by International Classification of Diseases, Ninth Revision.  Emerg Infect Dis. 2007;13(2):207-216
PubMed   |  Link to Article
Betancourt JA, Hakre S, Polyak CS, Pavlin JA. Evaluation of ICD-9 codes for syndromic surveillance in the electronic surveillance system for the early notification of community-based epidemics.  Mil Med. 2007;172(4):346-352
PubMed
Pattie DC, Atherton MJ, Cox KL. Electronic influenza monitoring: evaluation of body temperature to classify influenza-like illness in a syndromic surveillance system.  Qual Manag Health Care. 2009;18(2):91-102
PubMed   |  Link to Article

Figures

Tables

References

Lombardo JS, Burkom H, Pavlin J.CDC.  ESSENCE II and the framework for evaluating syndromic surveillance systems.  MMWR Morb Mortal Wkly Rep. 2004;53:(Suppl)  159-165
PubMed
German RR, Lee LM, Horan JM, Milstein RL, Pertowski CA, Waller MN.Guidelines Working Group Centers for Disease Control and Prevention (CDC).  Updated guidelines for evaluating public health surveillance systems: recommendations from the Guidelines Working Group.  MMWR Recomm Rep. 2001;50(RR-13):1-35
PubMed
Witkop CT, Duffy MR, Macias EA,  et al.  Novel Influenza A (H1N1) outbreak at the U.S. Air Force Academy: epidemiology and viral shedding duration.  Am J Prev Med. 2010;38(2):121-126
PubMed   |  Link to Article
Buehler JW, Sonticker A, Paladini M, Soper P, Mostashari F. Syndromic surveillance practices in the United States: findings from a survey of state, territorial, and selected local health departments.  Adv Dis Surveill. 2008;6:1-20
Ang BCH, Chen MIC, Goh TLH, Ng YY, Fan SW. An assessment of electronically captured data in the patient care enhancement system (PACES) for syndromic surveillance.  Ann Acad Med Singapore. 2005;34(9):539-4
PubMed
Ritzwoller DP, Kleinman K, Palen T,  et al; CDC.  Comparison of syndromic surveillance and a sentinel provider system in detecting an influenza outbreak—Denver, Colorado, 2003.  MMWR Morb Mortal Wkly Rep. 2005;54:(Suppl)  151-156
PubMed
Lewis MD, Pavlin JA, Mansfield JL,  et al.  Disease outbreak detection system using syndromic data in the greater Washington DC area.  Am J Prev Med. 2002;23(3):180-186
PubMed   |  Link to Article
Marsden-Haug N, Foster VB, Gould PL, Elbert E, Wang H, Pavlin JA. Code-based syndromic surveillance for influenzalike illness by International Classification of Diseases, Ninth Revision.  Emerg Infect Dis. 2007;13(2):207-216
PubMed   |  Link to Article
Betancourt JA, Hakre S, Polyak CS, Pavlin JA. Evaluation of ICD-9 codes for syndromic surveillance in the electronic surveillance system for the early notification of community-based epidemics.  Mil Med. 2007;172(4):346-352
PubMed
Pattie DC, Atherton MJ, Cox KL. Electronic influenza monitoring: evaluation of body temperature to classify influenza-like illness in a syndromic surveillance system.  Qual Manag Health Care. 2009;18(2):91-102
PubMed   |  Link to Article
CME
Also Meets CME requirements for:
Browse CME for all U.S. States
Accreditation Information
The American Medical Association is accredited by the Accreditation Council for Continuing Medical Education to provide continuing medical education for physicians. The AMA designates this journal-based CME activity for a maximum of 1 AMA PRA Category 1 CreditTM per course. Physicians should claim only the credit commensurate with the extent of their participation in the activity. Physicians who complete the CME course and score at least 80% correct on the quiz are eligible for AMA PRA Category 1 CreditTM.
Note: You must get at least of the answers correct to pass this quiz.
Please click the checkbox indicating that you have read the full article in order to submit your answers.
Your answers have been saved for later.
You have not filled in all the answers to complete this quiz
The following questions were not answered:
Sorry, you have unsuccessfully completed this CME quiz with a score of
The following questions were not answered correctly:
Commitment to Change (optional):
Indicate what change(s) you will implement in your practice, if any, based on this CME course.
Your quiz results:
The filled radio buttons indicate your responses. The preferred responses are highlighted
For CME Course: A Proposed Model for Initial Assessment and Management of Acute Heart Failure Syndromes
Indicate what changes(s) you will implement in your practice, if any, based on this CME course.

Multimedia

Some tools below are only available to our subscribers or users with an online account.

Related Content

Customize your page view by dragging & repositioning the boxes below.

Articles Related By Topic
Related Collections
PubMed Articles
JAMAevidence.com

Users' Guides to the Medical Literature
Influenza

The Rational Clinical Examination
Make the Diagnosis: Influenza