Dataset Information

Evaluating the utility of syndromic surveillance algorithms for screening to detect potentially clonal hospital infection outbreaks.

ABSTRACT:

Objective

The authors evaluated algorithms commonly used in syndromic surveillance for use as screening tools to detect potentially clonal outbreaks for review by infection control practitioners.

Design

Study phase 1 applied four aberrancy detection algorithms (CUSUM, EWMA, space-time scan statistic, and WSARE) to retrospective microbiologic culture data, producing a list of past candidate outbreak clusters. In phase 2, four infectious disease physicians categorized the phase 1 algorithm-identified clusters to ascertain algorithm performance. In phase 3, project members combined the algorithms to create a unified screening system and conducted a retrospective pilot evaluation.

Measurements

The study calculated recall and precision for each algorithm, and created precision-recall curves for various methods of combining the algorithms into a unified screening tool.

Results

Individual algorithm recall and precision ranged from 0.21 to 0.31 and from 0.053 to 0.29, respectively. Few candidate outbreak clusters were identified by more than one algorithm. The best method of combining the algorithms yielded an area under the precision-recall curve of 0.553. The phase 3 combined system detected all infection control-confirmed outbreaks during the retrospective evaluation period.

Limitations

Lack of phase 2 reviewers' agreement indicates that subjective expert review was an imperfect gold standard. Less conservative filtering of culture results and alternate parameter selection for each algorithm might have improved algorithm performance.

Conclusion

Hospital outbreak detection presents different challenges than traditional syndromic surveillance. Nevertheless, algorithms developed for syndromic surveillance have potential to form the basis of a combined system that might perform clinically useful hospital outbreak screening.

SUBMITTER: Carnevale RJ

PROVIDER: S-EPMC3128411 | biostudies-literature | 2011 Jul-Aug

REPOSITORIES: biostudies-literature

ACCESS DATA

Publications

Evaluating the utility of syndromic surveillance algorithms for screening to detect potentially clonal hospital infection outbreaks.

Carnevale Randy J RJ Talbot Thomas R TR Schaffner William W Bloch Karen C KC Daniels Titus L TL Miller Randolph A RA

Journal of the American Medical Informatics Association : JAMIA 20110523 4

<h4>Objective</h4>The authors evaluated algorithms commonly used in syndromic surveillance for use as screening tools to detect potentially clonal outbreaks for review by infection control practitioners.<h4>Design</h4>Study phase 1 applied four aberrancy detection algorithms (CUSUM, EWMA, space-time scan statistic, and WSARE) to retrospective microbiologic culture data, producing a list of past candidate outbreak clusters. In phase 2, four infectious disease physicians categorized the phase 1 al ...[more]

PMID: 21606134

Similar Datasets

Project description:Benchmarking surveillance systems requires realistic simulations of disease outbreaks. However, obtaining these data in sufficient quantity, with a realistic shape and covering a sufficient range of agents, size and duration, is known to be very difficult. The dataset of outbreak signals generated should reflect the likely distribution of authentic situations faced by the surveillance system, including very unlikely outbreak signals. We propose and evaluate a new approach based on the use of historical outbreak data to simulate tailored outbreak signals. The method relies on a homothetic transformation of the historical distribution followed by resampling processes (Binomial, Inverse Transform Sampling Method-ITSM, Metropolis-Hasting Random Walk, Metropolis-Hasting Independent, Gibbs Sampler, Hybrid Gibbs Sampler). We carried out an analysis to identify the most important input parameters for simulation quality and to evaluate performance for each of the resampling algorithms. Our analysis confirms the influence of the type of algorithm used and simulation parameters (i.e. days, number of cases, outbreak shape, overall scale factor) on the results. We show that, regardless of the outbreaks, algorithms and metrics chosen for the evaluation, simulation quality decreased with the increase in the number of days simulated and increased with the number of cases simulated. Simulating outbreaks with fewer cases than days of duration (i.e. overall scale factor less than 1) resulted in an important loss of information during the simulation. We found that Gibbs sampling with a shrinkage procedure provides a good balance between accuracy and data dependency. If dependency is of little importance, binomial and ITSM methods are accurate. Given the constraint of keeping the simulation within a range of plausible epidemiological curves faced by the surveillance system, our study confirms that our approach can be used to generate a large spectrum of outbreak signals.

Project description:BackgroundAlthough syndromic surveillance is increasingly used to detect unusual illness, there is a debate whether it is useful for detecting local outbreaks. We evaluated whether syndromic surveillance detects local outbreaks of lower-respiratory infections (LRIs) without swamping true signals by false alarms.Methods and findingsUsing retrospective hospitalization data, we simulated prospective surveillance for LRI-elevations. Between 1999-2006, a total of 290762 LRIs were included by date of hospitalization and patients place of residence (>80% coverage, 16 million population). Two large outbreaks of Legionnaires disease in the Netherlands were used as positive controls to test whether these outbreaks could have been detected as local LRI elevations. We used a space-time permutation scan statistic to detect LRI clusters. We evaluated how many LRI-clusters were detected in 1999-2006 and assessed likely causes for the cluster-signals by looking for significantly higher proportions of specific hospital discharge diagnoses (e.g. Legionnaires disease) and overlap with regional influenza elevations. We also evaluated whether the number of space-time signals can be reduced by restricting the scan statistic in space or time. In 1999-2006 the scan-statistic detected 35 local LRI clusters, representing on average 5 clusters per year. The known Legionnaires' disease outbreaks in 1999 and 2006 were detected as LRI-clusters, since cluster-signals were generated with an increased proportion of Legionnaires disease patients (p:<0.0001). 21 other clusters coincided with local influenza and/or respiratory syncytial virus activity, and 1 cluster appeared to be a data artifact. For 11 clusters no likely cause was defined, some possibly representing as yet undetected LRI-outbreaks. With restrictions on time and spatial windows the scan statistic still detected the Legionnaires' disease outbreaks, without loss of timeliness and with less signals generated in time (up to 42% decline).ConclusionsTo our knowledge this is the first study that systematically evaluates the performance of space-time syndromic surveillance with nationwide high coverage data over a longer period. The results show that syndromic surveillance can detect local LRI-outbreaks in a timely manner, independent of laboratory-based outbreak detection. Furthermore, since comparatively few new clusters per year were observed that would prompt investigation, syndromic hospital-surveillance could be a valuable tool for detection of local LRI-outbreaks.

Project description:BackgroundWaterborne outbreaks are still a risk in high-income countries, and their early detection is crucial to limit their societal consequences. Although syndromic surveillance is widely used for the purpose of detecting outbreaks days earlier than traditional surveillance systems, evidence of the effectiveness of such systems is lacking. Thus, our objective was to conduct a systematic review of the effectiveness of syndromic surveillance to detect waterborne outbreaks.MethodWe searched the Cochrane Library, Medline/PubMed, EMBASE, Scopus, and Web of Science for relevant published articles using a combination of the keywords 'drinking water', 'surveillance', and 'waterborne disease' for the period of 1990 to 2018. The references lists of the identified articles for full-text record assessment were screened, and searches in Google Scholar using the same key words were conducted. We assessed the risk of bias in the included articles using the ROBINS-I tool and PRECEPT for the cumulative body of evidence.ResultsFrom the 1959 articles identified, we reviewed 52 articles, of which 18 met the eligibility criteria. Twelve were descriptive/analytical studies, whereas six were simulation studies. There is no clear evidence for syndromic surveillance in terms of the ability to detect waterborne outbreaks (low sensitivity and high specificity). However, one simulation study implied that multiple sources of signals combined with spatial information may increase the timeliness in detecting a waterborne outbreak and reduce false alarms.ConclusionThis review demonstrates that there is no conclusive evidence on the effectiveness of syndromic surveillance for the detection of waterborne outbreaks, thus suggesting the need to focus on primary prevention measures to reduce the risk of waterborne outbreaks. Future studies should investigate methods for combining health and environmental data with an assessment of needed financial and human resources for implementing such surveillance systems. In addition, a more critical thematic narrative synthesis on the most promising sources of data, and an assessment of the basis for arguments that joint analysis of different data or dimensions of data (e.g. spatial and temporal) might perform better, should be carried out.Trial registrationPROSPERO: International prospective register of systematic reviews. 2019. CRD42019122332 .

Dataset Information

Evaluating the utility of syndromic surveillance algorithms for screening to detect potentially clonal hospital infection outbreaks.

Objective

Design

Measurements

Results

Limitations

Conclusion

Publications

Evaluating the utility of syndromic surveillance algorithms for screening to detect potentially clonal hospital infection outbreaks.

Similar Datasets

OmicsDI is part of the ELIXIR infrastructure

Tweets