Dataset Information

An indicator framework for the monitoring and evaluation of event-based surveillance systems.

ABSTRACT: Event-based surveillance (EBS) systems have been implemented globally to support early warning surveillance across human, animal, and environmental health in diverse settings, including at the community level, within health facilities, at border points of entry, and through media monitoring of internet-based sources. EBS systems should be evaluated periodically to ensure that they meet the objectives related to the early detection of health threats and to identify areas for improvement in the quality, efficiency, and usefulness of the systems. However, to date, there has been no comprehensive framework to guide the monitoring and evaluation of EBS systems; this absence of standardisation has hindered progress in the field. The Africa Centres for Disease Control and Prevention and US Centers for Disease Control and Prevention have collaborated to develop an EBS monitoring and evaluation indicator framework, adaptable to specific country contexts, that uses measures relating to input, activity, output, outcome, and impact to map the processes and expected results of EBS systems. Through the implementation and continued refinement of these indicators, countries can ensure the early detection of health threats and improve their ability to measure and describe the impacts of EBS systems, thus filling the current evidence gap regarding their effectiveness.

SUBMITTER: Crawley AW

PROVIDER: S-EPMC11348395 | biostudies-literature | 2024 Apr

REPOSITORIES: biostudies-literature

ACCESS DATA

Publications

An indicator framework for the monitoring and evaluation of event-based surveillance systems.

Crawley Adam W AW Mercy Kyeng K Shivji Sabrina S Lofgren Hannah H Trowbridge Daniella D Manthey Christine C Tebeje Yenew Kebede YK Clara Alexey Wil AW Landry Kimberly K Salyer Stephanie J SJ

The Lancet. Global health 20240213 4

Event-based surveillance (EBS) systems have been implemented globally to support early warning surveillance across human, animal, and environmental health in diverse settings, including at the community level, within health facilities, at border points of entry, and through media monitoring of internet-based sources. EBS systems should be evaluated periodically to ensure that they meet the objectives related to the early detection of health threats and to identify areas for improvement in the qu ...[more]

PMID: 38364834

Similar Datasets

Project description:According to the World Health Organization (WHO), around 60% of all outbreaks are detected using informal sources. In many public health institutes, including the WHO and the Robert Koch Institute (RKI), dedicated groups of public health agents sift through numerous articles and newsletters to detect relevant events. This media screening is one important part of event-based surveillance (EBS). Reading the articles, discussing their relevance, and putting key information into a database is a time-consuming process. To support EBS, but also to gain insights into what makes an article and the event it describes relevant, we developed a natural language processing framework for automated information extraction and relevance scoring. First, we scraped relevant sources for EBS as done at the RKI (WHO Disease Outbreak News and ProMED) and automatically extracted the articles' key data: disease, country, date, and confirmed-case count. For this, we performed named entity recognition in two steps: EpiTator, an open-source epidemiological annotation tool, suggested many different possibilities for each. We extracted the key country and disease using a heuristic with good results. We trained a naive Bayes classifier to find the key date and confirmed-case count, using the RKI's EBS database as labels which performed modestly. Then, for relevance scoring, we defined two classes to which any article might belong: The article is relevant if it is in the EBS database and irrelevant otherwise. We compared the performance of different classifiers, using bag-of-words, document and word embeddings. The best classifier, a logistic regression, achieved a sensitivity of 0.82 and an index balanced accuracy of 0.61. Finally, we integrated these functionalities into a web application called EventEpi where relevant sources are automatically analyzed and put into a database. The user can also provide any URL or text, that will be analyzed in the same way and added to the database. Each of these steps could be improved, in particular with larger labeled datasets and fine-tuning of the learning algorithms. The overall framework, however, works already well and can be used in production, promising improvements in EBS. The source code and data are publicly available under open licenses.

Project description:BackgroundSyndromic surveillance complements traditional public health surveillance by collecting and analysing health indicators in near real time. The rationale of syndromic surveillance is that it may detect health threats faster than traditional surveillance systems permitting more timely, and hence potentially more effective public health action. The effectiveness of syndromic surveillance largely relies on the methods used to detect aberrations. Very few studies have evaluated the performance of syndromic surveillance systems and consequently little is known about the types of events that such systems can and cannot detect.MethodsWe introduce a framework for the evaluation of syndromic surveillance systems that can be used in any setting based upon the use of simulated scenarios. For a range of scenarios this allows the time and probability of detection to be determined and uncertainty is fully incorporated. In addition, we demonstrate how such a framework can model the benefits of increases in the number of centres reporting syndromic data and also determine the minimum size of outbreaks that can or cannot be detected. Here, we demonstrate its utility using simulations of national influenza outbreaks and localised outbreaks of cryptosporidiosis.ResultsInfluenza outbreaks are consistently detected with larger outbreaks being detected in a more timely manner. Small cryptosporidiosis outbreaks (<1000 symptomatic individuals) are unlikely to be detected. We also demonstrate the advantages of having multiple syndromic data streams (e.g. emergency attendance data, telephone helpline data, general practice consultation data) as different streams are able to detect different outbreak types with different efficacy (e.g. emergency attendance data are useful for the detection of pandemic influenza but not for outbreaks of cryptosporidiosis). We also highlight that for any one disease, the utility of data streams may vary geographically, and that the detection ability of syndromic surveillance varies seasonally (e.g. an influenza outbreak starting in July is detected sooner than one starting later in the year). We argue that our framework constitutes a useful tool for public health emergency preparedness in multiple settings.ConclusionsThe proposed framework allows the exhaustive evaluation of any syndromic surveillance system and constitutes a useful tool for emergency preparedness and response.

Project description:BackgroundCurrent public concern over the spread of infectious diseases has underscored the importance of health surveillance systems for the speedy detection of disease outbreaks. Several international report-based monitoring systems have been developed, including GPHIN, Argus, HealthMap, and BioCaster. A vital feature of these report-based systems is the geo-temporal encoding of outbreak-related textual data. Until now, automated systems have tended to use an ad-hoc strategy for processing geo-temporal information, normally involving the detection of locations that match pre-determined criteria, and the use of document publication dates as a proxy for disease event dates. Although these strategies appear to be effective enough for reporting events at the country and province levels, they may be less effective at discovering geo-temporal information at more detailed levels of granularity. In order to improve the capabilities of current Web-based health surveillance systems, we introduce the design for a novel scheme called spatiotemporal zoning.MethodThe proposed scheme classifies news articles into zones according to the spatiotemporal characteristics of their content. In order to study the reliability of the annotation scheme, we analyzed the inter-annotator agreements on a group of human annotators for over 1000 reported events. Qualitative and quantitative evaluation is made on the results including the kappa and percentage agreement.ResultsThe reliability evaluation of our scheme yielded very promising inter-annotator agreement, more than a 0.9 kappa and a 0.9 percentage agreement for event type annotation and temporal attributes annotation, respectively, with a slight degradation for the spatial attribute. However, for events indicating an outbreak situation, the annotators usually had inter-annotator agreements with the lowest granularity location.ConclusionsWe developed and evaluated a novel spatiotemporal zoning annotation scheme. The results of the scheme evaluation indicate that our annotated corpus and the proposed annotation scheme are reliable and could be effectively used for developing an automatic system. Given the current advances in natural language processing techniques, including the availability of language resources and tools, we believe that a reliable automatic spatiotemporal zoning system can be achieved. In the next stage of this work, we plan to develop an automatic zoning system and evaluate its usability within an operational health surveillance system.

Dataset Information

An indicator framework for the monitoring and evaluation of event-based surveillance systems.

Publications

An indicator framework for the monitoring and evaluation of event-based surveillance systems.

Similar Datasets

OmicsDI is part of the ELIXIR infrastructure

Tweets