Active Annotation in Evaluating the Credibility of Web-Based Medical Information: Guidelines for Creating Training Data Sets for Machine Learning.
Ontology highlight
ABSTRACT: Background
The spread of false medical information on the web is rapidly accelerating. Establishing the credibility of web-based medical information has become a pressing necessity. Machine learning offers a solution that, when properly deployed, can be an effective tool in fighting medical misinformation on the web.Objective
The aim of this study is to present a comprehensive framework for designing and curating machine learning training data sets for web-based medical information credibility assessment. We show how to construct the annotation process. Our main objective is to support researchers from the medical and computer science communities. We offer guidelines on the preparation of data sets for machine learning models that can fight medical misinformation.Methods
We begin by providing the annotation protocol for medical experts involved in medical sentence credibility evaluation. The protocol is based on a qualitative study of our experimental data. To address the problem of insufficient initial labels, we propose a preprocessing pipeline for the batch of sentences to be assessed. It consists of representation learning, clustering, and reranking. We call this process active annotation.Results
We collected more than 10,000 annotations of statements related to selected medical subjects (psychiatry, cholesterol, autism, antibiotics, vaccines, steroids, birth methods, and food allergy testing) for less than US $7000 by employing 9 highly qualified annotators (certified medical professionals), and we release this data set to the general public. We developed an active annotation framework for more efficient annotation of noncredible medical statements. The application of qualitative analysis resulted in a better annotation protocol for our future efforts in data set creation.Conclusions
The results of the qualitative analysis support our claims of the efficacy of the presented method.
SUBMITTER: Nabozny A
PROVIDER: S-EPMC8665397 | biostudies-literature |
REPOSITORIES: biostudies-literature
Similar Datasets
Project description:Gene annotation databases (compendiums maintained by the scientific community that describe the biological functions performed by individual genes) are commonly used to evaluate the functional properties of experimentally derived gene sets. Overlap statistics, such as Fishers Exact test (FET), are often employed to assess these associations, but don't account for non-uniformity in the number of genes annotated to individual functions or the number of functions associated with individual genes. We find FET is strongly biased toward over-estimating overlap significance if a gene set has an unusually high number of annotations. To correct for these biases, we develop Annotation Enrichment Analysis (AEA), which properly accounts for the non-uniformity of annotations. We show that AEA is able to identify biologically meaningful functional enrichments that are obscured by numerous false-positive enrichment scores in FET, and we therefore suggest it be used to more accurately assess the biological properties of gene sets.
| S-EPMC3935204 | biostudies-other
Project description:Functional context for biological sequence is provided in the form of annotations. However, within a group of similar sequences there can be annotation heterogeneity in terms of coverage and specificity. This in turn can introduce issues regarding the interpretation of actual functional similarity and overall functional coherence of such a group. One way to mitigate such issues is through the use of visualization and statistical techniques. Therefore, in order to help interpret this annotation heterogeneity we created a web application that generates Gene Ontology annotation graphs for protein sets and their associated statistics from simple frequencies to enrichment values and Information Content based metrics. The publicly accessible website http://xldb.di.fc.ul.pt/gryfun/ currently accepts lists of UniProt accession numbers in order to create user-defined protein sets for subsequent annotation visualization and statistical assessment. GRYFUN is a freely available web application that allows GO annotation visualization of protein sets and which can be used for annotation coherence and cohesiveness analysis and annotation extension assessments within under-annotated protein sets.
| S-EPMC4368792 | biostudies-literature
Project description:MOTIVATION:The recent revolution in new sequencing technologies, as a part of the continuous process of adopting new innovative protocols has strongly impacted the interpretation of relations between phenotype and genotype. Thus, understanding the resulting gene sets has become a bottleneck that needs to be addressed. Automatic methods have been proposed to facilitate the interpretation of gene sets. While statistical functional enrichment analyses are currently well known, they tend to focus on well-known genes and to ignore new information from less-studied genes. To address such issues, applying semantic similarity measures is logical if the knowledge source used to annotate the gene sets is hierarchically structured. In this work, we propose a new method for analyzing the impact of different semantic similarity measures on gene set annotations. RESULTS:We evaluated the impact of each measure by taking into consideration the two following features that correspond to relevant criteria for a "good" synthetic gene set annotation: (i) the number of annotation terms has to be drastically reduced and the representative terms must be retained while annotating the gene set, and (ii) the number of genes described by the selected terms should be as large as possible. Thus, we analyzed nine semantic similarity measures to identify the best possible compromise between both features while maintaining a sufficient level of details. Using Gene Ontology to annotate the gene sets, we obtained better results with node-based measures that use the terms' characteristics than with measures based on edges that link the terms. The annotation of the gene sets achieved with the node-based measures did not exhibit major differences regardless of the characteristics of terms used.
| S-EPMC6258551 | biostudies-other
Project description:Background: The current pandemic of COVID-19 has changed the way health information is distributed through online platforms. These platforms have played a significant role in informing patients and the public with knowledge that has changed the virtual world forever. Simultaneously, there are growing concerns that much of the information is not credible, impacting patient health outcomes, causing human lives, and tremendous resource waste. With the increasing use of online platforms, patients/the public require new learning models and sharing medical knowledge. They need to be empowered with strategies to navigate disinformation on online platforms. Methods and Design: To meet the urgent need to combat health "misinformation," the research team proposes a structured approach to develop a quality benchmark, an evidence-based tool that identifies and addresses the determinants of online health information reliability. The specific methods to develop the intervention are the following: (1) systematic reviews: two comprehensive systematic reviews to understand the current state of the quality of online health information and to identify research gaps, (2) content analysis: develop a conceptual framework based on established and complementary knowledge translation approaches for analyzing the existing quality assessment tools and draft a unique set of quality of domains, (3) focus groups: multiple focus groups with diverse patients/the public and health information providers to test the acceptability and usability of the quality domains, (4) development and evaluation: a unique set of determinants of reliability will be finalized along with a preferred scoring classification. These items will be used to develop and validate a quality benchmark to assess the quality of online health information. Expected Outcomes: This multi-phase project informed by theory will lead to new knowledge that is intended to inform the development of a patient-friendly quality benchmark. This benchmark will inform best practices and policies in disseminating reliable web health information, thus reducing disparities in access to health knowledge and combat misinformation online. In addition, we envision the final product can be used as a gold standard for developing similar interventions for specific groups of patients or populations.
| S-EPMC8732749 | biostudies-literature
Project description:BACKGROUND:Three main changes were implemented in the Australian National Cervical Screening Program (NCSP) in December 2017: an increase in the recommended age to start screening, extended screening intervals, and change from the Papanicolaou (Pap) test to primary human papillomavirus screening (cervical screening test). The internet is a readily accessible source of information to explain the reasons for these changes to the public. It is important that web-based health information about changes to national screening programs is accessible and understandable for the general population. OBJECTIVE:This study aimed to evaluate Australian web-based resources that provide information about the changes to the cervical screening program. METHODS:The term cervical screening was searched in 3 search engines. The first 10 relevant results across the first 3 pages of each search engine were selected. Overall, 2 authors independently evaluated each website for readability (Flesch Reading Ease [FRE], Flesch-Kincaid Grade Level, and Simple Measure of Gobbledygook [SMOG] index), quality of information (Patient Education Materials Assessment Tool [PEMAT] for printable materials), credibility (Journal of the American Medical Association [JAMA] benchmark criteria and presence of Health on the Net Foundation code of conduct [HONcode] certification), website design, and usability with 5 simulation questions to assess the relevance of information. A descriptive analysis was conducted for the readability measures, PEMAT, and the JAMA benchmark criteria. RESULTS:Of the 49 websites identified in the search, 15 were eligible for inclusion. The consumer-focused websites were classed as fairly difficult to read (mean FRE score 51.8, SD 13.3). The highest FRE score (easiest to read) was 70.4 (Cancer Council Australia Cervical Screening Consumer Site), and the lowest FRE score (most difficult to read) was 33.0 (NCSP Clinical Guidelines). A total of 9 consumer-focused websites and 4 health care provider-focused websites met the recommended threshold (sixth to eighth grade; SMOG index) for readability. The mean PEMAT understandability scores were 87.7% (SD 6.0%) for consumer-focused websites and 64.9% (SD 13.8%) for health care provider-focused websites. The mean actionability scores were 58.1% (SD 19.1%) for consumer-focused websites and 36.7% (SD 11.0%) for health care provider-focused websites. Moreover, 9 consumer-focused and 3 health care provider-focused websites scored above 70% for understandability, and 2 consumer-focused websites had an actionability score above 70%. A total of 3 websites met all 4 of the JAMA benchmark criteria, and 2 websites displayed the HONcode. CONCLUSIONS:It is important for women to have access to information that is at an appropriate reading level to better understand the implications of the changes to the cervical screening program. These findings can help health care providers direct their patients toward websites that provide information on cervical screening that is written at accessible reading levels and has high understandability.
| S-EPMC7381085 | biostudies-literature
Project description:Collective intelligence is the idea that under the right circumstances collections of individuals are smarter than even the smartest individuals in the group (Suroweiki 2004), that is a group has an "intelligence" that is independent of the intelligence of its members. The ideology of collective intelligence undergirds much of the enthusiasm about the use of "volunteered" or crowdsourced geographic information. Literature from a variety of fields makes clear that not all groups possess collective intelligence, this paper identifies four pre-conditions for the emergence of collective intelligence and then examine the extent to which collectively generated mapping systems satisfy these conditions. However, the "intelligence" collectively generated maps is hard to assess because there are two difficult to reconcile perspectives on map quality- the credibility perspective and the accuracy perspective. Much of the current literature on user generated maps focuses on assessing the quality of individual contributions. However, because user generated maps are complex social systems and because the quality of a contribution is difficult to assess this strategy may not yield an "intelligent" end product. The existing literature on collective intelligence suggests that the structure of groups more important that the intelligence of group members. Applying this idea to user generated suggests that systems should be designed to foster conditions known to produce collective intelligence rather than privileging particular contributions/contributors. The paper concludes with some design recommendations and by considering the implications of collectively generated maps for both expert knowledge and traditional state sponsored mapping programs.
| S-EPMC4238431 | biostudies-literature
Project description:Providing ground truth is essential for activity recognition and behaviour analysis as it is needed for providing training data in methods of supervised learning, for providing context information for knowledge-based methods, and for quantifying the recognition performance. Semantic annotation extends simple symbolic labelling by assigning semantic meaning to the label, enabling further reasoning. In this paper, we present a novel approach to semantic annotation by means of plan operators. We provide a step by step description of the workflow to manually creating the ground truth annotation. To validate our approach, we create semantic annotation of the Carnegie Mellon University (CMU) grand challenge dataset, which is often cited, but, due to missing and incomplete annotation, almost never used. We show that it is possible to derive hidden properties, behavioural routines, and changes in initial and goal conditions in the annotated dataset. We evaluate the quality of the annotation by calculating the interrater reliability between two annotators who labelled the dataset. The results show very good overlapping (Cohen's ? of 0.8) between the annotators. The produced annotation and the semantic models are publicly available, in order to enable further usage of the CMU grand challenge dataset.
| S-EPMC6163329 | biostudies-literature
Project description:BackgroundUniversity students aged 18-30 years are a population group reporting low access to health care services, with high rates of avoidance and delay of medical care. This group also reports not having appropriate information about available health care services. However, university students are at risk for several health problems, and regular medical consultations are recommended in this period of life. New digital devices are popular among the young, and Web-apps can be used to facilitate easy access to information regarding health care services. A small number of electronic health (eHealth) tools have been developed with the purpose of displaying real-world health care services, and little is known about how such eHealth tools can improve access to care.ObjectiveThis paper describes the processes of co-creating and evaluating the beta version of a Web-app aimed at mapping and describing free or low-cost real-world health care services available in the Bordeaux area of France, which is specifically targeted to university students.MethodsThe co-creation process involves: (1) exploring the needs of students to know and access real-world health care services; (2) identifying the real-world health care services of interest for students; and (3) deciding on a user interface, and developing the beta version of the Web-app. Finally, the evaluation process involves: (1) testing the beta version of the Web-app with the target audience (university students aged 18-30 years); (2) collecting their feedback via a satisfaction survey; and (3) planning a long-term evaluation.ResultsThe co-creation process of the beta version of the Web-app was completed in August 2016 and is described in this paper. The evaluation process started on September 7, 2016. The project was completed in December 2016 and implementation of the Web-app is ongoing.ConclusionsWeb-apps are an innovative way to increase the health literacy of young people in terms of delivery of and access to health care. The creation of Web-apps benefits from the involvement of stakeholders (eg, students and health care providers) to correctly identify the real-world health care services to be displayed.
| S-EPMC5334513 | biostudies-literature
Project description:BACKGROUND: Model repositories such as BioModels Database provide computational models of biological systems for the scientific community. These models contain rich semantic annotations that link model entities to concepts in well-established bio-ontologies such as Gene Ontology. Consequently, thematically similar models are likely to share similar annotations. Based on this assumption, we argue that semantic annotations are a suitable tool to characterize sets of models. These characteristics improve model classification, allow to identify additional features for model retrieval tasks, and enable the comparison of sets of models. RESULTS: In this paper we discuss four methods for annotation-based feature extraction from model sets. We tested all methods on sets of models in SBML format which were composed from BioModels Database. To characterize each of these sets, we analyzed and extracted concepts from three frequently used ontologies, namely Gene Ontology, ChEBI and SBO. We find that three out of the methods are suitable to determine characteristic features for arbitrary sets of models: The selected features vary depending on the underlying model set, and they are also specific to the chosen model set. We show that the identified features map on concepts that are higher up in the hierarchy of the ontologies than the concepts used for model annotations. Our analysis also reveals that the information content of concepts in ontologies and their usage for model annotation do not correlate. CONCLUSIONS: Annotation-based feature extraction enables the comparison of model sets, as opposed to existing methods for model-to-keyword comparison, or model-to-model comparison.
| S-EPMC4405863 | biostudies-other
Project description:Members of the public share their views with researchers to improve health and social care research. Lay assessing is one way of doing this. This is where people, drawing upon personal and general life experience, comment on material, such as grant applications and patient information, to highlight strengths and weaknesses and to suggest improvements. This paper reports on setting up a training programme for lay assessors. Meetings were held between interested public and staff from research organisations. People discussed what lay assessing is, why they want to do it, skills and support needed and if training was wanted. They were invited to form a group to develop the training together. Training was delivered in the East Midlands. People who attended gave their thoughts about it by completing questionnaires and joining a feedback event. The group developed the structure of the training programme together and it oversaw the development of the training content by individual members. People who attended training reported feeling more confident about lay assessing. This was particularly so for those who had not done lay assessing before. They indicated how valuable it was to talk with others at the training. Our findings support the National Institute for Health Research recommendations for improving learning and development for public involvement in research. This project has created a solid base for local research organisations to work together in public involvement training. Lay assessor training is now part of a wider programme of shared resources called the Sharebank.Background Involving members of the public in research can improve its quality and incorporate the needs and views of patients. One method for doing this is lay assessing, where members of the public are consulted to improve research materials. This paper documents the establishment of a pilot training programme for lay assessors. It describes a way of working that embodies a regional, cross-organisational approach to co-producing training with members of the public. Methods Open meetings, led by AH, were held for existing and aspiring lay assessors to define lay assessing, motivations for doing it, skills required, associated learning and development needs, and to gauge interest for training. Those who attended meetings, including members of the public and staff, were invited to form a working group to co-produce the training programme. Training was delivered in modules at two centres in the East Midlands and evaluated through participant feedback at the end of each module and at an evaluation event. Feedback was through a mix of Likert scale scoring, open text and verbal responses. Results Discussions from the open meetings informed the development of the training by the working group. Led by AH, the working group, as a whole, co-produced the structure and format of the training and oversaw training content development by individuals within the group. Training was well-received by participants. Feedback through Likert scoring (n?=?14) indicated higher feelings of confidence in knowledge of relevant subject matter and in fulfilling the lay assessor role, particularly amongst those who had not done lay assessing before. Opportunities that the training afforded for interaction between participants - sharing of varied experiences and knowledge - and a 'learn by doing' approach was of particular value, as indicated by 10 responses to open-ended questions. Conclusions This project has created a solid foundation for collaboration between research organisations in the East Midlands in devising and delivering training in public involvement together. Our evaluation provides evidence in support of National Institute for Health Research (NIHR) recommendations on principles for learning and development for public involvement in research.
| S-EPMC5611661 | biostudies-other