Dataset Information

Fast and accurate semantic annotation of bioassays exploiting a hybrid of machine learning and user confirmation.

ABSTRACT: Bioinformatics and computer aided drug design rely on the curation of a large number of protocols for biological assays that measure the ability of potential drugs to achieve a therapeutic effect. These assay protocols are generally published by scientists in the form of plain text, which needs to be more precisely annotated in order to be useful to software methods. We have developed a pragmatic approach to describing assays according to the semantic definitions of the BioAssay Ontology (BAO) project, using a hybrid of machine learning based on natural language processing, and a simplified user interface designed to help scientists curate their data with minimum effort. We have carried out this work based on the premise that pure machine learning is insufficiently accurate, and that expecting scientists to find the time to annotate their protocols manually is unrealistic. By combining these approaches, we have created an effective prototype for which annotation of bioassay text within the domain of the training set can be accomplished very quickly. Well-trained annotations require single-click user approval, while annotations from outside the training set domain can be identified using the search feature of a well-designed user interface, and subsequently used to improve the underlying models. By drastically reducing the time required for scientists to annotate their assays, we can realistically advocate for semantic annotation to become a standard part of the publication process. Once even a small proportion of the public body of bioassay data is marked up, bioinformatics researchers can begin to construct sophisticated and useful searching and analysis algorithms that will provide a diverse and powerful set of tools for drug discovery researchers.

SUBMITTER: Clark AM

PROVIDER: S-EPMC4137659 | biostudies-literature | 2014

REPOSITORIES: biostudies-literature

ACCESS DATA

Publications

Fast and accurate semantic annotation of bioassays exploiting a hybrid of machine learning and user confirmation.

Clark Alex M AM Bunin Barry A BA Litterman Nadia K NK Schürer Stephan C SC Visser Ubbo U

PeerJ 20140814

Bioinformatics and computer aided drug design rely on the curation of a large number of protocols for biological assays that measure the ability of potential drugs to achieve a therapeutic effect. These assay protocols are generally published by scientists in the form of plain text, which needs to be more precisely annotated in order to be useful to software methods. We have developed a pragmatic approach to describing assays according to the semantic definitions of the BioAssay Ontology (BAO) p ...[more]

PMID: 25165633

Similar Datasets

Project description:BackgroundOne in five U.S. adults lives with some kind of mental health condition and 4.6% of all U.S. adults have a serious mental illness. The Internet has become the first place for these people to seek online mental health information for help. However, online mental health information is not well-organized and often of low quality. There have been efforts in building evidence-based mental health knowledgebases curated with information manually extracted from the high-quality scientific literature. Manual extraction is inefficient. Crowdsourcing can potentially be a low-cost mechanism to collect labeled data from non-expert laypeople. However, there is not an existing annotation tool integrated with popular crowdsourcing platforms to perform the information extraction tasks. In our previous work, we prototyped a Semantic Text Annotation Tool (STAT) to address this gap.ObjectiveWe aimed to refine the STAT prototype (1) to improve its usability and (2) to enhance the crowdsourcing workflow efficiency to facilitate the construction of evidence-based mental health knowledgebase, following a user-centered design (UCD) approach.MethodsFollowing UCD principles, we conducted four design iterations to improve the initial STAT prototype. In the first two iterations, usability testing focus groups were conducted internally with 8 participants recruited from a convenient sample, and the usability was evaluated with a modified System Usability Scale (SUS). In the following two iterations, usability testing was conducted externally using the Amazon Mechanical Turk (MTurk) platform. In each iteration, we summarized the usability testing results through thematic analysis, identified usability issues, and conducted a heuristic evaluation to map identified usability issues to Jakob Nielsen's usability heuristics. We collected suggested improvements in the usability testing sessions and enhanced STAT accordingly in the next UCD iteration. After four UCD iterations, we conducted a case study of the system on MTurk using mental health related scientific literature. We compared the performance of crowdsourcing workers with two expert annotators from two aspects: efficiency and quality.ResultsThe SUS score increased from 70.3 ± 12.5 to 81.1 ± 9.8 after the two internal UCD iterations as we improved STAT's functionality based on the suggested improvements. We then evaluated STAT externally through MTurk in the following two iterations. The SUS score decreased to 55.7 ± 20.1 in the third iteration, probably because of the complexity of the tasks. After further simplification of STAT and the annotation tasks with an improved annotation guideline, the SUS score increased to 73.8 ± 13.8 in the fourth iteration of UCD. In the evaluation case study, on average, the workers spent 125.5 ± 69.2 s on the onboarding tutorial and the crowdsourcing workers spent significantly less time on the annotation tasks compared to the two experts. In terms of annotation quality, the workers' annotation results achieved average F1-scores ranged from 0.62 to 0.84 for the different sentences.ConclusionsWe successfully developed a web-based semantic text annotation tool, STAT, to facilitate the curation of semantic web knowledgebases through four UCD iterations. The lessons learned from the UCD process could serve as a guide to further enhance STAT and the development and design of other crowdsourcing-based semantic text annotation tasks. Our study also showed that a well-organized, informative annotation guideline is as important as the annotation tool itself. Further, we learned that a crowdsourcing task should consist of multiple simple microtasks rather than a complicated task.

Project description:Accurate anatomical localization of intracranial electrodes is important for identifying the seizure foci in patients with epilepsy and for interpreting effects from cognitive studies employing intracranial electroencephalography. Localization is typically performed by coregistering postimplant computed tomography (CT) with preoperative magnetic resonance imaging (MRI). Electrodes are then detected in the CT, and the corresponding brain region is identified using the MRI. Many existing software packages for electrode localization chain together separate preexisting programs or rely on command line instructions to perform the various localization steps, making them difficult to install and operate for a typical user. Further, many packages provide solutions for some, but not all, of the steps needed for confident localization. We have developed software, Locate electrodes Graphical User Interface (LeGUI), that consists of a single interface to perform all steps needed to localize both surface and depth/penetrating intracranial electrodes, including coregistration of the CT to MRI, normalization of the MRI to the Montreal Neurological Institute template, automated electrode detection for multiple types of electrodes, electrode spacing correction and projection to the brain surface, electrode labeling, and anatomical targeting. The software is written in MATLAB, core image processing is performed using the Statistical Parametric Mapping toolbox, and standalone executable binaries are available for Windows, Mac, and Linux platforms. LeGUI was tested and validated on 51 datasets from two universities. The total user and computational time required to process a single dataset was approximately 1 h. Automatic electrode detection correctly identified 4362 of 4695 surface and depth electrodes with only 71 false positives. Anatomical targeting was verified by comparing electrode locations from LeGUI to locations that were assigned by an experienced neuroanatomist. LeGUI showed a 94% match with the 482 neuroanatomist-assigned locations. LeGUI combines all the features needed for fast and accurate anatomical localization of intracranial electrodes into a single interface, making it a valuable tool for intracranial electrophysiology research.

Dataset Information

Fast and accurate semantic annotation of bioassays exploiting a hybrid of machine learning and user confirmation.

Publications

Fast and accurate semantic annotation of bioassays exploiting a hybrid of machine learning and user confirmation.

Similar Datasets

OmicsDI is part of the ELIXIR infrastructure

Tweets