Dataset Information

Development and evaluation of a crowdsourcing methodology for knowledge base construction: identifying relationships between clinical problems and medications.

ABSTRACT: We describe a novel, crowdsourcing method for generating a knowledge base of problem-medication pairs that takes advantage of manually asserted links between medications and problems.Through iterative review, we developed metrics to estimate the appropriateness of manually entered problem-medication links for inclusion in a knowledge base that can be used to infer previously unasserted links between problems and medications.Clinicians manually linked 231,223 medications (55.30% of prescribed medications) to problems within the electronic health record, generating 41,203 distinct problem-medication pairs, although not all were accurate. We developed methods to evaluate the accuracy of the pairs, and after limiting the pairs to those meeting an estimated 95% appropriateness threshold, 11,166 pairs remained. The pairs in the knowledge base accounted for 183,127 total links asserted (76.47% of all links). Retrospective application of the knowledge base linked 68,316 medications not previously linked by a clinician to an indicated problem (36.53% of unlinked medications). Expert review of the combined knowledge base, including inferred and manually linked problem-medication pairs, found a sensitivity of 65.8% and a specificity of 97.9%.Crowdsourcing is an effective, inexpensive method for generating a knowledge base of problem-medication pairs that is automatically mapped to local terminologies, up-to-date, and reflective of local prescribing practices and trends.

SUBMITTER: McCoy AB

PROVIDER: S-EPMC3422843 | biostudies-other | 2012 Sep-Oct

REPOSITORIES: biostudies-other

ACCESS DATA

Publications

Development and evaluation of a crowdsourcing methodology for knowledge base construction: identifying relationships between clinical problems and medications.

McCoy Allison B AB Wright Adam A Laxmisan Archana A Ottosen Madelene J MJ McCoy Jacob A JA Butten David D Sittig Dean F DF

Journal of the American Medical Informatics Association : JAMIA 20120512 5

<h4>Objective</h4>We describe a novel, crowdsourcing method for generating a knowledge base of problem-medication pairs that takes advantage of manually asserted links between medications and problems.<h4>Methods</h4>Through iterative review, we developed metrics to estimate the appropriateness of manually entered problem-medication links for inclusion in a knowledge base that can be used to infer previously unasserted links between problems and medications.<h4>Results</h4>Clinicians manually li ...[more]

PMID: 22582202

Similar Datasets

Project description:BackgroundOntologies are key enabling technologies for the Semantic Web. The Web Ontology Language (OWL) is a semantic markup language for publishing and sharing ontologies.ObjectiveThe supply of customizable, computable, and formally represented molecular genetics information and health information, via electronic health record (EHR) interfaces, can play a critical role in achieving precision medicine. In this study, we used cystic fibrosis as an example to build an Ontology-based Knowledge Base prototype on Cystic Fibrobis (OntoKBCF) to supply such information via an EHR prototype. In addition, we elaborate on the construction and representation principles, approaches, applications, and representation challenges that we faced in the construction of OntoKBCF. The principles and approaches can be referenced and applied in constructing other ontology-based domain knowledge bases.MethodsFirst, we defined the scope of OntoKBCF according to possible clinical information needs about cystic fibrosis on both a molecular level and a clinical phenotype level. We then selected the knowledge sources to be represented in OntoKBCF. We utilized top-to-bottom content analysis and bottom-up construction to build OntoKBCF. Protégé-OWL was used to construct OntoKBCF. The construction principles included (1) to use existing basic terms as much as possible; (2) to use intersection and combination in representations; (3) to represent as many different types of facts as possible; and (4) to provide 2-5 examples for each type. HermiT 1.3.8.413 within Protégé-5.1.0 was used to check the consistency of OntoKBCF.ResultsOntoKBCF was constructed successfully, with the inclusion of 408 classes, 35 properties, and 113 equivalent classes. OntoKBCF includes both atomic concepts (such as amino acid) and complex concepts (such as "adolescent female cystic fibrosis patient") and their descriptions. We demonstrated that OntoKBCF could make customizable molecular and health information available automatically and usable via an EHR prototype. The main challenges include the provision of a more comprehensive account of different patient groups as well as the representation of uncertain knowledge, ambiguous concepts, and negative statements and more complicated and detailed molecular mechanisms or pathway information about cystic fibrosis.ConclusionsAlthough cystic fibrosis is just one example, based on the current structure of OntoKBCF, it should be relatively straightforward to extend the prototype to cover different topics. Moreover, the principles underpinning its development could be reused for building alternative human monogenetic diseases knowledge bases.

Project description:BackgroundOne in five U.S. adults lives with some kind of mental health condition and 4.6% of all U.S. adults have a serious mental illness. The Internet has become the first place for these people to seek online mental health information for help. However, online mental health information is not well-organized and often of low quality. There have been efforts in building evidence-based mental health knowledgebases curated with information manually extracted from the high-quality scientific literature. Manual extraction is inefficient. Crowdsourcing can potentially be a low-cost mechanism to collect labeled data from non-expert laypeople. However, there is not an existing annotation tool integrated with popular crowdsourcing platforms to perform the information extraction tasks. In our previous work, we prototyped a Semantic Text Annotation Tool (STAT) to address this gap.ObjectiveWe aimed to refine the STAT prototype (1) to improve its usability and (2) to enhance the crowdsourcing workflow efficiency to facilitate the construction of evidence-based mental health knowledgebase, following a user-centered design (UCD) approach.MethodsFollowing UCD principles, we conducted four design iterations to improve the initial STAT prototype. In the first two iterations, usability testing focus groups were conducted internally with 8 participants recruited from a convenient sample, and the usability was evaluated with a modified System Usability Scale (SUS). In the following two iterations, usability testing was conducted externally using the Amazon Mechanical Turk (MTurk) platform. In each iteration, we summarized the usability testing results through thematic analysis, identified usability issues, and conducted a heuristic evaluation to map identified usability issues to Jakob Nielsen's usability heuristics. We collected suggested improvements in the usability testing sessions and enhanced STAT accordingly in the next UCD iteration. After four UCD iterations, we conducted a case study of the system on MTurk using mental health related scientific literature. We compared the performance of crowdsourcing workers with two expert annotators from two aspects: efficiency and quality.ResultsThe SUS score increased from 70.3 ± 12.5 to 81.1 ± 9.8 after the two internal UCD iterations as we improved STAT's functionality based on the suggested improvements. We then evaluated STAT externally through MTurk in the following two iterations. The SUS score decreased to 55.7 ± 20.1 in the third iteration, probably because of the complexity of the tasks. After further simplification of STAT and the annotation tasks with an improved annotation guideline, the SUS score increased to 73.8 ± 13.8 in the fourth iteration of UCD. In the evaluation case study, on average, the workers spent 125.5 ± 69.2 s on the onboarding tutorial and the crowdsourcing workers spent significantly less time on the annotation tasks compared to the two experts. In terms of annotation quality, the workers' annotation results achieved average F1-scores ranged from 0.62 to 0.84 for the different sentences.ConclusionsWe successfully developed a web-based semantic text annotation tool, STAT, to facilitate the curation of semantic web knowledgebases through four UCD iterations. The lessons learned from the UCD process could serve as a guide to further enhance STAT and the development and design of other crowdsourcing-based semantic text annotation tasks. Our study also showed that a well-organized, informative annotation guideline is as important as the annotation tool itself. Further, we learned that a crowdsourcing task should consist of multiple simple microtasks rather than a complicated task.

Project description:BACKGROUND:Achieving good medication adherence is a major challenge for patients with chronic conditions. Our study aimed to assess the Threshold for Unacceptable Risk of Non-adherence (TURN), defined as the threshold at which physicians consider the health risks incurred by patients due to medication non-adherence unacceptable, for the most commonly prescribed drugs in France. METHODS:We conducted an online study using a crowdsourcing approach among French general practitioners and specialists from September 2016 to August 2017. Physicians assessed the TURN for various levels of missed doses by evaluating a series of randomly presented clinical vignettes, each presenting a given medication with a given therapeutic indication. For each "drug-indication group" (i.e., all drugs from the same pharmacological class with a similar therapeutic indication): 1) we described the distribution of physicians' assessments, 2) we provided a summary estimate of the TURN, defined as the frequency of missed doses above which 75% of the physicians' assessments were located; 3) we computed the number of pill boxes reimbursed in France in 2016 to put our results into context. RESULTS:We collected a total of 5365 assessments from 544 physicians, each of whom evaluated a random sample among 528 distinct clinical vignettes. Estimates of the TURN varied widely across drug-indication groups, ranging from risk considered unacceptable with 1 daily dose missed per month (e.g., insulin for diabetes) to risk always considered acceptable (e.g., anti-dementia drugs). Drugs with an estimated TURN of over one missing daily dose per week represented 44.9% of the prescription volume of the medications assessed in our study. CONCLUSIONS:According to physicians, the impact of non-adherence may vary greatly. Patient-physician discussions on the variable consequences of non-adherence could lead to a paradigm shift by seeking to reach "optimal adherence" depending on drugs rather than unrealistic "perfect adherence" to all drugs.

Dataset Information

Development and evaluation of a crowdsourcing methodology for knowledge base construction: identifying relationships between clinical problems and medications.

Publications

Development and evaluation of a crowdsourcing methodology for knowledge base construction: identifying relationships between clinical problems and medications.

Similar Datasets

OmicsDI is part of the ELIXIR infrastructure