A realistic assessment of methods for extracting gene/protein interactions from free text.
Ontology highlight
ABSTRACT: Background
The automated extraction of gene and/or protein interactions from the literature is one of the most important targets of biomedical text mining research. In this paper we present a realistic evaluation of gene/protein interaction mining relevant to potential non-specialist users. Hence we have specifically avoided methods that are complex to install or require reimplementation, and we coupled our chosen extraction methods with a state-of-the-art biomedical named entity tagger.Results
Our results show: that performance across different evaluation corpora is extremely variable; that the use of tagged (as opposed to gold standard) gene and protein names has a significant impact on performance, with a drop in F-score of over 20 percentage points being commonplace; and that a simple keyword-based benchmark algorithm when coupled with a named entity tagger outperforms two of the tools most widely used to extract gene/protein interactions.Conclusion
In terms of availability, ease of use and performance, the potential non-specialist user community interested in automatically extracting gene and/or protein interactions from free text is poorly served by current tools and systems. The public release of extraction tools that are easy to install and use, and that achieve state-of-art levels of performance should be treated as a high priority by the biomedical text mining community.
SUBMITTER: Kabiljo R
PROVIDER: S-EPMC2723093 | biostudies-literature |
REPOSITORIES: biostudies-literature
Similar Datasets
Project description:Social animals exhibit collective behavior whereby they negotiate to reach an agreement, such as the coordination of group motion. Bats are unique among most social animals, since they use active sensory echolocation by emitting ultrasonic waves and sensing echoes to navigate. Bats' use of active sensing may result in acoustic interference from peers, driving different behavior when they fly together rather than alone. The present study explores quantitative methods that can be used to understand whether bats flying in pairs move independently of each other or interact. The study used field data from bats in flight and is based on the assumption that interactions between two bats are evidenced in their flight patterns. To quantify pairwise interaction, we defined the strength of coupling using model-free methods from dynamical systems and information theory. We used a control condition to eliminate similarities in flight path due to environmental geometry. Our research question is whether these data-driven methods identify directed coupling between bats from their flight paths and, if so, whether the results are consistent between methods. Results demonstrate evidence of information exchange between flying bat pairs, and, in particular, we find significant evidence of rear-to-front coupling in bats' turning behavior when they fly in the absence of obstacles.
| S-EPMC7514148 | biostudies-literature
Project description:Relations between chemicals and diseases are one of the most queried biomedical interactions. Although expert manual curation is the standard method for extracting these relations from the literature, it is expensive and impractical to apply to large numbers of documents, and therefore alternative methods are required. We describe here a crowdsourcing workflow for extracting chemical-induced disease relations from free text as part of the BioCreative V Chemical Disease Relation challenge. Five non-expert workers on the CrowdFlower platform were shown each potential chemical-induced disease relation highlighted in the original source text and asked to make binary judgments about whether the text supported the relation. Worker responses were aggregated through voting, and relations receiving four or more votes were predicted as true. On the official evaluation dataset of 500 PubMed abstracts, the crowd attained a 0.505F-score (0.475 precision, 0.540 recall), with a maximum theoretical recall of 0.751 due to errors with named entity recognition. The total crowdsourcing cost was $1290.67 ($2.58 per abstract) and took a total of 7 h. A qualitative error analysis revealed that 46.66% of sampled errors were due to task limitations and gold standard errors, indicating that performance can still be improved. All code and results are publicly available athttps://github.com/SuLab/crowd_cid_relexDatabase URL:https://github.com/SuLab/crowd_cid_relex.
| S-EPMC4834205 | biostudies-literature
Project description:The Third i2b2 Workshop on Natural Language Processing Challenges for Clinical Records focused on the identification of medications, their dosages, modes (routes) of administration, frequencies, durations, and reasons for administration in discharge summaries. This challenge is referred to as the medication challenge. For the medication challenge, i2b2 released detailed annotation guidelines along with a set of annotated discharge summaries. Twenty teams representing 23 organizations and nine countries participated in the medication challenge. The teams produced rule-based, machine learning, and hybrid systems targeted to the task. Although rule-based systems dominated the top 10, the best performing system was a hybrid. Of all medication-related fields, durations and reasons were the most difficult for all systems to detect. While medications themselves were identified with better than 0.75 F-measure by all of the top 10 systems, the best F-measure for durations and reasons were 0.525 and 0.459, respectively. State-of-the-art natural language processing systems go a long way toward extracting medication names, dosages, modes, and frequencies. However, they are limited in recognizing duration and reason fields and would benefit from future research.
| S-EPMC2995677 | biostudies-literature
Project description:Evaluating patients' experience and satisfaction often calls for analyses of free-text data. Language and domain-specific information extraction can reduce costly manual preprocessing and enable the analysis of extensive collections of experience-based narratives. The research aims were to (1) elicit free-text narratives about experiences with health services of international students in Poland, (2) develop domain- and language-specific algorithms for the extraction of information relevant for the evaluation of quality and safety of health services, and (3) test the performance of information extraction algorithms' on questions about the patients' experiences with health services. The materials were free-text narratives about health clinic encounters produced by English-speaking foreigners recalling their experiences (n = 104) in healthcare facilities in Poland. A linguistic analysis of the text collection led to constructing a semantic-syntactic lexicon and a set of lexical-syntactic frames. These were further used to develop rule-based information extraction algorithms in the form of Python scripts. The extraction algorithms generated text classifications according to predefined queries. In addition, the narratives were classified by human readers. The algorithm-based and the human readers' classifications were highly correlated and significant (p < 0.01), indicating an excellent performance of the automatic query algorithms. The study results demonstrate that domain-specific and language-specific information extraction from free-text narratives can be used as an efficient and low-cost method for evaluating patient experiences and satisfaction with health services and built into software solutions for the quality evaluation in health care.
| S-EPMC9408527 | biostudies-literature
Project description:BackgroundEnzymatic and chemical reactions are key for understanding biological processes in cells. Curated databases of chemical reactions exist but these databases struggle to keep up with the exponential growth of the biomedical literature. Conventional text mining pipelines provide tools to automatically extract entities and relationships from the scientific literature, and partially replace expert curation, but such machine learning frameworks often require a large amount of labeled training data and thus lack scalability for both larger document corpora and new relationship types.ResultsWe developed an application of Snorkel, a weakly supervised learning framework, for extracting chemical reaction relationships from biomedical literature abstracts. For this work, we defined a chemical reaction relationship as the transformation of chemical A to chemical B. We built and evaluated our system on small annotated sets of chemical reaction relationships from two corpora: curated bacteria-related abstracts from the MetaCyc database (MetaCyc_Corpus) and a more general set of abstracts annotated with MeSH (Medical Subject Headings) term Bacteria (Bacteria_Corpus; a superset of MetaCyc_Corpus). For the MetaCyc_Corpus, we obtained 84% precision and 41% recall (55% F1 score). Extending to the more general Bacteria_Corpus decreased precision to 62% with only a four-point drop in recall to 37% (46% F1 score). Overall, the Bacteria_Corpus contained two orders of magnitude more candidate chemical reaction relationships (nine million candidates vs 68,0000 candidates) and had a larger class imbalance (2.5% positives vs 5% positives) as compared to the MetaCyc_Corpus. In total, we extracted 6871 chemical reaction relationships from nine million candidates in the Bacteria_Corpus.ConclusionsWith this work, we built a database of chemical reaction relationships from almost 900,000 scientific abstracts without a large training set of labeled annotations. Further, we showed the generalizability of our initial application built on MetaCyc documents enriched with chemical reactions to a general set of articles related to bacteria.
| S-EPMC7251675 | biostudies-literature
Project description:We develop a simple but rigorous model of protein-protein association kinetics based on diffusional association on free energy landscapes obtained by sampling configurations within and surrounding the native complex binding funnels. Guided by results obtained on exactly solvable model problems, we transform the problem of diffusion in a potential into free diffusion in the presence of an absorbing zone spanning the entrance to the binding funnel. The free diffusion problem is solved using a recently derived analytic expression for the rate of association of asymmetrically oriented molecules. Despite the required high steric specificity and the absence of long-range attractive interactions, the computed rates are typically on the order of 10(4)-10(6) M(-1) sec(-1), several orders of magnitude higher than rates obtained using a purely probabilistic model in which the association rate for free diffusion of uniformly reactive molecules is multiplied by the probability of a correct alignment of the two partners in a random collision. As the association rates of many protein-protein complexes are also in the 10(5)-10(6) M(-1) sec(-1) range, our results suggest that free energy barriers arising from desolvation and/or side-chain freezing during complex formation or increased ruggedness within the binding funnel, which are completely neglected in our simple diffusional model, do not contribute significantly to the dynamics of protein-protein association. The transparent physical interpretation of our approach that computes association rates directly from the size and geometry of protein-protein binding funnels makes it a useful complement to Brownian dynamics simulations.
| S-EPMC2279981 | biostudies-literature
Project description:BackgroundIncreasingly biological text mining research is focusing on the extraction of complex relationships relevant to the construction and curation of biological networks and pathways. However, one important category of pathway - metabolic pathways - has been largely neglected.Here we present a relatively simple method for extracting metabolic reaction information from free text that scores different permutations of assigned entities (enzymes and metabolites) within a given sentence based on the presence and location of stemmed keywords. This method extends an approach that has proved effective in the context of the extraction of protein-protein interactions.ResultsWhen evaluated on a set of manually-curated metabolic pathways using standard performance criteria, our method performs surprisingly well. Precision and recall rates are comparable to those previously achieved for the well-known protein-protein interaction extraction task.ConclusionsWe conclude that automated metabolic pathway construction is more tractable than has often been assumed, and that (as in the case of protein-protein interaction extraction) relatively simple text-mining approaches can prove surprisingly effective. It is hoped that these results will provide an impetus to further research and act as a useful benchmark for judging the performance of more sophisticated methods that are yet to be developed.
| S-EPMC3475109 | biostudies-literature
Project description:ObjectivesLifetime occupational history (OH) questionnaires often use open-ended questions to capture detailed information about study participants' jobs. Exposure assessors use this information, along with responses to job- and industry-specific questionnaires, to assign exposure estimates on a job-by-job basis. An alternative approach is to use information from the OH responses and the job- and industry-specific questionnaires to develop programmable decision rules for assigning exposures. As a first step in this process, we developed a systematic approach to extract the free-text OH responses and convert them into standardized variables that represented exposure scenarios.MethodsOur study population comprised 2408 subjects, reporting 11991 jobs, from a case-control study of renal cell carcinoma. Each subject completed a lifetime OH questionnaire that included verbatim responses, for each job, to open-ended questions including job title, main tasks and activities (task), tools and equipment used (tools), and chemicals and materials handled (chemicals). Based on a review of the literature, we identified exposure scenarios (occupations, industries, tasks/tools/chemicals) expected to involve possible exposure to chlorinated solvents, trichloroethylene (TCE) in particular, lead, and cadmium. We then used a SAS macro to review the information reported by study participants to identify jobs associated with each exposure scenario; this was done using previously coded standardized occupation and industry classification codes, and a priori lists of associated key words and phrases related to possibly exposed tasks, tools, and chemicals. Exposure variables representing the occupation, industry, and task/tool/chemicals exposure scenarios were added to the work history records of the study respondents. Our identification of possibly TCE-exposed scenarios in the OH responses was compared to an expert's independently assigned probability ratings to evaluate whether we missed identifying possibly exposed jobs.ResultsOur process added exposure variables for 52 occupation groups, 43 industry groups, and 46 task/tool/chemical scenarios to the data set of OH responses. Across all four agents, we identified possibly exposed task/tool/chemical exposure scenarios in 44-51% of the jobs in possibly exposed occupations. Possibly exposed task/tool/chemical exposure scenarios were found in a nontrivial 9-14% of the jobs not in possibly exposed occupations, suggesting that our process identified important information that would not be captured using occupation alone. Our extraction process was sensitive: for jobs where our extraction of OH responses identified no exposure scenarios and for which the sole source of information was the OH responses, only 0.1% were assessed as possibly exposed to TCE by the expert.ConclusionsOur systematic extraction of OH information found useful information in the task/chemicals/tools responses that was relatively easy to extract and that was not available from the occupational or industry information. The extracted variables can be used as inputs in the development of decision rules, especially for jobs where no additional information, such as job- and industry-specific questionnaires, is available.
| S-EPMC4053931 | biostudies-literature
Project description:Background and objectivePercutaneous coronary intervention (PCI) using drug-eluting stents (DES) is an indispensable treatment for coronary artery disease. However, to evaluate the performance of various types of stents for PCI, numerous resources are required. We extracted clinical information from free-text records and, using practice-based evidence, compared the efficacy of various DES.Materials and methodsWe developed a text mining tool based on regular expression and applied it to PCI reports stored in the electronic health records (EHRs) of Ajou University Hospital from 2010-2014. The PCI data were extracted from EHRs with a sensitivity of 0.996, a specificity of 1.000, and an F-measure of 0.995 when compared with a sample of 200 reports. Using these data, we compared the performance of stents by Kaplan-Meier analysis and the Cox hazard proportional regression.ResultsIn the self-validation analysis comparing the first-generation to the second-generation DES, the second-generation DES was superior to the first-generation DES (hazard ratio [HR]: 0.423, 95% confidence interval [CI]: 0.284-0.630) in terms of target vessel revascularization (TVR), showing similar findings to the established results of previous studies. Among the second-generation DES, the biodegradable-polymer DES tended to be superior, with a risk of TVR (HR: 0.568, 95% CI: 0.281-1.147) falling below than that for the durable-polymer DES approximately 1 year after the index procedure. The Endeavor stent had the highest TVR risk among the newer generation DES (HR: 2.576, 95% CI: 1.273-5.210).ConclusionsIn this study, we demonstrated how to construct a PCI data warehouse of PCI-related parameters obtained from free-text electronic records with high accuracy for use in the post surveillance of coronary stents in a time- and cost effective manner. Post surveillance of the practice based evidence in the PCI data warehouse indicated that the biodegradable-polymer DES might have a lower risk of TVR than the durable-polymer DES.
| S-EPMC5553787 | biostudies-literature
Project description:BACKGROUND: Functional annotation of proteins remains a challenging task. Currently the scientific literature serves as the main source for yet uncurated functional annotations, but curation work is slow and expensive. Automatic techniques that support this work are still lacking reliability. We developed a method to identify conserved protein interaction graphs and to predict missing protein functions from orthologs in these graphs. To enhance the precision of the results, we furthermore implemented a procedure that validates all predictions based on findings reported in the literature. RESULTS: Using this procedure, more than 80% of the GO annotations for proteins with highly conserved orthologs that are available in UniProtKb/Swiss-Prot could be verified automatically. For a subset of proteins we predicted new GO annotations that were not available in UniProtKb/Swiss-Prot. All predictions were correct (100% precision) according to the verifications from a trained curator. CONCLUSION: Our method of integrating CCSs and literature mining is thus a highly reliable approach to predict GO annotations for weakly characterized proteins with orthologs.
| S-EPMC2500093 | biostudies-literature