Dataset Information

A close look at protein function prediction evaluation protocols.

ABSTRACT: BACKGROUND:The recently held Critical Assessment of Function Annotation challenge (CAFA2) required its participants to submit predictions for a large number of target proteins regardless of whether they have previous annotations or not. This is in contrast to the original CAFA challenge in which participants were asked to submit predictions for proteins with no existing annotations. The CAFA2 task is more realistic, in that it more closely mimics the accumulation of annotations over time. In this study we compare these tasks in terms of their difficulty, and determine whether cross-validation provides a good estimate of performance. RESULTS:The CAFA2 task is a combination of two subtasks: making predictions on annotated proteins and making predictions on previously unannotated proteins. In this study we analyze the performance of several function prediction methods in these two scenarios. Our results show that several methods (structured support vector machine, binary support vector machines and guilt-by-association methods) do not usually achieve the same level of accuracy on these two tasks as that achieved by cross-validation, and that predicting novel annotations for previously annotated proteins is a harder problem than predicting annotations for uncharacterized proteins. We also find that different methods have different performance characteristics in these tasks, and that cross-validation is not adequate at estimating performance and ranking methods. CONCLUSIONS:These results have implications for the design of computational experiments in the area of automated function prediction and can provide useful insight for the understanding and design of future CAFA competitions.

SUBMITTER: Kahanda I

PROVIDER: S-EPMC4570743 | biostudies-literature | 2015

REPOSITORIES: biostudies-literature

ACCESS DATA

Publications

A close look at protein function prediction evaluation protocols.

Kahanda Indika I Funk Christopher S CS Ullah Fahad F Verspoor Karin M KM Ben-Hur Asa A

GigaScience 20150914

<h4>Background</h4>The recently held Critical Assessment of Function Annotation challenge (CAFA2) required its participants to submit predictions for a large number of target proteins regardless of whether they have previous annotations or not. This is in contrast to the original CAFA challenge in which participants were asked to submit predictions for proteins with no existing annotations. The CAFA2 task is more realistic, in that it more closely mimics the accumulation of annotations over time ...[more]

PMID: 26380075

Similar Datasets

Project description:?-Lactamases are hydrolytic enzymes capable of opening the ?-lactam ring of antibiotics such as penicillin, thus endowing the bacteria that produce them with antibiotic resistance. Of particular medical concern are metallo-?-lactamases (MBLs), with an active site built around coordinated Zn cations. MBLs are pan-reactive enzymes that can break down almost all classes of ?-lactams, including such last-resort antibiotics as carbapenems. They are not only broad-spectrum-reactive but are often plasmid-borne (e.g., the New Delhi enzyme, NDM), and can spread horizontally even among unrelated bacteria. Acquired MBLs are encoded by mobile genetic elements, which often include other resistance genes, making the microbiological situation particularly alarming. There is an urgent need to develop MBL inhibitors in order to rescue our antibiotic armory. A number of such efforts have been undertaken, most notably using the 3D structures of various MBLs as drug-design targets. Structure-guided drug discovery depends on the quality of the structures that are collected in the Protein Data Bank (PDB) and on the consistency of the information in dedicated ?-lactamase databases. We conducted a careful review of the crystal structures of class B ?-lactamases, concluding that the quality of these structures varies widely, especially in the regions where small molecules interact with the macromolecules. In a number of examples the interpretation of the bound ligands (e.g., inhibitors, substrate/product analogs) is doubtful or even incorrect, and it appears that in some cases the modeling of ligands was not supported by electron density. For ten MBL structures, alternative interpretations of the original diffraction data could be proposed and the new models have been deposited in the PDB. In four cases, these models, prepared jointly with the authors of the original depositions, superseded the previous deposits. This review emphasizes the importance of critical assessment of structural models describing key drug design targets at the level of the raw experimental data. Since the structures reviewed here are the basis for ongoing design of new MBL inhibitors, it is important to identify and correct the problems with ambiguous crystallographic interpretations, thus enhancing reproducibility in this highly medically relevant area.

Dataset Information

A close look at protein function prediction evaluation protocols.

Publications

A close look at protein function prediction evaluation protocols.

Similar Datasets

OmicsDI is part of the ELIXIR infrastructure

Tweets