Dataset Information

CSAR benchmark exercise of 2010: combined evaluation across all submitted scoring functions.

ABSTRACT: As part of the Community Structure-Activity Resource (CSAR) center, a set of 343 high-quality, protein-ligand crystal structures were assembled with experimentally determined K(d) or K(i) information from the literature. We encouraged the community to score the crystallographic poses of the complexes by any method of their choice. The goal of the exercise was to (1) evaluate the current ability of the field to predict activity from structure and (2) investigate the properties of the complexes and methods that appear to hinder scoring. A total of 19 different methods were submitted with numerous parameter variations for a total of 64 sets of scores from 16 participating groups. Linear regression and nonparametric tests were used to correlate scores to the experimental values. Correlation to experiment for the various methods ranged R(2) = 0.58-0.12, Spearman ? = 0.74-0.37, Kendall ? = 0.55-0.25, and median unsigned error = 1.00-1.68 pK(d) units. All types of scoring functions-force field based, knowledge based, and empirical-had examples with high and low correlation, showing no bias/advantage for any particular approach. The data across all the participants were combined to identify 63 complexes that were poorly scored across the majority of the scoring methods and 123 complexes that were scored well across the majority. The two sets were compared using a Wilcoxon rank-sum test to assess any significant difference in the distributions of >400 physicochemical properties of the ligands and the proteins. Poorly scored complexes were found to have ligands that were the same size as those in well-scored complexes, but hydrogen bonding and torsional strain were significantly different. These comparisons point to a need for CSAR to develop data sets of congeneric series with a range of hydrogen-bonding and hydrophobic characteristics and a range of rotatable bonds.

SUBMITTER: Smith RD

PROVIDER: S-EPMC3186041 | biostudies-literature | 2011 Sep

REPOSITORIES: biostudies-literature

ACCESS DATA

Publications

CSAR benchmark exercise of 2010: combined evaluation across all submitted scoring functions.

Smith Richard D RD Dunbar James B JB Ung Peter Man-Un PM Esposito Emilio X EX Yang Chao-Yie CY Wang Shaomeng S Carlson Heather A HA

Journal of chemical information and modeling 20110829 9

As part of the Community Structure-Activity Resource (CSAR) center, a set of 343 high-quality, protein-ligand crystal structures were assembled with experimentally determined K(d) or K(i) information from the literature. We encouraged the community to score the crystallographic poses of the complexes by any method of their choice. The goal of the exercise was to (1) evaluate the current ability of the field to predict activity from structure and (2) investigate the properties of the complexes an ...[more]

PMID: 21809884

Similar Datasets

Project description:The Community Structure-Activity Resource (CSAR) recently held its first blinded exercise based on data provided by Abbott, Vertex, and colleagues at the University of Michigan, Ann Arbor. A total of 20 research groups submitted results for the benchmark exercise where the goal was to compare different improvements for pose prediction, enrichment, and relative ranking of congeneric series of compounds. The exercise was built around blinded high-quality experimental data from four protein targets: LpxC, Urokinase, Chk1, and Erk2. Pose prediction proved to be the most straightforward task, and most methods were able to successfully reproduce binding poses when the crystal structure employed was co-crystallized with a ligand from the same chemical series. Multiple evaluation metrics were examined, and we found that RMSD and native contact metrics together provide a robust evaluation of the predicted poses. It was notable that most scoring functions underpredicted contacts between the hetero atoms (i.e., N, O, S, etc.) of the protein and ligand. Relative ranking was found to be the most difficult area for the methods, but many of the scoring functions were able to properly identify Urokinase actives from the inactives in the series. Lastly, we found that minimizing the protein and correcting histidine tautomeric states positively trended with low RMSD for pose prediction but minimizing the ligand negatively trended. Pregenerated ligand conformations performed better than those that were generated on the fly. Optimizing docking parameters and pretraining with the native ligand had a positive effect on the docking performance as did using restraints, substructure fitting, and shape fitting. Lastly, for both sampling and ranking scoring functions, the use of the empirical scoring function appeared to trend positively with the RMSD. Here, by combining the results of many methods, we hope to provide a statistically relevant evaluation and elucidate specific shortcomings of docking methodology for the community.

Dataset Information

CSAR benchmark exercise of 2010: combined evaluation across all submitted scoring functions.

Publications

CSAR benchmark exercise of 2010: combined evaluation across all submitted scoring functions.

Similar Datasets

OmicsDI is part of the ELIXIR infrastructure

Tweets