Dataset Information

Critical Assessment of Metagenome Interpretation-a benchmark of metagenomics software.

ABSTRACT: Methods for assembly, taxonomic profiling and binning are key to interpreting metagenome data, but a lack of consensus about benchmarking complicates performance assessment. The Critical Assessment of Metagenome Interpretation (CAMI) challenge has engaged the global developer community to benchmark their programs on highly complex and realistic data sets, generated from ?700 newly sequenced microorganisms and ?600 novel viruses and plasmids and representing common experimental setups. Assembly and genome binning programs performed well for species represented by individual genomes but were substantially affected by the presence of related strains. Taxonomic profiling and binning programs were proficient at high taxonomic ranks, with a notable performance decrease below family level. Parameter settings markedly affected performance, underscoring their importance for program reproducibility. The CAMI results highlight current challenges but also provide a roadmap for software selection to answer specific research questions.

SUBMITTER: Sczyrba A

PROVIDER: S-EPMC5903868 | biostudies-literature | 2017 Nov

REPOSITORIES: biostudies-literature

ACCESS DATA

Publications

Critical Assessment of Metagenome Interpretation-a benchmark of metagenomics software.

Sczyrba Alexander A Hofmann Peter P Belmann Peter P Koslicki David D Janssen Stefan S Dröge Johannes J Gregor Ivan I Majda Stephan S Fiedler Jessika J Dahms Eik E Bremges Andreas A Fritz Adrian A Garrido-Oter Ruben R Jørgensen Tue Sparholt TS Shapiro Nicole N Blood Philip D PD Gurevich Alexey A Bai Yang Y Turaev Dmitrij D DeMaere Matthew Z MZ Chikhi Rayan R Nagarajan Niranjan N Quince Christopher C Meyer Fernando F Balvočiūtė Monika M Hansen Lars Hestbjerg LH Sørensen Søren J SJ Chia Burton K H BKH Denis Bertrand B Froula Jeff L JL Wang Zhong Z Egan Robert R Don Kang Dongwan D Cook Jeffrey J JJ Deltel Charles C Beckstette Michael M Lemaitre Claire C Peterlongo Pierre P Rizk Guillaume G Lavenier Dominique D Wu Yu-Wei YW Singer Steven W SW Jain Chirag C Strous Marc M Klingenberg Heiner H Meinicke Peter P Barton Michael D MD Lingner Thomas T Lin Hsin-Hung HH Liao Yu-Chieh YC Silva Genivaldo Gueiros Z GGZ Cuevas Daniel A DA Edwards Robert A RA Saha Surya S Piro Vitor C VC Renard Bernhard Y BY Pop Mihai M Klenk Hans-Peter HP Göker Markus M Kyrpides Nikos C NC Woyke Tanja T Vorholt Julia A JA Schulze-Lefert Paul P Rubin Edward M EM Darling Aaron E AE Rattei Thomas T McHardy Alice C AC

Nature methods 20171002 11

Methods for assembly, taxonomic profiling and binning are key to interpreting metagenome data, but a lack of consensus about benchmarking complicates performance assessment. The Critical Assessment of Metagenome Interpretation (CAMI) challenge has engaged the global developer community to benchmark their programs on highly complex and realistic data sets, generated from ∼700 newly sequenced microorganisms and ∼600 novel viruses and plasmids and representing common experimental setups. Assembly a ...[more]

PMID: 28967888

Similar Datasets

Project description:BACKGROUND: Linear motifs (LMs) are abundant short regulatory sites used for modulating the functions of many eukaryotic proteins. They play important roles in post-translational modification, cell compartment targeting, docking sites for regulatory complex assembly and protein processing and cleavage. Methods for LM detection are now being developed that are strongly dependent on scores for motif conservation in homologous proteins. However, most LMs are found in natively disordered polypeptide segments that evolve rapidly, unhindered by structural constraints on the sequence. These regions of modular proteins are difficult to align using classical multiple sequence alignment programs that are specifically optimised to align the globular domains. As a consequence, poor motif alignment quality is hindering efforts to detect new LMs. RESULTS: We have developed a new benchmark, as part of the BAliBASE suite, designed to assess the ability of standard multiple alignment methods to detect and align LMs. The reference alignments are organised into different test sets representing real alignment problems and contain examples of experimentally verified functional motifs, extracted from the Eukaryotic Linear Motif (ELM) database. The benchmark has been used to evaluate and compare a number of multiple alignment programs. With distantly related proteins, the worst alignment program correctly aligns 48% of LMs compared to 73% for the best program. However, the performance of all the programs is adversely affected by the introduction of other sequences containing false positive motifs. The ranking of the alignment programs based on LM alignment quality is similar to that observed when considering full-length protein alignments, however little correlation was observed between LM and overall alignment quality for individual alignment test cases. CONCLUSION: We have shown that none of the programs currently available is capable of reliably aligning LMs in distantly related sequences and we have highlighted a number of specific problems. The results of the tests suggest possible ways to improve program accuracy for difficult, divergent sequences.

Dataset Information

Critical Assessment of Metagenome Interpretation-a benchmark of metagenomics software.

Publications

Critical Assessment of Metagenome Interpretation-a benchmark of metagenomics software.

Similar Datasets

OmicsDI is part of the ELIXIR infrastructure

Tweets