Dataset Information

Critical Assessment of Metagenome Interpretation: the second round of challenges.

ABSTRACT: Evaluating metagenomic software is key for optimizing metagenome interpretation and focus of the Initiative for the Critical Assessment of Metagenome Interpretation (CAMI). The CAMI II challenge engaged the community to assess methods on realistic and complex datasets with long- and short-read sequences, created computationally from around 1,700 new and known genomes, as well as 600 new plasmids and viruses. Here we analyze 5,002 results by 76 program versions. Substantial improvements were seen in assembly, some due to long-read data. Related strains still were challenging for assembly and genome recovery through binning, as was assembly quality for the latter. Profilers markedly matured, with taxon profilers and binners excelling at higher bacterial ranks, but underperforming for viruses and Archaea. Clinical pathogen detection results revealed a need to improve reproducibility. Runtime and memory usage analyses identified efficient programs, including top performers with other metrics. The results identify challenges and guide researchers in selecting methods for analyses.

SUBMITTER: Meyer F

PROVIDER: S-EPMC9007738 | biostudies-literature | 2022 Apr

REPOSITORIES: biostudies-literature

ACCESS DATA

Publications

Critical Assessment of Metagenome Interpretation: the second round of challenges.

Meyer Fernando F Fritz Adrian A Deng Zhi-Luo ZL Koslicki David D Lesker Till Robin TR Gurevich Alexey A Robertson Gary G Alser Mohammed M Antipov Dmitry D Beghini Francesco F Bertrand Denis D Brito Jaqueline J JJ Brown C Titus CT Buchmann Jan J Buluç Aydin A Chen Bo B Chikhi Rayan R Clausen Philip T L C PTLC Cristian Alexandru A Dabrowski Piotr Wojciech PW Darling Aaron E AE Egan Rob R Eskin Eleazar E Georganas Evangelos E Goltsman Eugene E Gray Melissa A MA Hansen Lars Hestbjerg LH Hofmeyr Steven S Huang Pingqin P Irber Luiz L Jia Huijue H Jørgensen Tue Sparholt TS Kieser Silas D SD Klemetsen Terje T Kola Axel A Kolmogorov Mikhail M Korobeynikov Anton A Kwan Jason J LaPierre Nathan N Lemaitre Claire C Li Chenhao C Limasset Antoine A Malcher-Miranda Fabio F Mangul Serghei S Marcelino Vanessa R VR Marchet Camille C Marijon Pierre P Meleshko Dmitry D Mende Daniel R DR Milanese Alessio A Nagarajan Niranjan N Nissen Jakob J Nurk Sergey S Oliker Leonid L Paoli Lucas L Peterlongo Pierre P Piro Vitor C VC Porter Jacob S JS Rasmussen Simon S Rees Evan R ER Reinert Knut K Renard Bernhard B Robertsen Espen Mikal EM Rosen Gail L GL Ruscheweyh Hans-Joachim HJ Sarwal Varuni V Segata Nicola N Seiler Enrico E Shi Lizhen L Sun Fengzhu F Sunagawa Shinichi S Sørensen Søren Johannes SJ Thomas Ashleigh A Tong Chengxuan C Trajkovski Mirko M Tremblay Julien J Uritskiy Gherman G Vicedomini Riccardo R Wang Zhengyang Z Wang Ziye Z Wang Zhong Z Warren Andrew A Willassen Nils Peder NP Yelick Katherine K You Ronghui R Zeller Georg G Zhao Zhengqiao Z Zhu Shanfeng S Zhu Jie J Garrido-Oter Ruben R Gastmeier Petra P Hacquard Stephane S Häußler Susanne S Khaledi Ariane A Maechler Friederike F Mesny Fantin F Radutoiu Simona S Schulze-Lefert Paul P Smit Nathiana N Strowig Till T Bremges Andreas A Sczyrba Alexander A McHardy Alice Carolyn AC

Nature methods 20220408 4

Evaluating metagenomic software is key for optimizing metagenome interpretation and focus of the Initiative for the Critical Assessment of Metagenome Interpretation (CAMI). The CAMI II challenge engaged the community to assess methods on realistic and complex datasets with long- and short-read sequences, created computationally from around 1,700 new and known genomes, as well as 600 new plasmids and viruses. Here we analyze 5,002 results by 76 program versions. Substantial improvements were seen ...[more]

PMID: 35396482

Similar Datasets

Project description:Computing protein structure from amino acid sequence information has been a long-standing grand challenge. Critical assessment of structure prediction (CASP) conducts community experiments aimed at advancing solutions to this and related problems. Experiments are conducted every 2 years. The 2020 experiment (CASP14) saw major progress, with the second generation of deep learning methods delivering accuracy comparable with experiment for many single proteins. There is an expectation that these methods will have much wider application in computational structural biology. Here we summarize results from the most recent experiment, CASP15, in 2022, with an emphasis on new deep learning-driven progress. Other papers in this special issue of proteins provide more detailed analysis. For single protein structures, the AlphaFold2 deep learning method is still superior to other approaches, but there are two points of note. First, although AlphaFold2 was the core of all the most successful methods, there was a wide variety of implementation and combination with other methods. Second, using the standard AlphaFold2 protocol and default parameters only produces the highest quality result for about two thirds of the targets, and more extensive sampling is required for the others. The major advance in this CASP is the enormous increase in the accuracy of computed protein complexes, achieved by the use of deep learning methods, although overall these do not fully match the performance for single proteins. Here too, AlphaFold2 based method perform best, and again more extensive sampling than the defaults is often required. Also of note are the encouraging early results on the use of deep learning to compute ensembles of macromolecular structures. Critically for the usability of computed structures, for both single proteins and protein complexes, deep learning derived estimates of both local and global accuracy are of high quality, however the estimates in interface regions are slightly less reliable. CASP15 also included computation of RNA structures for the first time. Here, the classical approaches produced better agreement with experiment than the new deep learning ones, and accuracy is limited. Also, for the first time, CASP included the computation of protein-ligand complexes, an area of special interest for drug design. Here too, classical methods were still superior to deep learning ones. Many new approaches were discussed at the CASP conference, and it is clear methods will continue to advance.

Dataset Information

Critical Assessment of Metagenome Interpretation: the second round of challenges.

Publications

Critical Assessment of Metagenome Interpretation: the second round of challenges.

Similar Datasets

OmicsDI is part of the ELIXIR infrastructure

Tweets