Dataset Information

METhodological RadiomICs Score (METRICS): a quality scoring tool for radiomics research endorsed by EuSoMII.

ABSTRACT:

Purpose

To propose a new quality scoring tool, METhodological RadiomICs Score (METRICS), to assess and improve research quality of radiomics studies.

Methods

We conducted an online modified Delphi study with a group of international experts. It was performed in three consecutive stages: Stage#1, item preparation; Stage#2, panel discussion among EuSoMII Auditing Group members to identify the items to be voted; and Stage#3, four rounds of the modified Delphi exercise by panelists to determine the items eligible for the METRICS and their weights. The consensus threshold was 75%. Based on the median ranks derived from expert panel opinion and their rank-sum based conversion to importance scores, the category and item weights were calculated.

Result

In total, 59 panelists from 19 countries participated in selection and ranking of the items and categories. Final METRICS tool included 30 items within 9 categories. According to their weights, the categories were in descending order of importance: study design, imaging data, image processing and feature extraction, metrics and comparison, testing, feature processing, preparation for modeling, segmentation, and open science. A web application and a repository were developed to streamline the calculation of the METRICS score and to collect feedback from the radiomics community.

Conclusion

In this work, we developed a scoring tool for assessing the methodological quality of the radiomics research, with a large international panel and a modified Delphi protocol. With its conditional format to cover methodological variations, it provides a well-constructed framework for the key methodological concepts to assess the quality of radiomic research papers.

Critical relevance statement

A quality assessment tool, METhodological RadiomICs Score (METRICS), is made available by a large group of international domain experts, with transparent methodology, aiming at evaluating and improving research quality in radiomics and machine learning.

Key points

• A methodological scoring tool, METRICS, was developed for assessing the quality of radiomics research, with a large international expert panel and a modified Delphi protocol. • The proposed scoring tool presents expert opinion-based importance weights of categories and items with a transparent methodology for the first time. • METRICS accounts for varying use cases, from handcrafted radiomics to entirely deep learning-based pipelines. • A web application has been developed to help with the calculation of the METRICS score ( https://metricsscore.github.io/metrics/METRICS.html ) and a repository created to collect feedback from the radiomics community ( https://github.com/metricsscore/metrics ).

SUBMITTER: Kocak B

PROVIDER: S-EPMC10792137 | biostudies-literature | 2024 Jan

REPOSITORIES: biostudies-literature

ACCESS DATA

Publications

METhodological RadiomICs Score (METRICS): a quality scoring tool for radiomics research endorsed by EuSoMII.

Kocak Burak B Akinci D'Antonoli Tugba T Mercaldo Nathaniel N Alberich-Bayarri Angel A Baessler Bettina B Ambrosini Ilaria I Andreychenko Anna E AE Bakas Spyridon S Beets-Tan Regina G H RGH Bressem Keno K Buvat Irene I Cannella Roberto R Cappellini Luca Alessandro LA Cavallo Armando Ugo AU Chepelev Leonid L LL Chu Linda Chi Hang LCH Demircioglu Aydin A deSouza Nandita M NM Dietzel Matthias M Fanni Salvatore Claudio SC Fedorov Andrey A Fournier Laure S LS Giannini Valentina V Girometti Rossano R Groot Lipman Kevin B W KBW Kalarakis Georgios G Kelly Brendan S BS Klontzas Michail E ME Koh Dow-Mu DM Kotter Elmar E Lee Ho Yun HY Maas Mario M Marti-Bonmati Luis L Müller Henning H Obuchowski Nancy N Orlhac Fanny F Papanikolaou Nikolaos N Petrash Ekaterina E Pfaehler Elisabeth E Pinto Dos Santos Daniel D Ponsiglione Andrea A Sabater Sebastià S Sardanelli Francesco F Seeböck Philipp P Sijtsema Nanna M NM Stanzione Arnaldo A Traverso Alberto A Ugga Lorenzo L Vallières Martin M van Dijk Lisanne V LV van Griethuysen Joost J M JJM van Hamersvelt Robbert W RW van Ooijen Peter P Vernuccio Federica F Wang Alan A Williams Stuart S Witowski Jan J Zhang Zhongyi Z Zwanenburg Alex A Cuocolo Renato R

Insights into imaging 20240117 1

<h4>Purpose</h4>To propose a new quality scoring tool, METhodological RadiomICs Score (METRICS), to assess and improve research quality of radiomics studies.<h4>Methods</h4>We conducted an online modified Delphi study with a group of international experts. It was performed in three consecutive stages: Stage#1, item preparation; Stage#2, panel discussion among EuSoMII Auditing Group members to identify the items to be voted; and Stage#3, four rounds of the modified Delphi exercise by panelists to ...[more]

PMID: 38228979

Similar Datasets

Project description:ObjectiveThe main aim of the present systematic review was a comprehensive overview of the Radiomics Quality Score (RQS)-based systematic reviews to highlight common issues and challenges of radiomics research application and evaluate the relationship between RQS and review features.MethodsThe literature search was performed on multiple medical literature archives according to PRISMA guidelines for systematic reviews that reported radiomic quality assessment through the RQS. Reported scores were converted to a 0-100% scale. The Mann-Whitney and Kruskal-Wallis tests were used to compare RQS scores and review features.ResultsThe literature research yielded 345 articles, from which 44 systematic reviews were finally included in the analysis. Overall, the median of RQS was 21.00% (IQR = 11.50). No significant differences of RQS were observed in subgroup analyses according to targets (oncological/not oncological target, neuroradiology/body imaging focus and one imaging technique/more than one imaging technique, characterization/prognosis/detection/other).ConclusionsOur review did not reveal a significant difference of quality of radiomic articles reported in systematic reviews, divided in different subgroups. Furthermore, low overall methodological quality of radiomics research was found independent of specific application domains. While the RQS can serve as a reference tool to improve future study designs, future research should also be aimed at improving its reliability and developing new tools to meet an ever-evolving research space.Key points• Radiomics is a promising high-throughput method that may generate novel imaging biomarkers to improve clinical decision-making process, but it is an inherently complex analysis and often lacks reproducibility and generalizability. • The Radiomics Quality Score serves a necessary role as the de facto reference tool for assessing radiomics studies. • External auditing of radiomics studies, in addition to the standard peer-review process, is valuable to highlight common limitations and provide insights to improve future study designs and practical applicability of the radiomics models.

Project description:BackgroundSystematic reviews within the field of animal research are becoming more common. However, in animal translational research, issues related to methodological quality and quality of reporting continue to arise, potentially leading to underestimation or overestimation of the effects of interventions or prevent studies from being replicated. The various tools and checklists available to ensure good-quality studies and proper reporting include both unique and/or overlapping items and/or simply lack necessary elements or are too situational to certain conditions or diseases. Currently, there is no tool available, which covers all aspects of animal models, from bench-top activities to animal facilities, hence a new tool is needed. This tool should be designed to be able to assess all kinds of animal studies such as old, new, low quality, high quality, interventional and noninterventional on. It should do this on multiple levels through items on quality of reporting, methodological (technical) quality, and risk of bias, for use in assessing the overall quality of studies involving animal research.MethodsDuring a systematic review of meningioma models in animals, we developed a novel unifying tool that can assess all types of animal studies from multiple perspectives. The tool was inspired by the Collaborative Approach to Meta Analysis and Review of Animal Data from Experimental Studies (CAMARADES) checklist, the ARRIVE 2.0 guidelines, and SYRCLE's risk of bias tool, while also incorporating unique items. We used the interrater agreement percentage and Cohen's kappa index to test the interrater agreement between two independent reviewers for the items in the tool.ResultsThere was high interrater agreement across all items (92.9%, 95% CI 91.0-94.8). Cohen's kappa index showed quality of reporting had the best mean index of 0.86 (95%-CI 0.78-0.94), methodological quality had a mean index of 0.83 (95%-CI 0.78-0.94) and finally the items from SYRCLE's risk of bias had a mean kappa index of 0.68 (95%-CI 0.57-0.79).ConclusionsThe Critical Appraisal of Methodological (technical) Quality, Quality of Reporting and Risk of Bias in Animal Research (CRIME-Q) tool unifies a broad spectrum of information (both unique items and items inspired by other methods) about the quality of reporting and methodological (technical) quality, and contains items from SYRCLE's risk of bias. The tool is intended for use in assessing overall study quality across multiple domains and items and is not, unlike other tools, restricted to any particular model or study design (whether interventional or noninterventional). It is also easy to apply when designing and conducting animal experiments to ensure proper reporting and design in terms of replicability, transparency, and validity.

Project description:BackgroundModifiable risky health behaviors, such as tobacco use, excessive alcohol use, being overweight, lack of physical activity, and unhealthy eating habits, are some of the major factors for developing chronic health conditions. Social media platforms have become indispensable means of communication in the digital era. They provide an opportunity for individuals to express themselves, as well as share their health-related concerns with peers and health care providers, with respect to risky behaviors. Such peer interactions can be utilized as valuable data sources to better understand inter-and intrapersonal psychosocial mediators and the mechanisms of social influence that drive behavior change.ObjectiveThe objective of this review is to summarize computational and quantitative techniques facilitating the analysis of data generated through peer interactions pertaining to risky health behaviors on social media platforms.MethodsWe performed a systematic review of the literature in September 2020 by searching three databases-PubMed, Web of Science, and Scopus-using relevant keywords, such as "social media," "online health communities," "machine learning," "data mining," etc. The reporting of the studies was directed by the PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) guidelines. Two reviewers independently assessed the eligibility of studies based on the inclusion and exclusion criteria. We extracted the required information from the selected studies.ResultsThe initial search returned a total of 1554 studies, and after careful analysis of titles, abstracts, and full texts, a total of 64 studies were included in this review. We extracted the following key characteristics from all of the studies: social media platform used for conducting the study, risky health behavior studied, the number of posts analyzed, study focus, key methodological functions and tools used for data analysis, evaluation metrics used, and summary of the key findings. The most commonly used social media platform was Twitter, followed by Facebook, QuitNet, and Reddit. The most commonly studied risky health behavior was nicotine use, followed by drug or substance abuse and alcohol use. Various supervised and unsupervised machine learning approaches were used for analyzing textual data generated from online peer interactions. Few studies utilized deep learning methods for analyzing textual data as well as image or video data. Social network analysis was also performed, as reported in some studies.ConclusionsOur review consolidates the methodological underpinnings for analyzing risky health behaviors and has enhanced our understanding of how social media can be leveraged for nuanced behavioral modeling and representation. The knowledge gained from our review can serve as a foundational component for the development of persuasive health communication and effective behavior modification technologies aimed at the individual and population levels.

Dataset Information

METhodological RadiomICs Score (METRICS): a quality scoring tool for radiomics research endorsed by EuSoMII.

Purpose

Methods

Result

Conclusion

Critical relevance statement

Key points

Publications

METhodological RadiomICs Score (METRICS): a quality scoring tool for radiomics research endorsed by EuSoMII.

Similar Datasets

OmicsDI is part of the ELIXIR infrastructure

Tweets