Dataset Information

Repeatability and Reproducibility of Radiomic Features: A Systematic Review.

ABSTRACT: PURPOSE:An ever-growing number of predictive models used to inform clinical decision making have included quantitative, computer-extracted imaging biomarkers, or "radiomic features." Broadly generalizable validity of radiomics-assisted models may be impeded by concerns about reproducibility. We offer a qualitative synthesis of 41 studies that specifically investigated the repeatability and reproducibility of radiomic features, derived from a systematic review of published peer-reviewed literature. METHODS AND MATERIALS:The PubMed electronic database was searched using combinations of the broad Haynes and Ingui filters along with a set of text words specific to cancer, radiomics (including texture analyses), reproducibility, and repeatability. This review has been reported in compliance with Preferred Reporting Items for Systematic Reviews and Meta-Analyses guidelines. From each full-text article, information was extracted regarding cancer type, class of radiomic feature examined, reporting quality of key processing steps, and statistical metric used to segregate stable features. RESULTS:Among 624 unique records, 41 full-text articles were subjected to review. The studies primarily addressed non-small cell lung cancer and oropharyngeal cancer. Only 7 studies addressed in detail every methodologic aspect related to image acquisition, preprocessing, and feature extraction. The repeatability and reproducibility of radiomic features are sensitive at various degrees to processing details such as image acquisition settings, image reconstruction algorithm, digital image preprocessing, and software used to extract radiomic features. First-order features were overall more reproducible than shape metrics and textural features. Entropy was consistently reported as one of the most stable first-order features. There was no emergent consensus regarding either shape metrics or textural features; however, coarseness and contrast appeared among the least reproducible. CONCLUSIONS:Investigations of feature repeatability and reproducibility are currently limited to a small number of cancer types. Reporting quality could be improved regarding details of feature extraction software, digital image manipulation (preprocessing), and the cutoff value used to distinguish stable features.

SUBMITTER: Traverso A

PROVIDER: S-EPMC6690209 | biostudies-literature | 2018 Nov

REPOSITORIES: biostudies-literature

ACCESS DATA

Publications

Repeatability and Reproducibility of Radiomic Features: A Systematic Review.

Traverso Alberto A Wee Leonard L Dekker Andre A Gillies Robert R

International journal of radiation oncology, biology, physics 20180605 4

<h4>Purpose</h4>An ever-growing number of predictive models used to inform clinical decision making have included quantitative, computer-extracted imaging biomarkers, or "radiomic features." Broadly generalizable validity of radiomics-assisted models may be impeded by concerns about reproducibility. We offer a qualitative synthesis of 41 studies that specifically investigated the repeatability and reproducibility of radiomic features, derived from a systematic review of published peer-reviewed l ...[more]

PMID: 30170872

Similar Datasets

Project description:ObjectivesWe aimed to investigate the influence of magnetic resonance fingerprinting (MRF) dictionary design on radiomic features using in vivo human brain scans.MethodsScan-rescans of three-dimensional MRF and conventional T1-weighted imaging were performed on 21 healthy volunteers (9 males and 12 females; mean age, 41.3 ± 14.6 years; age range, 22-72 years). Five patients with multiple sclerosis (3 males and 2 females; mean age, 41.2 ± 7.3 years; age range, 32-53 years) were also included. MRF data were reconstructed using various dictionaries with different step sizes. First- and second-order radiomic features were extracted from each dataset. Intra-dictionary repeatability and inter-dictionary reproducibility were evaluated using intraclass correlation coefficients (ICCs). Features with ICCs > 0.90 were considered acceptable. Relative changes were calculated to assess inter-dictionary biases.ResultsThe overall scan-rescan ICCs of MRF-based radiomics ranged from 0.86 to 0.95, depending on dictionary step size. No significant differences were observed in the overall scan-rescan repeatability of MRF-based radiomic features and conventional T1-weighted imaging (p = 1.00). Intra-dictionary repeatability was insensitive to dictionary step size differences. MRF-based radiomic features varied among dictionaries (overall ICC for inter-dictionary reproducibility, 0.62-0.99), especially when step sizes were large. First-order and gray level co-occurrence matrix features were the most reproducible feature classes among different step size dictionaries. T1 map-derived radiomic features provided higher repeatability and reproducibility among dictionaries than those obtained with T2 maps.ConclusionMRF-based radiomic features are highly repeatable in various dictionary step sizes. Caution is warranted when performing MRF-based radiomics using datasets containing maps generated from different dictionaries.Key points• MRF-based radiomic features are highly repeatable in various dictionary step sizes. • Use of different MRF dictionaries may result in variable radiomic features, even when the same MRF acquisition data are used. • Caution is needed when performing radiomic analysis using data reconstructed from different dictionaries.

Project description:PurposeTo investigate reproducibility of myocardial radiomic features with cardiac MRI.Materials and methodsTest-retest studies were performed with a 3-T MRI system using commonly used cardiac MRI sequences of cine balanced steady-state free precession (cine bSSFP), T1-weighted and T2-weighted imaging, and quantitative T1 and T2 mapping in phantom experiments and 10 healthy participants (mean ± standard deviation age, 29 years ± 13). In addition, this study assessed repeatability in 51 patients (56 years ± 14) who underwent imaging twice during the same session. Three readers independently delineated the myocardium to investigate inter- and intraobserver reproducibility of radiomic features. A total of 1023 radiomic features were extracted by using PyRadiomics (https://pyradiomics.readthedocs.io/) with 11 image filters and six feature families. The intraclass correlation coefficient (ICC) was estimated to assess reproducibility and repeatability, and features with ICCs greater than or equal to 0.8 were considered reproducible.ResultsDifferent reproducibility patterns were observed among sequences in in vivo test-retest studies. In cine bSSFP, the gray-level run-length matrix was the most reproducible feature family, and the wavelet low-pass filter applied horizontally and vertically was the most reproducible image filter. In T1 and T2 maps, intensity-based statistics (first-order) and gray-level co-occurrence matrix features were the most reproducible feature families, without a dominant reproducible image filter. Across all sequences, gray-level nonuniformity was the most frequently identified reproducible feature name. In inter- and intraobserver reproducibility studies, respectively, only 32%-47% and 61%-73% of features were identified as reproducible.ConclusionOnly a small subset of myocardial radiomic features was reproducible, and these reproducible radiomic features varied among different sequences. Supplemental material is available for this article. © RSNA, 2020See also the commentary by Leiner in this issue.

Project description:Knowledge of the intrinsic variability of radiomic features is essential to the proper interpretation of changes in these features over time. The primary aim of this study was to assess the test-retest repeatability of radiomic features extracted from 18F-FDG PET images of cervical tumors. The impact of different image preprocessing methods was also explored. Methods: Patients with cervical cancer underwent baseline and repeat 18F-FDG PET/CT imaging within 7 d. PET images were reconstructed using 2 methods: ordered-subset expectation maximization (PETOSEM) or ordered-subset expectation maximization with point-spread function (PETPSF). Tumors were segmented to produce whole-tumor volumes of interest (VOIWT) and 40% isocontours (VOI40). Voxels were either left at the default size or resampled to 3-mm isotropic voxels. SUV was discretized to a fixed number of bins (32, 64, or 128). Radiomic features were extracted from both VOIs, and repeatability was then assessed using the Lin concordance correlation coefficient (CCC). Results: Eleven patients were enrolled and completed the test-retest PET/CT imaging protocol. Shape, neighborhood gray-level difference matrix, and gray-level cooccurrence matrix features were repeatable, with a mean CCC value of 0.81. Radiomic features extracted from PETOSEM images showed significantly better repeatability than features extracted from PETPSF images (P < 0.001). Radiomic features extracted from VOI40 were more repeatable than features extracted from VOIWT (P < 0.001). For most features (78.4%), a change in bin number or voxel size resulted in less than a 10% change in feature value. All gray-level emphasis and gray-level run emphasis features showed poor repeatability (CCC values < 0.52) when extracted from VOIWT but were highly repeatable (mean CCC values > 0.96) when extracted from VOI40 Conclusion: Shape, gray-level cooccurrence matrix, and neighborhood gray-level difference matrix radiomic features were consistently repeatable, whereas gray-level run length matrix and gray-level zone length matrix features were highly variable. Radiomic features extracted from VOI40 were more repeatable than features extracted from VOIWT Changes in voxel size or SUV discretization parameters typically resulted in relatively small differences in feature value, though several features were highly sensitive to these changes.

Project description:PurposeTo assess the contribution of a generative adversarial network (GAN) to improve intermanufacturer reproducibility of radiomic features (RFs).Materials and methodsThe authors retrospectively developed a cycle-GAN to translate texture information from chest radiographs acquired using one manufacturer (Siemens) to chest radiographs acquired using another (Philips), producing fake chest radiographs with different textures. The authors prospectively evaluated the ability of this texture-translation cycle-GAN to reduce the intermanufacturer variability of RFs extracted from the lung parenchyma. This study assessed the cycle-GAN's ability to fool several machine learning (ML) classifiers tasked with recognizing the manufacturer on the basis of chest radiography inputs. The authors also evaluated the cycle-GAN's ability to mislead radiologists who were asked to perform the same recognition task. Finally, the authors tested whether the cycle-GAN had an impact on radiomic diagnostic accuracy for chest radiography in patients with congestive heart failure (CHF).ResultsRFs, extracted from chest radiographs after the cycle-GAN's texture translation (fake chest radiographs), showed decreased intermanufacturer RF variability. Using cycle-GAN-generated chest radiographs as inputs, ML classifiers categorized the fake chest radiographs as belonging to the target manufacturer rather than to a native one. Moreover, cycle-GAN fooled two experienced radiologists who identified fake chest radiographs as belonging to a target manufacturer class. Finally, reducing intermanufacturer RF variability with cycle-GAN improved the discriminative power of RFs for patients without CHF versus patients with CHF (from 55% to 73.5%, P < .001).ConclusionBoth ML classifiers and radiologists had difficulty recognizing the chest radiographs' manufacturer. The cycle-GAN improved RF intermanufacturer reproducibility and discriminative power for identifying patients with CHF. This deep learning approach may help counteract the sensitivity of RFs to differences in acquisition.Supplemental material is available for this article.© RSNA, 2020See also the commentary by Alderson in this issue.

Project description:PURPOSE:Recent efforts have demonstrated that radiomic features extracted from the peritumoral region, the area surrounding the tumor parenchyma, have clinical utility in various cancer types. However, as like any radiomic features, peritumoral features could also be unstable and/or nonreproducible. Hence, the purpose of this study was to assess the stability and reproducibility of computed tomography (CT) radiomic features extracted from the peritumoral regions of lung lesions where stability was defined as the consistency of a feature by different segmentations, and reproducibility was defined as the consistency of a feature to different image acquisitions. METHODS:Stability was measured utilizing the "moist run" dataset and reproducibility was measured utilizing the Reference Image Database to Evaluate Therapy Response test-retest dataset. Peritumoral radiomic features were extracted from incremental distances of 3-12 mm outside the tumor segmentation. A total of 264 statistical, histogram, and texture radiomic features were assessed from the selected peritumoral region-of-interests (ROIs). All features (except wavelet texture features) were extracted using standardized algorithms defined by the Image Biomarker Standardisation Initiative. Stability and reproducibility of features were assessed using the concordance correlation coefficient. The clinical utility of stable and reproducible peritumoral features was tested in three previously published lung cancer datasets using overall survival as the endpoint. RESULTS:Features found to be stable and reproducible, regardless of the peritumoral distances, included statistical, histogram, and a subset of texture features suggesting that these features are less affected by changes (e.g., size or shape) of the peritumoral region due to different segmentations and image acquisitions. The stability and reproducibility of Laws and wavelet texture features were inconsistent across all peritumoral distances. The analyses also revealed that a subset of features were consistently stable irrespective of the initial parameters (e.g., seed point) for a given segmentation algorithm. No significant differences were found in stability for features that were extracted from ROIs bounded by a lung parenchyma mask versus ROIs that were not bounded by a lung parenchyma mask (i.e., peritumoral regions that extended outside of lung parenchyma). After testing the clinical utility of peritumoral features, stable and reproducible features were shown to be more likely to create repeatable models than unstable and nonreproducible features. CONCLUSIONS:This study identified a subset of stable and reproducible CT radiomic features extracted from the peritumoral region of lung lesions. The stable and reproducible features identified in this study could be applied to a feature selection pipeline for CT radiomic analyses. According to our findings, top performing features in survival models were more likely to be stable and reproducible hence, it may be best practice to utilize them to achieve repeatable studies and reduce the chance of overfitting.

Project description:BackgroundExisting studies have demonstrated that imaging parameters may affect radiomic features. However, the influence of feature calculating parameters has been overlooked. The purpose of this study is to investigate the influence of feature calculating parameters (gray-level range and bin size) on the reproducibility of CT radiomic features.MethodsThirty-six CT scans from an anthropomorphic thoracic phantom were acquired with different imaging parameters including effective dose, pitch, slice thicknesses and reconstruction kernels. The influence of feature calculating parameters was investigated in terms of three gray-level ranges and eleven gray-level bin sizes. Feature reproducibility was assessed by the intraclass correlation coefficient (ICC) with the cutoff value of 0.8 and the coefficient of variation (CV) with the cutoff value of 20%. The agreements of reproducible features in different ranges and bin sizes were analyzed by Kendall's W test and Kappa test. The proportions of reproducible features, in terms of two calculating, four imaging and two segmentation parameters, were evaluated using Cochran's Q test and Dunn's test.ResultsFor the three gray-level ranges, 50% (44/88) of features were reproducible with a perfect agreement (Kendall's W coefficient 0.844, P<0.001). Of the 72 features that may be influenced by gray-level bin size, 33.3% (24/72) were reproducible for 11 bin sizes with a perfect agreement (Kendall's W coefficient 0.879, P<0.001). For the proportions of reproducible features, there was no statistically significant difference among three ranges (P=0.420), but there was among eleven bin sizes (P=0.013). The proportions of reproducible features in feature calculating parameters were statistically significantly lower than those in imaging parameters (adjusted P<0.05).ConclusionsFeature calculating parameters may have a greater influence than imaging parameters on the reproducibility of CT radiomic features, which should be given special attention in clinical applications.

Dataset Information

Repeatability and Reproducibility of Radiomic Features: A Systematic Review.

Publications

Repeatability and Reproducibility of Radiomic Features: A Systematic Review.

Similar Datasets

OmicsDI is part of the ELIXIR infrastructure

Tweets