Repeatability and reproducibility study of radiomic features on a phantom and human cohort.
Ontology highlight
ABSTRACT:
SUBMITTER: Jha AK
PROVIDER: S-EPMC7820018 | biostudies-literature | 2021 Jan
REPOSITORIES: biostudies-literature
Publications
Repeatability and reproducibility study of radiomic features on a phantom and human cohort.
Jha A K AK Mithun S S Jaiswar V V Sherkhane U B UB Purandare N C NC Prabhash K K Rangarajan V V Dekker A A Wee L L Traverso A A
Scientific reports 20210121 1
The repeatability and reproducibility of radiomic features extracted from CT scans need to be investigated to evaluate the temporal stability of imaging features with respect to a controlled scenario (test-retest), as well as their dependence on acquisition parameters such as slice thickness, or tube current. Only robust and stable features should be used in prognostication/prediction models to improve generalizability across multiple institutions. In this study, we investigated the repeatabilit ...[more]
Similar Datasets
Project description:PurposeAn ever-growing number of predictive models used to inform clinical decision making have included quantitative, computer-extracted imaging biomarkers, or "radiomic features." Broadly generalizable validity of radiomics-assisted models may be impeded by concerns about reproducibility. We offer a qualitative synthesis of 41 studies that specifically investigated the repeatability and reproducibility of radiomic features, derived from a systematic review of published peer-reviewed literature.Methods and materialsThe PubMed electronic database was searched using combinations of the broad Haynes and Ingui filters along with a set of text words specific to cancer, radiomics (including texture analyses), reproducibility, and repeatability. This review has been reported in compliance with Preferred Reporting Items for Systematic Reviews and Meta-Analyses guidelines. From each full-text article, information was extracted regarding cancer type, class of radiomic feature examined, reporting quality of key processing steps, and statistical metric used to segregate stable features.ResultsAmong 624 unique records, 41 full-text articles were subjected to review. The studies primarily addressed non-small cell lung cancer and oropharyngeal cancer. Only 7 studies addressed in detail every methodologic aspect related to image acquisition, preprocessing, and feature extraction. The repeatability and reproducibility of radiomic features are sensitive at various degrees to processing details such as image acquisition settings, image reconstruction algorithm, digital image preprocessing, and software used to extract radiomic features. First-order features were overall more reproducible than shape metrics and textural features. Entropy was consistently reported as one of the most stable first-order features. There was no emergent consensus regarding either shape metrics or textural features; however, coarseness and contrast appeared among the least reproducible.ConclusionsInvestigations of feature repeatability and reproducibility are currently limited to a small number of cancer types. Reporting quality could be improved regarding details of feature extraction software, digital image manipulation (preprocessing), and the cutoff value used to distinguish stable features.
| S-EPMC6690209 | biostudies-literature
Project description:BackgroundExisting studies have demonstrated that imaging parameters may affect radiomic features. However, the influence of feature calculating parameters has been overlooked. The purpose of this study is to investigate the influence of feature calculating parameters (gray-level range and bin size) on the reproducibility of CT radiomic features.MethodsThirty-six CT scans from an anthropomorphic thoracic phantom were acquired with different imaging parameters including effective dose, pitch, slice thicknesses and reconstruction kernels. The influence of feature calculating parameters was investigated in terms of three gray-level ranges and eleven gray-level bin sizes. Feature reproducibility was assessed by the intraclass correlation coefficient (ICC) with the cutoff value of 0.8 and the coefficient of variation (CV) with the cutoff value of 20%. The agreements of reproducible features in different ranges and bin sizes were analyzed by Kendall's W test and Kappa test. The proportions of reproducible features, in terms of two calculating, four imaging and two segmentation parameters, were evaluated using Cochran's Q test and Dunn's test.ResultsFor the three gray-level ranges, 50% (44/88) of features were reproducible with a perfect agreement (Kendall's W coefficient 0.844, P<0.001). Of the 72 features that may be influenced by gray-level bin size, 33.3% (24/72) were reproducible for 11 bin sizes with a perfect agreement (Kendall's W coefficient 0.879, P<0.001). For the proportions of reproducible features, there was no statistically significant difference among three ranges (P=0.420), but there was among eleven bin sizes (P=0.013). The proportions of reproducible features in feature calculating parameters were statistically significantly lower than those in imaging parameters (adjusted P<0.05).ConclusionsFeature calculating parameters may have a greater influence than imaging parameters on the reproducibility of CT radiomic features, which should be given special attention in clinical applications.
| S-EPMC7417756 | biostudies-literature
Project description:For multicenter clinical studies, characterizing the robustness of image-derived radiomics features is essential. Features calculated on PET images have been shown to be very sensitive to image noise. The purpose of this work was to investigate the efficacy of a relatively simple harmonization strategy on feature robustness and agreement. A purpose-built texture pattern phantom was scanned on 10 different PET scanners in 7 institutions with various different image acquisition and reconstruction protocols. An image harmonization technique based on equalizing a contrast-to-noise ratio was employed to generate a "harmonized" alongside a "standard" dataset for a reproducibility study. In addition, a repeatability study was performed with images from a single PET scanner of variable image noise, varying the binning time of the reconstruction. Feature agreement was measured using the intraclass correlation coefficient (ICC). In the repeatability study, 81/93 features had a lower ICC on the images with the highest image noise as compared to the images with the lowest image noise. Using the harmonized dataset significantly improved the feature agreement for five of the six investigated feature classes over the standard dataset. For three feature classes, high feature agreement corresponded with higher sensitivity to the different patterns, suggesting a way to select suitable features for predictive models.
| S-EPMC9025788 | biostudies-literature
Project description:PurposeAlthough quantitative image biomarkers (radiomics) show promising value for cancer diagnosis, prognosis, and treatment assessment, these biomarkers still lack reproducibility. In this systematic review, we aimed to assess the progress in radiomics reproducibility and repeatability in the recent years.Methods and materialsFour hundred fifty-one abstracts were retrieved according to the original PubMed search pattern with the publication dates ranging from 2017/05/01 to 2020/12/01. Each abstract including the keywords was independently screened by four observers. Forty-two full-text articles were selected for further analysis. Patient population data, radiomic feature classes, feature extraction software, image preprocessing, and reproducibility results were extracted from each article. To support the community with a standardized reporting strategy, we propose a specific reporting checklist to evaluate the feasibility to reproduce each study.ResultsMany studies continue to under-report essential reproducibility information: all but one clinical and all but two phantom studies missed to report at least one important item reporting image acquisition. The studies included in this review indicate that all radiomic features are sensitive to image acquisition, reconstruction, tumor segmentation, and interpolation. However, the amount of sensitivity is feature dependent, for instance, textural features were, in general, less robust than statistical features.ConclusionsRadiomics repeatability, reproducibility, and reporting quality can substantially be improved regarding feature extraction software and settings, image preprocessing and acquisition, cutoff values for stable feature selection. Our proposed radiomics reporting checklist can serve to simplify and improve the reporting and, eventually, guarantee the possibility to fully replicate and validate radiomic studies.
| S-EPMC8591412 | biostudies-literature
Project description:BackgroundPhoton-counting detector computed tomography (PCD-CT) may influence imaging characteristics for various clinical conditions due to higher signal and contrast-to-noise ratio in virtual monoenergetic images (VMI). Radiomics analysis relies on quantification of image characteristics. We evaluated the impact of different VMI reconstructions on radiomic features in in vitro and in vivo PCD-CT datasets.MethodsAn organic phantom consisting of twelve samples (four oranges, four onions, and four apples) was scanned five times. Twenty-three patients who had undergone coronary computed tomography angiography on a first generation PCD-CT system with the same image acquisitions were analyzed. VMIs were reconstructed at 6 keV levels (40, 55, 70, 90, 120, and 190 keV). The phantoms and the patients' left ventricular myocardium (LVM) were segmented for all reconstructions. Ninety-three original radiomic features were extracted. Repeatability and reproducibility were evaluated through intraclass correlations coefficient (ICC) and post hoc paired samples ANOVA t test.ResultsThere was excellent repeatability for radiomic features in phantom scans (all ICC = 1.00). Among all VMIs, 36/93 radiomic features (38.7%) in apples, 28/93 (30.1%) in oranges, and 33/93 (35.5%) in onions were not significantly different. For LVM, the percentage of stable features was high between VMIs ≥ 90 keV (90 versus 120 keV, 77.4%; 90 versus 190 keV, 83.9%; 120 versus 190 keV, 89.3%), while comparison to lower VMI levels led to fewer reproducible features (40 versus 55 keV, 8.6%).ConclusionsVMI levels influence the stability of radiomic features in an organic phantom and patients' LVM; stability decreases considerably below 90 keV.Relevance statementSpectral reconstructions significantly influence radiomic features in vitro and in vivo, necessitating standardization and careful attention to these reconstruction parameters before clinical implementation.Key points• Radiomic features have an excellent repeatability within the same PCD-CT acquisition and reconstruction. • Differences in VMI lead to decreased reproducibility for radiomic features. • VMI ≥ 90 keV increased the reproducibility of the radiomic features.
| S-EPMC10597903 | biostudies-literature
Project description:ObjectivesWe aimed to investigate the influence of magnetic resonance fingerprinting (MRF) dictionary design on radiomic features using in vivo human brain scans.MethodsScan-rescans of three-dimensional MRF and conventional T1-weighted imaging were performed on 21 healthy volunteers (9 males and 12 females; mean age, 41.3 ± 14.6 years; age range, 22-72 years). Five patients with multiple sclerosis (3 males and 2 females; mean age, 41.2 ± 7.3 years; age range, 32-53 years) were also included. MRF data were reconstructed using various dictionaries with different step sizes. First- and second-order radiomic features were extracted from each dataset. Intra-dictionary repeatability and inter-dictionary reproducibility were evaluated using intraclass correlation coefficients (ICCs). Features with ICCs > 0.90 were considered acceptable. Relative changes were calculated to assess inter-dictionary biases.ResultsThe overall scan-rescan ICCs of MRF-based radiomics ranged from 0.86 to 0.95, depending on dictionary step size. No significant differences were observed in the overall scan-rescan repeatability of MRF-based radiomic features and conventional T1-weighted imaging (p = 1.00). Intra-dictionary repeatability was insensitive to dictionary step size differences. MRF-based radiomic features varied among dictionaries (overall ICC for inter-dictionary reproducibility, 0.62-0.99), especially when step sizes were large. First-order and gray level co-occurrence matrix features were the most reproducible feature classes among different step size dictionaries. T1 map-derived radiomic features provided higher repeatability and reproducibility among dictionaries than those obtained with T2 maps.ConclusionMRF-based radiomic features are highly repeatable in various dictionary step sizes. Caution is warranted when performing MRF-based radiomics using datasets containing maps generated from different dictionaries.Key points• MRF-based radiomic features are highly repeatable in various dictionary step sizes. • Use of different MRF dictionaries may result in variable radiomic features, even when the same MRF acquisition data are used. • Caution is needed when performing radiomic analysis using data reconstructed from different dictionaries.
| S-EPMC9213334 | biostudies-literature
Project description:Radiomics-the high throughput extraction of quantitative features from medical images and their correlation with clinical and biological endpoints- is the subject of active and extensive research. Although the field shows promise, the generalizability of radiomic signatures is affected significantly by differences in scan acquisition and reconstruction settings. Previous studies reported on the sensitivity of radiomic features (RFs) to test-retest variability, inter-observer segmentation variability, and intra-scanner variability. A framework involving robust radiomics analysis and the application of a post-reconstruction feature harmonization method using ComBat was recently proposed to address these challenges. In this study, we investigated the reproducibility of RFs across different scanners and scanning parameters using this framework. We analysed thirteen scans of a ten-layer phantom that were acquired differently. Each layer was subdivided into sixteen regions of interest (ROIs), and the scans were compared in a pairwise manner, resulting in seventy-eight different scenarios. Ninety-one RFs were extracted from each ROI. As hypothesized, we demonstrate that the reproducibility of a given RF is not a constant but is dependent on the heterogeneity found in the data under analysis. The number (%) of reproducible RFs varied across the pairwise scenarios investigated, having a wide range between 8 (8.8%) and 78 (85.7%) RFs. Furthermore, in contrast to what has been previously reported, and as hypothesized in the robust radiomics analysis framework, our results demonstrate that ComBat cannot be applied to all RFs but rather on a percentage of those-the "ComBatable" RFs-which differed depending on the data being harmonized. The number (%) of reproducible RFs following ComBat harmonization varied across the pairwise scenarios investigated, ranging from 14 (15.4%) to 80 (87.9%) RFs, and was found to depend on the heterogeneity in the data. We conclude that the standardization of image acquisition protocols remains the cornerstone for improving the reproducibility of RFs, and the generalizability of the signatures developed. Our proposed approach helps identify the reproducible RFs across different datasets.
| S-EPMC8104396 | biostudies-literature
Project description:PURPOSE:To determine the in vitro accuracy, test-retest repeatability, and interplatform reproducibility of T1 quantification protocols used for dynamic contrast-enhanced MRI at 1.5 and 3?T. METHODS:A T1 phantom with 14 samples was imaged at eight centers with a common inversion-recovery spin-echo (IR-SE) protocol and a variable flip angle (VFA) protocol using seven flip angles, as well as site-specific protocols (VFA with different flip angles, variable repetition time, proton density, and Look-Locker inversion recovery). Factors influencing the accuracy (deviation from reference NMR T1 measurements) and repeatability were assessed using general linear mixed models. Interplatform reproducibility was assessed using coefficients of variation. RESULTS:For the common IR-SE protocol, accuracy (median error across platforms?=?1.4-5.5%) was influenced predominantly by T1 sample (P?<?10-6 ), whereas test-retest repeatability (median error?=?0.2-8.3%) was influenced by the scanner (P?<?10-6 ). For the common VFA protocol, accuracy (median error?=?5.7-32.2%) was influenced by field strength (P?=?0.006), whereas repeatability (median error?=?0.7-25.8%) was influenced by the scanner (P?<?0.0001). Interplatform reproducibility with the common VFA was lower at 3?T than 1.5?T (P?=?0.004), and lower than that of the common IR-SE protocol (coefficient of variation 1.5T: VFA/IR-SE?=?11.13%/8.21%, P?=?0.028; 3?T: VFA/IR-SE?=?22.87%/5.46%, P?=?0.001). Among the site-specific protocols, Look-Locker inversion recovery and VFA (2-3 flip angles) protocols showed the best accuracy and repeatability (errors?<?15%). CONCLUSIONS:The VFA protocols with 2 to 3 flip angles optimized for different applications achieved acceptable balance of extensive spatial coverage, accuracy, and repeatability in T1 quantification (errors?<?15%). Further optimization in terms of flip-angle choice for each tissue application, and the use of B1 correction, are needed to improve the robustness of VFA protocols for T1 mapping. Magn Reson Med 79:2564-2575, 2018. © 2017 International Society for Magnetic Resonance in Medicine.
| S-EPMC5821553 | biostudies-literature
Project description:PurposeTo investigate reproducibility of myocardial radiomic features with cardiac MRI.Materials and methodsTest-retest studies were performed with a 3-T MRI system using commonly used cardiac MRI sequences of cine balanced steady-state free precession (cine bSSFP), T1-weighted and T2-weighted imaging, and quantitative T1 and T2 mapping in phantom experiments and 10 healthy participants (mean ± standard deviation age, 29 years ± 13). In addition, this study assessed repeatability in 51 patients (56 years ± 14) who underwent imaging twice during the same session. Three readers independently delineated the myocardium to investigate inter- and intraobserver reproducibility of radiomic features. A total of 1023 radiomic features were extracted by using PyRadiomics (https://pyradiomics.readthedocs.io/) with 11 image filters and six feature families. The intraclass correlation coefficient (ICC) was estimated to assess reproducibility and repeatability, and features with ICCs greater than or equal to 0.8 were considered reproducible.ResultsDifferent reproducibility patterns were observed among sequences in in vivo test-retest studies. In cine bSSFP, the gray-level run-length matrix was the most reproducible feature family, and the wavelet low-pass filter applied horizontally and vertically was the most reproducible image filter. In T1 and T2 maps, intensity-based statistics (first-order) and gray-level co-occurrence matrix features were the most reproducible feature families, without a dominant reproducible image filter. Across all sequences, gray-level nonuniformity was the most frequently identified reproducible feature name. In inter- and intraobserver reproducibility studies, respectively, only 32%-47% and 61%-73% of features were identified as reproducible.ConclusionOnly a small subset of myocardial radiomic features was reproducible, and these reproducible radiomic features varied among different sequences. Supplemental material is available for this article. © RSNA, 2020See also the commentary by Leiner in this issue.
| S-EPMC7377242 | biostudies-literature
Project description:We aimed to evaluate radiomic features' stability across different region of interest (ROI) sizes in CT and MR images. We chose a phantom with a homogenous internal structure so no differences for a feature extracted from ROIs of different sizes would be expected. For this, we scanned a plastic cup filled with sodium chloride solution ten times in CT and per MR sequence (T1-weighted-gradient-echo and T2-weighted-turbo-inversion-recovery-magnitude). We placed sphere-shaped ROIs of different diameters (4, 8, and 16 mm, and 4, 8, and 16 pixels) into the phantom's center. Features were extracted using PyRadiomics. We assessed feature stability across ROI sizes with overall concordance correlation coefficients (OCCCs). Differences were tested for significance with the Mann-Whitney U-test. Of 93 features, 87 T1w-derived, 87 TIRM-derived, and 70 CT-derived features were significantly different between ROI sizes. Among MR-derived features, OCCCs showed excellent (>0.90) agreement for mean, median, and root mean squared for ROI sizes between 4 and 16 mm and pixels. We further observed excellent agreement for 10th and 90th percentile in T1w and 10th percentile in T2w TIRM images. There was no excellent agreement among the OCCCs of CT-derived features. In summary, many features indicated significant differences and only few showed excellent agreement across varying ROI sizes, although we examined a homogenous phantom. Since we considered a small phantom in an experimental setting, further studies to investigate this size effect would be necessary for a generalization. Nevertheless, we believe knowledge about this effect is crucial in interpreting radiomics studies, as features that supposedly discriminate disease entities may only indicate a systematic difference in ROI size.
| S-EPMC8293351 | biostudies-literature