Test-Retest Data for Radiomics Feature Stability Analysis: Generalizable or Study-Specific?
Ontology highlight
ABSTRACT:
SUBMITTER: van Timmeren JE
PROVIDER: S-EPMC6037932 | biostudies-literature |
REPOSITORIES: biostudies-literature
Similar Datasets
Project description:Besides basic measurements as maximum standardized uptake value (SUV)max or SUVmean derived from 18F-FDG positron emission tomography (PET) scans, more advanced quantitative imaging features (i.e. "Radiomics" features) are increasingly investigated for treatment monitoring, outcome prediction, or as potential biomarkers. With these prospected applications of Radiomics features, it is a requisite that they provide robust and reliable measurements. The aim of our study was therefore to perform an integrated stability analysis of a large number of PET-derived features in non-small cell lung carcinoma (NSCLC), based on both a test-retest and an inter-observer setup.Eleven NSCLC patients were included in the test-retest cohort. Patients underwent repeated PET imaging within a one day interval, before any treatment was delivered. Lesions were delineated by applying a threshold of 50% of the maximum uptake value within the tumor. Twenty-three NSCLC patients were included in the inter-observer cohort. Patients underwent a diagnostic whole body PET-computed tomography (CT). Lesions were manually delineated based on fused PET-CT, using a standardized clinical delineation protocol. Delineation was performed independently by five observers, blinded to each other. Fifteen first order statistics, 39 descriptors of intensity volume histograms, eight geometric features and 44 textural features were extracted. For every feature, test-retest and inter-observer stability was assessed with the intra-class correlation coefficient (ICC) and the coefficient of variability, normalized to mean and range. Similarity between test-retest and inter-observer stability rankings of features was assessed with Spearman's rank correlation coefficient.Results showed that the majority of assessed features had both a high test-retest (71%) and inter-observer (91%) stability in terms of their ICC. Overall, features more stable in repeated PET imaging were also found to be more robust against inter-observer variability.Results suggest that further research of quantitative imaging features is warranted with respect to more advanced applications of PET imaging as being used for treatment monitoring, outcome prediction or imaging biomarkers.
| S-EPMC4533992 | biostudies-literature
Project description:Quantitative radiomics features, extracted from medical images, characterize tumour-phenotypes and have been shown to provide prognostic value in predicting clinical outcomes. Stability of radiomics features extracted from apparent diffusion coefficient (ADC)-maps is essential for reliable correlation with the underlying pathology and its clinical applications. Within a multicentre, multi-vendor trial we established a method to analyse radiomics features from ADC-maps of ovarian (n = 12), lung (n = 19), and colorectal liver metastasis (n = 30) cancer patients who underwent repeated (<7 days) diffusion-weighted imaging at 1.5 T and 3 T. From these ADC-maps, 1322 features describing tumour shape, texture and intensity were retrospectively extracted and stable features were selected using the concordance correlation coefficient (CCC > 0.85). Although some features were tissue- and/or respiratory motion-specific, 122 features were stable for all tumour-entities. A large proportion of features were stable across different vendors and field strengths. By extracting stable phenotypic features, fitting-dimensionality is reduced and reliable prognostic models can be created, paving the way for clinical implementation of ADC-based radiomics.
| S-EPMC6423042 | biostudies-literature
Project description:Aims: To evaluate the repeatability of cardiac magnetic resonance (CMR) radiomics features on test-retest scanning using a multi-centre multi-vendor dataset with a varied case-mix. Methods and Results: The sample included 54 test-retest studies from the VOLUMES resource (thevolumesresource.com). Images were segmented according to a pre-defined protocol to select three regions of interest (ROI) in end-diastole and end-systole: right ventricle, left ventricle (LV), and LV myocardium. We extracted radiomics shape features from all three ROIs and, additionally, first-order and texture features from the LV myocardium. Overall, 280 features were derived per study. For each feature, we calculated intra-class correlation coefficient (ICC), within-subject coefficient of variation, and mean relative difference. We ranked robustness of features according to mean ICC stratified by feature category, ROI, and cardiac phase, demonstrating a wide range of repeatability. There were features with good and excellent repeatability (ICC ≥ 0.75) within all feature categories and ROIs. A high proportion of first-order and texture features had excellent repeatability (ICC ≥ 0.90), however, these categories also contained features with the poorest repeatability (ICC < 0.50). Conclusion: CMR radiomic features have a wide range of repeatability. This paper is intended as a reference for future researchers to guide selection of the most robust features for clinical CMR radiomics models. Further work in larger and richer datasets is needed to further define the technical performance and clinical utility of CMR radiomics.
| S-EPMC7738466 | biostudies-literature
Project description:BackgroundDigital biomarkers (DB), as captured using sensors embedded in modern smart devices, are a promising technology for home-based sign and symptom monitoring in Parkinson disease (PD).ObjectiveDespite extensive application in recent studies, test-retest reliability and longitudinal stability of DB have not been well addressed in this context. We utilized the large-scale m-Power data set to establish the test-retest reliability and longitudinal stability of gait, balance, voice, and tapping tasks in an unsupervised and self-administered daily life setting in patients with PD and healthy controls (HC).MethodsIntraclass correlation coefficients were computed to estimate the test-retest reliability of features that also differentiate between patients with PD and healthy volunteers. In addition, we tested for longitudinal stability of DB measures in PD and HC, as well as for their sensitivity to PD medication effects.ResultsAmong the features differing between PD and HC, only a few tapping and voice features had good to excellent test-retest reliabilities and medium to large effect sizes. All other features performed poorly in this respect. Only a few features were sensitive to medication effects. The longitudinal analyses revealed significant alterations over time across a variety of features and in particular for the tapping task.ConclusionsThese results indicate the need for further development of more standardized, sensitive, and reliable DB for application in self-administered remote studies in patients with PD. Motivational, learning, and other confounders may cause variations in performance that need to be considered in DB longitudinal applications.
| S-EPMC8477293 | biostudies-literature
Project description:Conducting side experiments termed robustness experiments, to identify features that are stable with respect to rescans, annotation, or other confounding effects is an important element in radiomics research. However, the matter of how to include the finding of these experiments into the model building process still needs to be explored. Three different methods for incorporating prior knowledge into a radiomics modelling process were evaluated: the naïve approach (ignoring feature quality), the most common approach consisting of removing unstable features, and a novel approach using data augmentation for information transfer (DAFIT). Multiple experiments were conducted using both synthetic and publicly available real lung imaging patient data. Ignoring additional information from side experiments resulted in significantly overestimated model performances meaning the estimated mean area under the curve achieved with a model was increased. Removing unstable features improved the performance estimation, while slightly decreasing the model performance, i.e. decreasing the area under curve achieved with the model. The proposed approach was superior both in terms of the estimation of the model performance and the actual model performance. Our experiments show that data augmentation can prevent biases in performance estimation and has several advantages over the plain omission of the unstable feature. The actual gain that can be obtained depends on the quality and applicability of the prior information on the features in the given domain. This will be an important topic of future research.
| S-EPMC6971266 | biostudies-literature
Project description:Radiomics is a newly emerging field that involves the extraction of massive quantitative features from biomedical images by using data-characterization algorithms. Distinctive imaging features identified from biomedical images can be used for prognosis and therapeutic response prediction, and they can provide a noninvasive approach for personalized therapy. So far, many of the published radiomics studies utilize existing out of the box algorithms to identify the prognostic markers from biomedical images that are not specific to radiomics data. To better utilize biomedical images, we propose a novel machine learning approach, stability selection supervised principal component analysis (SSSuperPCA) that identifies stable features from radiomics big data coupled with dimension reduction for right-censored survival outcomes. The proposed approach allows us to identify a set of stable features that are highly associated with the survival outcomes in a simple yet meaningful manner, while controlling the per-family error rate. We evaluate the performance of SSSuperPCA using simulations and real data sets for non-small cell lung cancer and head and neck cancer, and compare it with other machine learning algorithms. The results demonstrate that our method has a competitive edge over other existing methods in identifying the prognostic markers from biomedical imaging data for the prediction of right-censored survival outcomes.
| S-EPMC7501167 | biostudies-literature
Project description:ObjectivesTo compare image quality of deep learning reconstruction (AiCE) for radiomics feature extraction with filtered back projection (FBP), hybrid iterative reconstruction (AIDR 3D), and model-based iterative reconstruction (FIRST).MethodsEffects of image reconstruction on radiomics features were investigated using a phantom that realistically mimicked a 65-year-old patient's abdomen with hepatic metastases. The phantom was scanned at 18 doses from 0.2 to 4 mGy, with 20 repeated scans per dose. Images were reconstructed with FBP, AIDR 3D, FIRST, and AiCE. Ninety-three radiomics features were extracted from 24 regions of interest, which were evenly distributed across three tissue classes: normal liver, metastatic core, and metastatic rim. Features were analyzed in terms of their consistent characterization of tissues within the same image (intraclass correlation coefficient ≥ 0.75), discriminative power (Kruskal-Wallis test p value < 0.05), and repeatability (overall concordance correlation coefficient ≥ 0.75).ResultsThe median fraction of consistent features across all doses was 6%, 8%, 6%, and 22% with FBP, AIDR 3D, FIRST, and AiCE, respectively. Adequate discriminative power was achieved by 48%, 82%, 84%, and 92% of features, and 52%, 20%, 17%, and 39% of features were repeatable, respectively. Only 5% of features combined consistency, discriminative power, and repeatability with FBP, AIDR 3D, and FIRST versus 13% with AiCE at doses above 1 mGy and 17% at doses ≥ 3 mGy. AiCE was the only reconstruction technique that enabled extraction of higher-order features.ConclusionsAiCE more than doubled the yield of radiomics features at doses typically used clinically. Inconsistent tissue characterization within CT images contributes significantly to the poor stability of radiomics features.Key points• Image quality of CT images reconstructed with filtered back projection and iterative methods is inadequate for the majority of radiomics features due to inconsistent tissue characterization, low discriminative power, or low repeatability. • Deep learning reconstruction enhances image quality for radiomics and more than doubled the feature yield at doses that are typically used in clinical CT imaging. • Image reconstruction algorithms can optimize image quality for more reliable quantification of tissues in CT images.
| S-EPMC9213380 | biostudies-literature
Project description:The validity and reliability of the Lafayette stability platform are well-established for double leg testing. However, no evaluation of single leg (SL) stance on the platform was discovered yet. Therefore, this study aimed to investigate the reliability of conducting the SL stance on the Lafayette platform. Thirty-six healthy and active university students (age 23.2 ± 3.2 years; BMI 21.1 ± 3.1 kg/m2) were tested twice, one week apart (week 1; W1, week 2; W2). They stood on their dominant leg with eyes-open (EO) and eyes-closed (EC) in random order. Three successful trials of 20 seconds each were recorded. The duration during which the platform was maintained within 0° of tilt was referred to as time in balance (TIB). At all-time points, TIB was consistently longer in EO (EOW1: 17.02 ± 1.04s; EOW2: 17.32 ± 1.03s) compared to EC (ECW1: 11.55 ± 1.73s; ECW2: 13.08 ± 1.82s). A ±10 seconds difference was demonstrated in the Bland-Altman analysis in both EO and EC. Lower standard error of measurement (SEM) and coefficient of variation (CV) indicated consistent output. High intraclass correlation coefficient (ICC) values were seen between weeks (EO = 0.74; EC = 0.76) and within weeks (EOW1 = 0.79; EOW2 = 0.86; ECW1 = 0.71; ECW2 = 0.71). Although statistical measures (i.e., SEM, CV, and ICC) indicated good reliability of Lafayette for SL tasks, the wide agreement interval is yet to be clinically meaningful. Factors underlying the wide variation need to be identified before Lafayette is used for TIB assessment.
| S-EPMC9844846 | biostudies-literature
Project description:Objective(s)To examine the psychometric properties of the idiographic Goal-Based Outcome (GBO) tool for young people: test-retest stability, convergent validity, and sensitivity to an intervention.MethodsThis measure validation study used data from a randomized controlled trial of school-based humanistic counseling. We used multilevel analyses to assess test-retest stability, convergent validity of the GBO tool against nomothetic measures of mental wellbeing, and sensitivity to an intervention.ResultsThe GBO tool showed acceptable stability over a 6-24 week period; moderate convergent validity with nomothetic measures of mental well-being, self-esteem, and depression; and greater sensitivity to an intervention than a measure of psychological distress.ConclusionsThe GBO tool shows evidence of having acceptable psychometric properties and is suitable for monitoring change on individual goals. It may also have the capacity to function as a population-level indicator of outcomes in conjunction with the use of other measures of mental health and wellbeing.
| S-EPMC10087879 | biostudies-literature
Project description:IntroductionThe nicotine metabolite ratio (NMR), the ratio of 3-hydroxycotinine to cotinine, is a biomarker used in smoking cessation research, with several retrospective studies suggesting that NMR predicts treatment outcome. To be maximally useful in tailoring treatment, estimates of NMR should be stable over time. The present study is the first to examine the short-term test-retest reliability of NMR among treatment-seeking smokers.MethodsBlood NMR was assessed at two time points, approximately 2-3 weeks apart and prior to intervention, among 72 healthy adult smokers (49% female; 35% non-White) enrolled in a cessation trial (http://ClinicalTrials.gov ID: NCT01314001).ResultsMean NMR was stable from Time-1 to Time-2, with no significant change between assessments; test-retest reliability for NMR values was excellent (ICC[2,1] = 0.87). Test-retest reliability remained acceptable to high when NMR was categorized, as in recent clinical trials. Classification of participants as slow (quartile 1, NMR ≤ 0.24) or normal/fast NMR (quartiles 2-4, NMR ≥ 0.25) was consistent from Time-1 to Time-2 for 96% of participants (κ = 0.89). Though classification of participants into NMR quartiles was less consistent from Time-1 to Time-2 (67% agreement; weighted κ = 0.73), all reclassifications occurred between adjacent quartiles.ConclusionsOverall, these data support the use of a single NMR assessment for association studies with smoking phenotypes and in smokers seeking to quit, and they encourage large-scale efforts to determine optimal NMR cutpoints for tailoring treatment selection.
| S-EPMC4654759 | biostudies-literature