Dataset Information

False Discovery Rates in PET and CT Studies with Texture Features: A Systematic Review.

ABSTRACT:

Purpose

A number of recent publications have proposed that a family of image-derived indices, called texture features, can predict clinical outcome in patients with cancer. However, the investigation of multiple indices on a single data set can lead to significant inflation of type-I errors. We report a systematic review of the type-I error inflation in such studies and review the evidence regarding associations between patient outcome and texture features derived from positron emission tomography (PET) or computed tomography (CT) images.

Methods

For study identification PubMed and Scopus were searched (1/2000-9/2013) using combinations of the keywords texture, prognostic, predictive and cancer. Studies were divided into three categories according to the sources of the type-I error inflation and the use or not of an independent validation dataset. For each study, the true type-I error probability and the adjusted level of significance were estimated using the optimum cut-off approach correction, and the Benjamini-Hochberg method. To demonstrate explicitly the variable selection bias in these studies, we re-analyzed data from one of the published studies, but using 100 random variables substituted for the original image-derived indices. The significance of the random variables as potential predictors of outcome was examined using the analysis methods used in the identified studies.

Results

Fifteen studies were identified. After applying appropriate statistical corrections, an average type-I error probability of 76% (range: 34-99%) was estimated with the majority of published results not reaching statistical significance. Only 3/15 studies used a validation dataset. For the 100 random variables examined, 10% proved to be significant predictors of survival when subjected to ROC and multiple hypothesis testing analysis.

Conclusions

We found insufficient evidence to support a relationship between PET or CT texture features and patient survival. Further fit for purpose validation of these image-derived biomarkers should be supported by appropriate biological and statistical evidence before their association with patient outcome is investigated in prospective studies.

SUBMITTER: Chalkidou A

PROVIDER: S-EPMC4418696 | biostudies-literature | 2015

REPOSITORIES: biostudies-literature

ACCESS DATA

Publications

False Discovery Rates in PET and CT Studies with Texture Features: A Systematic Review.

Chalkidou Anastasia A O'Doherty Michael J MJ Marsden Paul K PK

PloS one 20150504 5

<h4>Purpose</h4>A number of recent publications have proposed that a family of image-derived indices, called texture features, can predict clinical outcome in patients with cancer. However, the investigation of multiple indices on a single data set can lead to significant inflation of type-I errors. We report a systematic review of the type-I error inflation in such studies and review the evidence regarding associations between patient outcome and texture features derived from positron emission ...[more]

PMID: 25938522

Similar Datasets

Project description:All large-scale LC-MS/MS post-translational methylation site discovery experiments require methylpeptide spectrum matches (methyl-PSMs) to be identified at acceptably low false discovery rates (FDRs). To meet estimated methyl-PSM FDRs, methyl-PSM filtering criteria are often determined using the target-decoy approach. The efficacy of this methyl-PSM filtering approach has, however, yet to be thoroughly evaluated. Here we conduct a systematic analysis of methyl-PSM FDRs across a range of sample preparation workflows (each differing in their exposure to the alcohols methanol and isopropanol) and mass spectrometric instrument platforms (each employing a different mode of MS/MS dissociation). Through 13CD3-methionine labeling (heavy-methyl SILAC) of S. cerevisiae cells and in-depth manual data inspection, accurate lists of true positive methyl-PSMs were determined, allowing methyl-PSM FDRs to be compared to target-decoy approach-derived methyl-PSM FDR estimates. These results show that global FDR estimates produce extremely unreliable methyl-PSM filtering criteria; we demonstrate that this is an unavoidable consequence of the high number of amino acid combinations capable of producing peptide sequences that are isobaric to methylated peptides of a different sequence. Separate methyl-PSM FDR estimates were also found to be unreliable due to prevalent sources of false positive methyl-PSMs that produce high peptide identity score distributions. Incorrect methylation site localizations, peptides containing cysteinyl-S-β-propionamide, and methylated glutamic or aspartic acid residues can partially, but not wholly, account for these false positive methyl-PSMs. Together these results indicate that the target-decoy approach is an unreliable means of estimating methyl-PSM FDRs and methyl-PSM filtering criteria. We suggest that orthogonal methylpeptide validation (e.g. heavy-methyl SILAC or its offshoots) should be considered a prerequisite for obtaining high confidence methyl-PSMs in large-scale LC-MS/MS methylation site discovery experiments, and make recommendations on how to reduce methyl-PSM FDRs in samples not amenable to heavy isotope labeling.

Project description:ObjectiveTo determine the intra-, inter- and test-retest variability of CT-based texture analysis (CTTA) metrics.Materials and methodsIn this study, we conducted a series of CT imaging experiments using a texture phantom to evaluate the performance of a CTTA panel on routine abdominal imaging protocols. The phantom comprises of three different regions with various textures found in tumors. The phantom was scanned on two CT scanners viz. the Philips Brilliance 64 CT and Toshiba Aquilion Prime 160 CT scanners. The intra-scanner variability of the CTTA metrics was evaluated across imaging parameters such as slice thickness, field of view, post-reconstruction filtering, tube voltage, and tube current. For each scanner and scanning parameter combination, we evaluated the performance of eight different types of texture quantification techniques on a predetermined region of interest (ROI) within the phantom image using 235 different texture metrics. We conducted the repeatability (test-retest) and robustness (intra-scanner) test on both the scanners and the reproducibility test was conducted by comparing the inter-scanner differences in the repeatability and robustness to identify reliable CTTA metrics. Reliable metrics are those metrics that are repeatable, reproducible and robust.ResultsAs expected, the robustness, repeatability and reproducibility of CTTA metrics are variably sensitive to various scanner and scanning parameters. Entropy of Fast Fourier Transform-based texture metrics was overall most reliable across the two scanners and scanning conditions. Post-processing techniques that reduce image noise while preserving the underlying edges associated with true anatomy or pathology bring about significant differences in radiomic reliability compared to when they were not used.ConclusionFollowing large-scale validation, identification of reliable CTTA metrics can aid in conducting large-scale multicenter CTTA analysis using sample sets acquired using different imaging protocols, scanners etc.

Dataset Information

False Discovery Rates in PET and CT Studies with Texture Features: A Systematic Review.

Purpose

Methods

Results

Conclusions

Publications

False Discovery Rates in PET and CT Studies with Texture Features: A Systematic Review.

Similar Datasets

OmicsDI is part of the ELIXIR infrastructure

Tweets