Dataset Information

Assessment of reproducibility of cancer survival risk predictions across medical centers.

ABSTRACT: BACKGROUND: Two most important considerations in evaluation of survival prediction models are 1) predictability - ability to predict survival risks accurately and 2) reproducibility - ability to generalize to predict samples generated from different studies. We present approaches for assessment of reproducibility of survival risk score predictions across medical centers. METHODS: Reproducibility was evaluated in terms of consistency and transferability. Consistency is the agreement of risk scores predicted between two centers. Transferability from one center to another center is the agreement of the risk scores of the second center predicted by each of the two centers. The transferability can be: 1) model transferability - whether a predictive model developed from one center can be applied to predict the samples generated from other centers and 2) signature transferability - whether signature markers of a predictive model developed from one center can be applied to predict the samples from other centers. We considered eight prediction models, including two clinical models, two gene expression models, and their combinations. Predictive performance of the eight models was evaluated by several common measures. Correlation coefficients between predicted risk scores of different centers were computed to assess reproducibility - consistency and transferability. RESULTS: Two public datasets, the lung cancer data generated from four medical centers and colon cancer data generated from two medical centers, were analyzed. The risk score estimates for lung cancer patients predicted by three of four centers agree reasonably well. In general, a good prediction model showed better cross-center consistency and transferability. The risk scores for the colon cancer patients from one (Moffitt) medical center that were predicted by the clinical models developed from the another (Vanderbilt) medical center were shown to have excellent model transferability and signature transferability. CONCLUSIONS: This study illustrates an analytical approach to assessing reproducibility of predictive models and signatures. Based on the analyses of the two cancer datasets, we conclude that the models with clinical variables appear to perform reasonable well with high degree of consistency and transferability. There should have more investigations on the reproducibility of prediction models including gene expression data across studies.

SUBMITTER: Chen HC

PROVIDER: S-EPMC3598915 | biostudies-literature | 2013

REPOSITORIES: biostudies-literature

ACCESS DATA

Publications

Assessment of reproducibility of cancer survival risk predictions across medical centers.

Chen Hung-Chia HC Chen James J JJ

BMC medical research methodology 20130220

<h4>Background</h4>Two most important considerations in evaluation of survival prediction models are 1) predictability - ability to predict survival risks accurately and 2) reproducibility - ability to generalize to predict samples generated from different studies. We present approaches for assessment of reproducibility of survival risk score predictions across medical centers.<h4>Methods</h4>Reproducibility was evaluated in terms of consistency and transferability. Consistency is the agreement ...[more]

PMID: 23425000

Similar Datasets

Project description:BackgroundTransperineal ultrasound (TPUS) is a vital examination method for diagnosing pelvic floor diseases. However, the quality of TPUS largely relies on the operator's experience, and there is a lack of studies on the evaluation of TPUS quality. Therefore, the objective of this study was to assess the quality of TPUS examinations in Chinese tertiary medical centers.MethodsThis multicenter study conducted in 44 Chinese tertiary medical centers recruited postpartum women between September 2020 and September 2021. All participants underwent a standardized inquiry and TPUS examination. The participating centers were required to submit 5 parts of ultrasound data to the National Ultrasound Quality Control Center: 2-dimensional images at rest, 2-dimensional images at strain; 4-dimensional images of the levator ani hiatus; 4-dimensional images of the levator ani muscle; and 4-dimensional images of the anal sphincter. Quality assessment was performed by 2 experts with more than 5 years of experience in TPUS, and the reasons for nonqualification were stated.ResultsIn this study, 31 hospitals that were distributed across 20 provinces in China were included, submitting 2,251 cases in total. The overall qualified rate ranged from 12.00% to 86.92%. In each part, the qualified rate of 2-dimensional images at rest, 2-dimensional images at straining, levator ani hiatus, levator ani muscle, and anal sphincter was 94.27% (2,122/2,251), 78.54% (1,768/2,251), 85.52% (1,925/2,251), 93.03% (2,094/2,251), and 88.09% (1,983/2,251), respectively. Most of the nonqualified images belonged to 2-dimensional images at strain, and the errors in image acquisition (221/483, 45.76%) and measurement (262/483, 54.24%) were the main reasons for nonqualification. For levator ani hiatus images, error in image acquisition (275/326, 84.36%) was the main reason for nonqualification. Reconstruction error was the most common reason for nonqualification for levator ani muscle (133/157, 84.71%) and anal sphincter images (133/268, 49.63%).ConclusionsThis multicenter study assessed the quality of TPUS in tertiary medical centers in China and identified the common reasons for nonqualification in each part. These findings can aid in forming the basis for quality control management and training for TPUS.

Project description:BackgroundReproducibility is receiving increased attention across many domains of science and genomics is no exception. Efforts to identify copy number variations (CNVs) from exome sequence (ES) data have been increasing. Many algorithms have been published to discover CNVs from exomes and a major challenge is the reproducibility in other datasets. Here we test exome CNV calling reproducibility under three conditions: data generated by different sequencing centers; varying sample sizes; and varying capture methodology.MethodsFour CNV tools were tested: eXome Hidden Markov Model (XHMM), Copy Number Inference From Exome Reads (CoNIFER), EXCAVATOR, and Copy Number Analysis for Targeted Resequencing (CONTRA). To examine the reproducibility, we ran the callers on four datasets, varying sample sizes of N = 10, 30, 75, 100, 300, and data with different capture methodology. We examined the false negative (FN) calls and false positive (FP) calls for potential limitations of the CNV callers. The positive predictive value (PPV) was measured by checking the CNV call concordance against single nucleotide polymorphism array.ResultsUsing independently generated datasets, we examined the PPV for each dataset and observed wide range of PPVs. The PPV values were highly data dependent (p <0.001). For the sample sizes and capture method analyses, we tested the callers in triplicates. Both analyses resulted in wide ranges of PPVs, even for the same test. Interestingly, negative correlations between the PPV and the sample sizes were observed for CoNIFER (ρ = -0.80). Further examination of FN calls showed that 44 % of these were missed by all callers and were attributed to the CNV size (46 % spanned ≤3 exons). Overlap of the FP calls showed that FPs were unique to each caller, indicative of algorithm dependency.ConclusionsOur results demonstrate that further improvements in CNV callers are necessary to improve reproducibility and to include wider spectrum of CNVs (including the small CNVs). These CNV callers should be evaluated on multiple independent, heterogeneously generated datasets of varying size to increase robustness and utility. These approaches to the evaluation of exome CNV are essential to support wide utility and applicability of CNV discovery in exome studies.

Dataset Information

Assessment of reproducibility of cancer survival risk predictions across medical centers.

Publications

Assessment of reproducibility of cancer survival risk predictions across medical centers.

Similar Datasets

OmicsDI is part of the ELIXIR infrastructure

Tweets