Assessing the validity and reproducibility of genome-scale predictions.
Ontology highlight
ABSTRACT:
SUBMITTER: Sugden LA
PROVIDER: S-EPMC3810853 | biostudies-other | 2013 Nov
REPOSITORIES: biostudies-other
Publications
Assessing the validity and reproducibility of genome-scale predictions.
Sugden Lauren A LA Tackett Michael R MR Savva Yiannis A YA Thompson William A WA Lawrence Charles E CE
Bioinformatics (Oxford, England) 20130917 22
<h4>Motivation</h4>Validation and reproducibility of results is a central and pressing issue in genomics. Several recent embarrassing incidents involving the irreproducibility of high-profile studies have illustrated the importance of this issue and the need for rigorous methods for the assessment of reproducibility.<h4>Results</h4>Here, we describe an existing statistical model that is very well suited to this problem. We explain its utility for assessing the reproducibility of validation exper ...[more]
Similar Datasets
Project description:BackgroundReproducibility is receiving increased attention across many domains of science and genomics is no exception. Efforts to identify copy number variations (CNVs) from exome sequence (ES) data have been increasing. Many algorithms have been published to discover CNVs from exomes and a major challenge is the reproducibility in other datasets. Here we test exome CNV calling reproducibility under three conditions: data generated by different sequencing centers; varying sample sizes; and varying capture methodology.MethodsFour CNV tools were tested: eXome Hidden Markov Model (XHMM), Copy Number Inference From Exome Reads (CoNIFER), EXCAVATOR, and Copy Number Analysis for Targeted Resequencing (CONTRA). To examine the reproducibility, we ran the callers on four datasets, varying sample sizes of N = 10, 30, 75, 100, 300, and data with different capture methodology. We examined the false negative (FN) calls and false positive (FP) calls for potential limitations of the CNV callers. The positive predictive value (PPV) was measured by checking the CNV call concordance against single nucleotide polymorphism array.ResultsUsing independently generated datasets, we examined the PPV for each dataset and observed wide range of PPVs. The PPV values were highly data dependent (p <0.001). For the sample sizes and capture method analyses, we tested the callers in triplicates. Both analyses resulted in wide ranges of PPVs, even for the same test. Interestingly, negative correlations between the PPV and the sample sizes were observed for CoNIFER (ρ = -0.80). Further examination of FN calls showed that 44 % of these were missed by all callers and were attributed to the CNV size (46 % spanned ≤3 exons). Overlap of the FP calls showed that FPs were unique to each caller, indicative of algorithm dependency.ConclusionsOur results demonstrate that further improvements in CNV callers are necessary to improve reproducibility and to include wider spectrum of CNVs (including the small CNVs). These CNV callers should be evaluated on multiple independent, heterogeneously generated datasets of varying size to increase robustness and utility. These approaches to the evaluation of exome CNV are essential to support wide utility and applicability of CNV discovery in exome studies.
| S-EPMC4976506 | biostudies-literature
Project description:BACKGROUND:The evaluation of competencies in the clinical field is essential for health professionals, as it allows the acquisition of these competencies to be tracked. The objective of this study was to create and evaluate the validity and reliability of a tool for measuring clinical competencies in physical therapy (PT) students to assess the quality of their performance in a professional context. METHODS:A descriptive study was designed. The Measurement Tool for Clinical Competencies in PT (MTCCP) was developed based on the evaluation of 39 experts: 15 clinicians and 24 instructors. The content validity was evaluated using the Content Validity Index (CVI). Three professors were invited to apply the tool to 10 students. Cronbach's alpha, exploratory factor analysis, and the intraclass correlation coefficient were used to determine the reliability and validity of the scale. RESULTS:The CVI was positive-higher than 0.8. Principal component analysis confirmed the construct validity of the tool for two main factors: clinical reasoning (first factor) and professional behavior (second factor). With regard to reliability, the MTCCP achieved an internal congruence of 0.982. The inter-evaluator reproducibility for clinical reasoning, professional behavior, and the total MTCCP score was almost perfect; the ICCs were 0.984, 0.930, and 0.983, respectively. CONCLUSIONS:The MTCCP is a valid and reliable instrument for assessing the performance of PT students in hospital settings and can be used to determine what skills students feel less confident using and what additional training/learning opportunities could be provided. Further research is needed to determine whether the MTCCP has similar validity and reproducibility in other Spanish-speaking national and international PT programs.
| S-EPMC6260878 | biostudies-other
Project description:In healthy arteries, blood flow is regulated by microvascular tone assessed by changes in blood flow volume and vascular resistance to endothelium-dependent and -independent vasodilators. We developed a novel method of using intravascular ultrasound (IVUS) and a Doppler flow wire to measure changes in blood flow volume and vascular resistance of the profunda arterial bed. We assessed the variability over 6 months in measuring microvascular endothelium-dependent dilation to acetylcholine and endothelium-independent dilation to adenosine in 20 subjects who were part of a larger study of Gulf War Illness without obstructive peripheral artery disease. Vasomotor function was assessed by Infusions of control (dextrose), acetylcholine (10-6M), adenosine (50?g), and nitroglycerin (25?g/ml). 400 IVUS and 240 flow velocity images were measured a mean 6 (SD = 2) months apart blind to measurement and infusion stage. The mean (SD) baseline profunda flow was 227 (172) ml/min and vascular resistance 4.6 x 104 (2.4 x 104) dynes-s/cm5. The intraclass correlation coefficients for 6-month variability for vascular function were excellent (range 0.827-0.995). Bland-Altman analyses showed mean differences of less than 2% for microvascular endothelium-dependent function (flow volume and resistance) and less than 1% for macrovascular endothelium-dependent function with acceptable limits of agreement. In 49 subjects assessing concurrent validity of the technique against atherosclerosis risk factors, we observed greater impairment in microvascular endothelium-dependent function per year of age (flow volume = -1.4% (p = 0.018), vascular resistance = 1.5% (p = 0.015)) and current smoking (flow volume = -36.7% (p = .006), vascular resistance = 50.0% (p<0.001)). This novel method of assessing microvascular vasomotor function had acceptable measurement reproducibility and validity.
| S-EPMC6347364 | biostudies-literature
Project description:RNA interference-based screening is a powerful new genomic technology that addresses gene function en masse. To evaluate factors influencing hit list composition and reproducibility, the authors performed 2 identically designed small interfering RNA (siRNA)-based, whole-genome screens for host factors supporting yellow fever virus infection. These screens represent 2 separate experiments completed 5 months apart and allow the direct assessment of the reproducibility of a given siRNA technology when performed in the same environment. Candidate hit lists generated by sum rank, median absolute deviation, z-score, and strictly standardized mean difference were compared within and between whole-genome screens. Application of these analysis methodologies within a single screening data set using a fixed threshold equivalent to a p-value < or = 0.001 resulted in hit lists ranging from 82 to 1140 members and highlighted the tremendous impact analysis methodology has on hit list composition. Intra- and interscreen reproducibility was significantly influenced by the analysis methodology and ranged from 32% to 99%. This study also highlighted the power of testing at least 2 independent siRNAs for each gene product in primary screens. To facilitate validation, the authors conclude by suggesting methods to reduce false discovery at the primary screening stage. In this study, they present the first comprehensive comparison of multiple analysis strategies and demonstrate the impact of the analysis methodology on the composition of the "hit list." Therefore, they propose that the entire data set derived from functional genome-scale screens, especially if publicly funded, should be made available as is done with data derived from gene expression and genome-wide association studies.
| S-EPMC3149892 | biostudies-literature
Project description:Metabolic profiling (metabolomics) aims at measuring small molecules (metabolites) in complex samples like blood or urine for human health studies. While biomarker-based assessment often relies on a single molecule, metabolic profiling combines several metabolites to create a more complex and more specific fingerprint of the disease. However, in contrast to genomics, there is no unique metabolomics setup able to measure the entire metabolome. This challenge leads to tedious and resource consuming preliminary studies to be able to design the right metabolomics experiment. In that context, computer assisted metabolic profiling can be of strong added value to design metabolomics studies more quickly and efficiently. We propose a constraint-based modelling approach which predicts in silico profiles of metabolites that are more likely to be differentially abundant under a given metabolic perturbation (e.g. due to a genetic disease), using flux simulation. In genome-scale metabolic networks, the fluxes of exchange reactions, also known as the flow of metabolites through their external transport reactions, can be simulated and compared between control and disease conditions in order to calculate changes in metabolite import and export. These import/export flux differences would be expected to induce changes in circulating biofluid levels of those metabolites, which can then be interpreted as potential biomarkers or metabolites of interest. In this study, we present SAMBA (SAMpling Biomarker Analysis), an approach which simulates fluxes in exchange reactions following a metabolic perturbation using random sampling, compares the simulated flux distributions between the baseline and modulated conditions, and ranks predicted differentially exchanged metabolites as potential biomarkers for the perturbation. We show that there is a good fit between simulated metabolic exchange profiles and experimental differential metabolites detected in plasma, such as patient data from the disease database OMIM, and metabolic trait-SNP associations found in mGWAS studies. These biomarker recommendations can provide insight into the underlying mechanism or metabolic pathway perturbation lying behind observed metabolite differential abundances, and suggest new metabolites as potential avenues for further experimental analyses.
| S-EPMC10914266 | biostudies-literature
Project description:Association studies have implicated many genes in asthma pathogenesis, with replicated associations between single-nucleotide polymorphisms (SNPs) and asthma reported for more than 30 genes. Genome-wide genotyping enables simultaneous evaluation of most of this variation, and facilitates more comprehensive analysis of other common genetic variation around these candidate genes for association with asthma.To use available genome-wide genotypic data to assess the reproducibility of previously reported associations with asthma and to evaluate the contribution of additional common genetic variation surrounding these loci to asthma susceptibility.Illumina Human Hap 550Kv3 BeadChip (Illumina, San Diego, CA) SNP arrays were genotyped in 422 nuclear families participating in the Childhood Asthma Management Program. Genes with at least one SNP demonstrating prior association with asthma in two or more populations were tested for evidence of association with asthma, using family-based association testing.We identified 39 candidate genes from the literature, using prespecified criteria. Of the 160 SNPs previously genotyped in these 39 genes, 10 SNPs in 6 genes were significantly associated with asthma (including the first independent replication for asthma-associated integrin beta(3) [ITGB3]). Evaluation of 619 additional common variants included in the Illumina 550K array revealed additional evidence of asthma association for 15 genes, although none were significant after adjustment for multiple comparisons.We replicated asthma associations for a minority of candidate genes. Pooling genome-wide association study results from multiple studies will increase the power to appreciate marginal effects of genes and further clarify which candidates are true "asthma genes."
| S-EPMC2695495 | biostudies-literature
Project description:BackgroundReproducible detection of inherited variants with whole genome sequencing (WGS) is vital for the implementation of precision medicine and is a complicated process in which each step affects variant call quality. Systematically assessing reproducibility of inherited variants with WGS and impact of each step in the process is needed for understanding and improving quality of inherited variants from WGS.ResultsTo dissect the impact of factors involved in detection of inherited variants with WGS, we sequence triplicates of eight DNA samples representing two populations on three short-read sequencing platforms using three library kits in six labs and call variants with 56 combinations of aligners and callers. We find that bioinformatics pipelines (callers and aligners) have a larger impact on variant reproducibility than WGS platform or library preparation. Single-nucleotide variants (SNVs), particularly outside difficult-to-map regions, are more reproducible than small insertions and deletions (indels), which are least reproducible when > 5 bp. Increasing sequencing coverage improves indel reproducibility but has limited impact on SNVs above 30×.ConclusionsOur findings highlight sources of variability in variant detection and the need for improvement of bioinformatics pipelines in the era of precision medicine with WGS.
| S-EPMC8722114 | biostudies-literature
Project description:BackgroundAn essential characteristic of health impact assessment (HIA) is that it seeks to predict the future consequences of possible decisions for health. These predictions have to be valid, but as yet it is unclear how validity should be defined in HIA.AimsTo examine the philosophical basis for predictions and the relevance of different forms of validity to HIA.ConclusionsHIA is valid if formal validity, plausibility and predictive validity are in order. Both formal validity and plausibility can usually be established, but establishing predictive validity implies outcome evaluation of HIA. This is seldom feasible owing to long time lags, migration, measurement problems, a lack of data and sensitive indicators, and the fact that predictions may influence subsequent events. Predictive validity most often is not attainable in HIA and we have to make do with formal validity and plausibility. However, in political science, this is by no means exceptional.
| S-EPMC2652952 | biostudies-literature
Project description: Not available
| S-EPMC10092202 | biostudies-literature
Project description:BackgroundSimulation is being increasingly used worldwide in healthcare education. However, it is costly both in terms of finances and human resources. As a consequence, several institutions have designed programs offering several short immersive simulation sessions, each followed by short debriefings. Although debriefing is recommended, no tool exists to assess appropriateness of short debriefings after such simulation sessions. We have developed the Simulation in Healthcare retrOaction Rating Tool (SHORT) to assess short debriefings, and provide some validity evidence for its use.MethodsWe designed this scale based on our experience and previously published instruments, and tested it by assessing short debriefings of simulation sessions offered to emergency medicine residents at Laval University (Canada) from 2015 to 2016. Analysis of its reliability and validity was done using Standards for educational and psychological testing. Generalizability theory was used for testing internal structure evidence for validity.ResultsTwo raters independently assessed 22 filmed short debriefings. Mean debriefing length was 10:35 (min 7:21; max 14:32). Calculated generalizability (reliability) coefficients are ??=?0.80 and ?-?3?=?0.82. The generalizability coefficient for a single rater assessing three debriefings is ??=?0.84.ConclusionsThe G study shows a high generalizability coefficient (????0.80), which demonstrates a high reliability. The response process evidence for validity provides evidence that no errors were associated with using the instrument. Further studies should be done to demonstrate validity of the English version of the instrument and to validate its use by novice raters trained in the use of the SHORT.
| S-EPMC6419351 | biostudies-literature