Robust interlaboratory reproducibility of a gene expression signature measurement consistent with the needs of a new generation of diagnostic tools.
Ontology highlight
ABSTRACT:
SUBMITTER: Ach RA
PROVIDER: S-EPMC1904205 | biostudies-literature | 2007
REPOSITORIES: biostudies-literature
Publications
Robust interlaboratory reproducibility of a gene expression signature measurement consistent with the needs of a new generation of diagnostic tools.
Ach Robert A RA Floore Arno A Curry Bo B Lazar Vladimir V Glas Annuska M AM Pover Rob R Tsalenko Anya A Ripoche Hugues H Cardoso Fatima F d'Assignies Mahasti Saghatchian MS Bruhn Laurakay L Van't Veer Laura J LJ
BMC genomics 20070607
<h4>Background</h4>The increasing use of DNA microarrays in biomedical research, toxicogenomics, pharmaceutical development, and diagnostics has focused attention on the reproducibility and reliability of microarray measurements. While the reproducibility of microarray gene expression measurements has been the subject of several recent reports, there is still a need for systematic investigation into what factors most contribute to variability of measured expression levels observed among differen ...[more]
Similar Datasets
Project description:The hemagglutination inhibition (HI) assay is a prominent and commonly accepted method used to determine quantitative antibody titers for influenza virus. However, the reproducibility and consistency of this assay may be affected by several factors, including its reliance on biological reagents that are difficult to standardize, such as red blood cells. This report assesses HI assay performance across three accredited, global laboratories when using test virus and a human serum panel aliquoted and distributed from a centrally located reagent stock. The panel of human sera comprised samples with expected low, medium, and high HI titers against two influenza viruses: A/H1N1/California/07/2009 and B/Victoria/Brisbane/60/2008. HI analysis followed a consensus test protocol. Overall, the HI assay reproducibility within each laboratory was high for both influenza strains, with a within-assay run and intraday precision of 100%. Interlab reproducibility was assessed by comparing the geometric mean titer (GMT) of each sample at each laboratory to the consensus GMT of the sample. A/H1N1 had 100% interlab reproducibility, and none of the individual laboratory GMT values exceeded a 2-fold difference compared to the consensus GMT in any tested sample. B/Victoria had an overall reproducibility of 83%. The results demonstrate that with standardization of key reagents and the use of a common protocol by trained staff, the biologically based HI assay can provide similar results between geographically dispersed laboratories. IMPORTANCE Licensure of influenza vaccines relies on the hemagglutination inhibition (HI) assay as the primary method to determine quantitative functional antibody titers. The HI assay is also widely used for influenza virus surveillance, characterization, and epidemiology studies. However, the HI assay has a notable lack of reproducibility and consistency. If serology results are required from multiple concurrent studies supporting the development and regulatory approval of a product, the testing capacity of any given testing laboratory may be exceeded and data from more than one testing laboratory included in regulatory filings. Thus, understanding the reproducibility of HI assay results over time and between testing laboratories is necessary to support a robust clinical trial serology data set. Our results demonstrate that with standardization of key reagents and use of a common protocol by experienced and trained staff, the biologically based HI assay can provide similar results between geographically dispersed laboratories.
| S-EPMC8865924 | biostudies-literature
Project description:Today, next-generation whole-genome sequencing (WGS) is increasingly used to determine the genetic relationships of bacteria on a nearly whole-genome level for infection control purposes and molecular surveillance. Here, we conducted a multicenter ring trial comprising five laboratories to determine the reproducibility and accuracy of WGS-based typing. The participating laboratories sequenced 20 blind-coded Staphylococcus aureus DNA samples using 250-bp paired-end chemistry for library preparation in a single sequencing run on an Illumina MiSeq sequencer. The run acceptance criteria were sequencing outputs >5.6 Gb and Q30 read quality scores of >75%. Subsequently, spa typing, multilocus sequence typing (MLST), ribosomal MLST, and core genome MLST (cgMLST) were performed by the participants. Moreover, discrepancies in cgMLST target sequences in comparisons with the included and also published sequence of the quality control strain ATCC 25923 were resolved using Sanger sequencing. All five laboratories fulfilled the run acceptance criteria in a single sequencing run without any repetition. Of the 400 total possible typing results, 394 of the reported spa types, sequence types (STs), ribosomal STs (rSTs), and cgMLST cluster types were correct and identical among all laboratories; only six typing results were missing. An analysis of cgMLST allelic profiles corroborated this high reproducibility; only 3 of 183,927 (0.0016%) cgMLST allele calls were wrong. Sanger sequencing confirmed all 12 discrepancies of the ring trial results in comparison with the published sequence of ATCC 25923. In summary, this ring trial demonstrated the high reproducibility and accuracy of current next-generation sequencing-based bacterial typing for molecular surveillance when done with nearly completely locked-down methods.
| S-EPMC5328459 | biostudies-literature
Project description:The application of toxicogenomics as a predictive tool for chemical risk assessment has been under evaluation by the toxicology community for more than a decade. However, toxicogenomics predominately remains a tool for investigative research rather than for regulatory risk assessment. In this study, we aimed to determine whether the current generation of microarray technology in combination with an in vitro experimental design was capable of generating robust, reproducible data of sufficient quality to show promise as a tool for regulatory risk assessment. To this end, we designed a prospective collaborative study to determine the level of inter- and intra-laboratory reproducibility between three independent laboratories. All test sites adopted the same protocols for all aspects of the toxicogenomic experiment including cell culture, chemical exposure, RNA extraction, microarray data generation and analysis. As a case study, the genotoxic carcinogen B[a]P and the human hepatocyte cell line HepG2 were used to generate three comparable toxicogenomic data sets. High levels of technical reproducibility were demonstrated using a widely employed gene expression microarray platform. While differences at the global transcriptome level were still observed between the test sites, a common subset of B[a]P responsive genes was identified across all test sites which included both genes previously reported in the literature as B[a]P responsive in addition to the same most highly up and down regulated genes. Pending further analysis, these preliminary data show promise that the current generation of microarray technology in combination with a standard in vitro experimental design can produce robust data that can be reproducibly generated in independent laboratories. Future work will need to determine whether in vitro model(s) can be not only reproducible but also predictive for a range of toxic chemicals with different mechanisms of action. Such an approach may potentially be part of future in vitro testing regimes for regulatory risk assessment.
2011-12-31 | GSE25936 | GEO
Project description:OBJECTIVES:To determine the reliability, internal consistency, measurement error, convergent validity, and floor and ceiling effects of three quality assessment tools commonly used to evaluate the quality of diagnostic test accuracy studies in physical therapy. A secondary aim was to describe the quality of a sample of diagnostic accuracy studies. STUDY DESIGN AND SETTING:50 studies were randomly selected from a comprehensive database of physical therapy-relevant diagnostic accuracy studies. Two reviewers independently rated each study with the Quality Assessment of Diagnostic Accuracy Studies (QUADAS), Quality Assessment of Diagnostic Accuracy Studies 2 (QUADAS-2) and Diagnostic Accuracy Quality Scale (DAQS) tools in random sequence. RESULTS:Only 7% of QUADAS items, 14% of QUADAS-2 items, and 33% of DAQS items had at least moderate inter-rater reliability (kappa>0.40). Internal consistency and convergent validity measures were acceptable (>0.70) in 33% and 50% of cases respectively. Floor or ceiling effects were not present in any tool. The quality of studies was mixed: most avoided case-control sampling strategies and used the same reference standard on all subjects, but many failed to enroll a consecutive or random sample of subjects or provide confidence intervals about estimates of diagnostic accuracy. CONCLUSION:The QUADAS, QUADAS-2 and DAQS tools provide unreliable estimates of the quality of studies of diagnostic accuracy in physical therapy.
| S-EPMC7082671 | biostudies-literature
Project description:When reporting research findings, scientists document the steps they followed so that others can verify and build upon the research. When those steps have been described in sufficient detail that others can retrace the steps and obtain similar results, the research is said to be reproducible. Computers play a vital role in many research disciplines and present both opportunities and challenges for reproducibility. Computers can be programmed to execute analysis tasks, and those programs can be repeated and shared with others. The deterministic nature of most computer programs means that the same analysis tasks, applied to the same data, will often produce the same outputs. However, in practice, computational findings often cannot be reproduced because of complexities in how software is packaged, installed, and executed-and because of limitations associated with how scientists document analysis steps. Many tools and techniques are available to help overcome these challenges; here we describe seven such strategies. With a broad scientific audience in mind, we describe the strengths and limitations of each approach, as well as the circumstances under which each might be applied. No single strategy is sufficient for every scenario; thus we emphasize that it is often useful to combine approaches.
| S-EPMC4940747 | biostudies-other
Project description:Over the past few years, mass spectrometry has emerged as a technology to complement and potentially replace standard immunoassays in routine clinical core laboratories. Application of mass spectrometry to protein and peptide measurement can provide advantages including high sensitivity, the ability to multiplex analytes, and high specificity at the amino acid sequence level. In our previous study, we demonstrated excellent reproducibility of mass spectrometry-selective reaction monitoring (MS-SRM) assays when applying standardized standard operating procedures (SOPs) to measure synthetic peptides in a complex sample, as lack of reproducibility has been a frequent criticism leveled at the use of mass spectrometers in the clinical laboratory compared to immunoassays. Furthermore, an important caveat of SRM-based assays for proteins is that many low-abundance analytes require some type of enrichment before detection with MS. This adds a level of complexity to the procedure and the potential for irreproducibility increases, especially across different laboratories with different operators. The purpose of this study was to test the interlaboratory reproducibility of SRM assays with various upfront enrichment strategies and different types of clinical samples (representing real-world body fluids commonly encountered in routine clinical laboratories). Three different, previously published enrichment strategies for low-abundance analytes and a no-enrichment strategy for high-abundance analytes were tested across four different laboratories using different liquid chromatography-SRM (LC-SRM) platforms and previously developed SOPs. The results demonstrated that these assays were indeed reproducible with coefficients of variation of less than 30% for the measurement of important clinical proteins across all four laboratories in real world samples.
| S-EPMC3761372 | biostudies-literature
Project description:The application of toxicogenomics as a predictive tool for chemical risk assessment has been under evaluation by the toxicology community for more than a decade. However, toxicogenomics predominately remains a tool for investigative research rather than for regulatory risk assessment. In this study, we aimed to determine whether the current generation of microarray technology in combination with an in vitro experimental design was capable of generating robust, reproducible data of sufficient quality to show promise as a tool for regulatory risk assessment. To this end, we designed a prospective collaborative study to determine the level of inter- and intra-laboratory reproducibility between three independent laboratories. All test sites adopted the same protocols for all aspects of the toxicogenomic experiment including cell culture, chemical exposure, RNA extraction, microarray data generation and analysis. As a case study, the genotoxic carcinogen B[a]P and the human hepatocyte cell line HepG2 were used to generate three comparable toxicogenomic data sets. High levels of technical reproducibility were demonstrated using a widely employed gene expression microarray platform. While differences at the global transcriptome level were still observed between the test sites, a common subset of B[a]P responsive genes was identified across all test sites which included both genes previously reported in the literature as B[a]P responsive in addition to the same most highly up and down regulated genes. Pending further analysis, these preliminary data show promise that the current generation of microarray technology in combination with a standard in vitro experimental design can produce robust data that can be reproducibly generated in independent laboratories. Future work will need to determine whether in vitro model(s) can be not only reproducible but also predictive for a range of toxic chemicals with different mechanisms of action. Such an approach may potentially be part of future in vitro testing regimes for regulatory risk assessment. The aim of this study was to evaluate the inter- and intra-laboratory reproducibility of toxicogenomics data for toxicity testing in a regulatory setting. 3 Test Centres performed the same experimental protocol treating HepG2 cells with 3 mM Benzo[a]pyrene (B[a]P) (in 0.5% DMSO) for 24 hours prior to RNA extraction and microarray analysis. Three biological replicates of B[a]P and DMSO treatments were analysed by microarray analysis in each Test Centre. Two Test Centres analysed one sample in triplicate to determine technical reproducibility of the chosen array platform. One Centre analysed two samples in duplicate. 24 samples in total were used in this study
2011-12-31 | E-GEOD-25936 | biostudies-arrayexpress
Project description:The egg yolk precursor protein vitellogenin is widely used as a biomarker of estrogen exposure in male fish. However, standardized methodology is lacking and little is known regarding the reproducibility of results among laboratories using different equipment, reagents, protocols, and data analysis programs. To address this data gap we tested the reproducibility across laboratories to evaluate vitellogenin gene (vtg) expression and assessed the value of using a freely available software data analysis program. Samples collected from studies of male fathead minnows (Pimephales promelas) exposed to 17?-ethinylestradiol (EE2) and minnows exposed to processed wastewater effluent were evaluated for vtg expression in 4 laboratories. Our results indicate reasonable consistency among laboratories if the free software for expression analysis LinRegPCR is used, with 3 of 4 laboratories detecting vtg in fish exposed to 5?ng/L EE2 (n?=?5). All 4 laboratories detected significantly increased vtg levels in 15 male fish exposed to wastewater effluent compared with 15 male fish held in a control stream. Finally, we were able to determine that the source of high interlaboratory variability from complementary deoxyribonucleic acid (cDNA) to quantitative polymerase chain reaction (qPCR) analyses was the expression analysis software unique to each real-time qPCR machine. We successfully eliminated the interlaboratory variability by reanalyzing raw fluorescence data with independent freeware, which yielded cycle thresholds and polymerase chain reaction (PCR) efficiencies that calculated results independently of proprietary software. Our results suggest that laboratories engaged in monitoring programs should validate their PCR protocols and analyze their gene expression data following the guidelines established in the present study for all gene expression biomarkers. Environ Toxicol Chem 2017;36:3102-3107. Published 2017 Wiley Periodicals Inc. on behalf of SETAC. This article is a US government work and, as such, is in the public domain in the United States of America.
| S-EPMC5894818 | biostudies-literature
Project description:Age determination of minerals using the U-Pb technique is widely used to quantify time in Earth's history. A number of geochronology laboratories produce the highest precision U-Pb dates employing the EARTHTIME 202Pb-205Pb-233U-235U tracer solution for isotope dilution, and the EARTHTIME ET100 and ET2000 solutions for system calibration and laboratory intercalibration. Here, we report ET100 and ET2000 solution data from the geochronology laboratory of University of Geneva obtained between 2008 and 2021 and compare the most recent data with results from the geochronology laboratories of Princeton University and ETH Zürich. This compilation demonstrates that (i) the choice of the thermal ionization mass spectrometer model has no influence on precision and accuracy of the data; (ii) the often observed excess scatter of apparent ET100 solution 206Pb/238U dates can be mitigated by more careful tracer-sample equilibration; and (iii) natural zircon reference materials are not suitable for evaluating intra-laboratory repeatability and inter-laboratory reproducibility, since they combine several phenomena of natural system complexities (especially domains of different age within the same zircon grain, and residual loss of radiogenic lead in domains of high decay damage after chemical abrasion pre-treatment). We provide our best estimates of apparent dates for the ET100 solution (206Pb/238U date, 100.173 ± 0.003 Ma), for ET2000 solution (207Pb/206Pb date, 1999.935 ± 0.063 Ma), as well as for natural reference zircon Temora-2 (206Pb/238U date, 417.353 ± 0.052 Ma). These data will allow U-Pb laboratories to evaluate their analytical performance and to independently calibrate non-EARTHTIME tracer solutions in use.
| S-EPMC8262554 | biostudies-literature
Project description:A critical question facing the field of metabolomics is whether data obtained from different centers can be effectively compared and combined. An important aspect of this is the interlaboratory precision (reproducibility) of the analytical protocols used. We analyzed human samples in six laboratories using different instrumentation but a common protocol (the AbsoluteIDQ p180 kit) for the measurement of 189 metabolites via liquid chromatography (LC) or flow injection analysis (FIA) coupled to tandem mass spectrometry (MS/MS). In spiked quality control (QC) samples 82% of metabolite measurements had an interlaboratory precision of <20%, while 83% of averaged individual laboratory measurements were accurate to within 20%. For 20 typical biological samples (serum and plasma from healthy individuals) the median interlaboratory coefficient of variation (CV) was 7.6%, with 85% of metabolites exhibiting a median interlaboratory CV of <20%. Precision was largely independent of the type of sample (serum or plasma) or the anticoagulant used but was reduced in a sample from a patient with dyslipidaemia. The median interlaboratory accuracy and precision of the assay for standard reference plasma (NIST SRM 1950) were 107% and 6.7%, respectively. Likely sources of irreproducibility were the near limit of detection (LOD) typical abundance of some metabolites and the degree of manual review and optimization of peak integration in the LC-MS/MS data after acquisition. Normalization to a reference material was crucial for the semi-quantitative FIA measurements. This is the first interlaboratory assessment of a widely used, targeted metabolomics assay illustrating the reproducibility of the protocol and how data generated on different instruments could be directly integrated in large-scale epidemiological studies.
| S-EPMC6317696 | biostudies-literature