Dataset Information

The balance of reproducibility, sensitivity, and specificity of lists of differentially expressed genes in microarray studies.

ABSTRACT: BACKGROUND: Reproducibility is a fundamental requirement in scientific experiments. Some recent publications have claimed that microarrays are unreliable because lists of differentially expressed genes (DEGs) are not reproducible in similar experiments. Meanwhile, new statistical methods for identifying DEGs continue to appear in the scientific literature. The resultant variety of existing and emerging methods exacerbates confusion and continuing debate in the microarray community on the appropriate choice of methods for identifying reliable DEG lists. RESULTS: Using the data sets generated by the MicroArray Quality Control (MAQC) project, we investigated the impact on the reproducibility of DEG lists of a few widely used gene selection procedures. We present comprehensive results from inter-site comparisons using the same microarray platform, cross-platform comparisons using multiple microarray platforms, and comparisons between microarray results and those from TaqMan - the widely regarded "standard" gene expression platform. Our results demonstrate that (1) previously reported discordance between DEG lists could simply result from ranking and selecting DEGs solely by statistical significance (P) derived from widely used simple t-tests; (2) when fold change (FC) is used as the ranking criterion with a non-stringent P-value cutoff filtering, the DEG lists become much more reproducible, especially when fewer genes are selected as differentially expressed, as is the case in most microarray studies; and (3) the instability of short DEG lists solely based on P-value ranking is an expected mathematical consequence of the high variability of the t-values; the more stringent the P-value threshold, the less reproducible the DEG list is. These observations are also consistent with results from extensive simulation calculations. CONCLUSION: We recommend the use of FC-ranking plus a non-stringent P cutoff as a straightforward and baseline practice in order to generate more reproducible DEG lists. Specifically, the P-value cutoff should not be stringent (too small) and FC should be as large as possible. Our results provide practical guidance to choose the appropriate FC and P-value cutoffs when selecting a given number of DEGs. The FC criterion enhances reproducibility, whereas the P criterion balances sensitivity and specificity.

SUBMITTER: Shi L

PROVIDER: S-EPMC2537561 | biostudies-literature | 2008

REPOSITORIES: biostudies-literature

ACCESS DATA

Publications

The balance of reproducibility, sensitivity, and specificity of lists of differentially expressed genes in microarray studies.

Shi Leming L Jones Wendell D WD Jensen Roderick V RV Harris Stephen C SC Perkins Roger G RG Goodsaid Federico M FM Guo Lei L Croner Lisa J LJ Boysen Cecilie C Fang Hong H Qian Feng F Amur Shashi S Bao Wenjun W Barbacioru Catalin C CC Bertholet Vincent V Cao Xiaoxi Megan XM Chu Tzu-Ming TM Collins Patrick J PJ Fan Xiao-Hui XH Frueh Felix W FW Fuscoe James C JC Guo Xu X Han Jing J Herman Damir D Hong Huixiao H Kawasaki Ernest S ES Li Quan-Zhen QZ Luo Yuling Y Ma Yunqing Y Mei Nan N Peterson Ron L RL Puri Raj K RK Shippy Richard R Su Zhenqiang Z Sun Yongming Andrew YA Sun Hongmei H Thorn Brett B Turpaz Yaron Y Wang Charles C Wang Sue Jane SJ Warrington Janet A JA Willey James C JC Wu Jie J Xie Qian Q Zhang Liang L Zhang Lu L Zhong Sheng S Wolfinger Russell D RD Tong Weida W

BMC bioinformatics 20080812

<h4>Background</h4>Reproducibility is a fundamental requirement in scientific experiments. Some recent publications have claimed that microarrays are unreliable because lists of differentially expressed genes (DEGs) are not reproducible in similar experiments. Meanwhile, new statistical methods for identifying DEGs continue to appear in the scientific literature. The resultant variety of existing and emerging methods exacerbates confusion and continuing debate in the microarray community on the ...[more]

PMID: 18793455

Similar Datasets

Project description:BackgroundTo identify differentially expressed genes (DEGs) from microarray data, users of the Affymetrix GeneChip system need to select both a preprocessing algorithm to obtain expression-level measurements and a way of ranking genes to obtain the most plausible candidates. We recently recommended suitable combinations of a preprocessing algorithm and gene ranking method that can be used to identify DEGs with a higher level of sensitivity and specificity. However, in addition to these recommendations, researchers also want to know which combinations enhance reproducibility.ResultsWe compared eight conventional methods for ranking genes: weighted average difference (WAD), average difference (AD), fold change (FC), rank products (RP), moderated t statistic (modT), significance analysis of microarrays (samT), shrinkage t statistic (shrinkT), and intensity-based moderated t statistic (ibmT) with six preprocessing algorithms (PLIER, VSN, FARMS, multi-mgMOS (mmgMOS), MBEI, and GCRMA). A total of 36 real experimental datasets was evaluated on the basis of the area under the receiver operating characteristic curve (AUC) as a measure for both sensitivity and specificity. We found that the RP method performed well for VSN-, FARMS-, MBEI-, and GCRMA-preprocessed data, and the WAD method performed well for mmgMOS-preprocessed data. Our analysis of the MicroArray Quality Control (MAQC) project's datasets showed that the FC-based gene ranking methods (WAD, AD, FC, and RP) had a higher level of reproducibility: The percentages of overlapping genes (POGs) across different sites for the FC-based methods were higher overall than those for the t-statistic-based methods (modT, samT, shrinkT, and ibmT). In particular, POG values for WAD were the highest overall among the FC-based methods irrespective of the choice of preprocessing algorithm.ConclusionOur results demonstrate that to increase sensitivity, specificity, and reproducibility in microarray analyses, we need to select suitable combinations of preprocessing algorithms and gene ranking methods. We recommend the use of FC-based methods, in particular RP or WAD.

Project description:BackgroundMicroarray experiments are often performed with a small number of biological replicates, resulting in low statistical power for detecting differentially expressed genes and concomitant high false positive rates. While increasing sample size can increase statistical power and decrease error rates, with too many samples, valuable resources are not used efficiently. The issue of how many replicates are required in a typical experimental system needs to be addressed. Of particular interest is the difference in required sample sizes for similar experiments in inbred vs. outbred populations (e.g. mouse and rat vs. human).ResultsWe hypothesize that if all other factors (assay protocol, microarray platform, data pre-processing) were equal, fewer individuals would be needed for the same statistical power using inbred animals as opposed to unrelated human subjects, as genetic effects on gene expression will be removed in the inbred populations. We apply the same normalization algorithm and estimate the variance of gene expression for a variety of cDNA data sets (humans, inbred mice and rats) comparing two conditions. Using one sample, paired sample or two independent sample t-tests, we calculate the sample sizes required to detect a 1.5-, 2-, and 4-fold changes in expression level as a function of false positive rate, power and percentage of genes that have a standard deviation below a given percentile.ConclusionsFactors that affect power and sample size calculations include variability of the population, the desired detectable differences, the power to detect the differences, and an acceptable error rate. In addition, experimental design, technical variability and data pre-processing play a role in the power of the statistical tests in microarrays. We show that the number of samples required for detecting a 2-fold change with 90% probability and a p-value of 0.01 in humans is much larger than the number of samples commonly used in present day studies, and that far fewer individuals are needed for the same statistical power when using inbred animals rather than unrelated human subjects.

Project description:BACKGROUND: This paper presents a unified framework for finding differentially expressed genes (DEGs) from the microarray data. The proposed framework has three interrelated modules: (i) gene ranking, ii) significance analysis of genes and (iii) validation. The first module uses two gene selection algorithms, namely, a) two-way clustering and b) combined adaptive ranking to rank the genes. The second module converts the gene ranks into p-values using an R-test and fuses the two sets of p-values using the Fisher's omnibus criterion. The DEGs are selected using the FDR analysis. The third module performs three fold validations of the obtained DEGs. The robustness of the proposed unified framework in gene selection is first illustrated using false discovery rate analysis. In addition, the clustering-based validation of the DEGs is performed by employing an adaptive subspace-based clustering algorithm on the training and the test datasets. Finally, a projection-based visualization is performed to validate the DEGs obtained using the unified framework. RESULTS: The performance of the unified framework is compared with well-known ranking algorithms such as t-statistics, Significance Analysis of Microarrays (SAM), Adaptive Ranking, Combined Adaptive Ranking and Two-way Clustering. The performance curves obtained using 50 simulated microarray datasets each following two different distributions indicate the superiority of the unified framework over the other reported algorithms. Further analyses on 3 real cancer datasets and 3 Parkinson's datasets show the similar improvement in performance. First, a 3 fold validation process is provided for the two-sample cancer datasets. In addition, the analysis on 3 sets of Parkinson's data is performed to demonstrate the scalability of the proposed method to multi-sample microarray datasets. CONCLUSION: This paper presents a unified framework for the robust selection of genes from the two-sample as well as multi-sample microarray experiments. Two different ranking methods used in module 1 bring diversity in the selection of genes. The conversion of ranks to p-values, the fusion of p-values and FDR analysis aid in the identification of significant genes which cannot be judged based on gene ranking alone. The 3 fold validation, namely, robustness in selection of genes using FDR analysis, clustering, and visualization demonstrate the relevance of the DEGs. Empirical analyses on 50 artificial datasets and 6 real microarray datasets illustrate the efficacy of the proposed approach. The analyses on 3 cancer datasets demonstrate the utility of the proposed approach on microarray datasets with two classes of samples. The scalability of the proposed unified approach to multi-sample (more than two sample classes) microarray datasets is addressed using three sets of Parkinson's Data. Empirical analyses show that the unified framework outperformed other gene selection methods in selecting differentially expressed genes from microarray data.

Project description:BackgroundMicroarrays have been widely used for the analysis of gene expression and several commercial platforms are available. The combined use of multiple platforms can overcome the inherent biases of each approach, and may represent an alternative that is complementary to RT-PCR for identification of the more robust changes in gene expression profiles. In this paper, we combined statistical and functional analysis for the cross platform validation of two oligonucleotide-based technologies, Affymetrix (AFFX) and Applied Biosystems (ABI), and for the identification of differentially expressed genes.ResultsIn this study, we analysed differentially expressed genes after treatment of an ovarian carcinoma cell line with a cell cycle inhibitor. Treated versus control RNA was analysed for expression of 16425 genes represented on both platforms. We assessed reproducibility between replicates for each platform using CAT plots, and we found it high for both, with better scores for AFFX. We then applied integrative correlation analysis to assess reproducibility of gene expression patterns across studies, bypassing the need for normalizing expression measurements across platforms. We identified 930 genes as differentially expressed on AFFX and 908 on ABI, with approximately 80% common to both platforms. Despite the different absolute values, the range of intensities of the differentially expressed genes detected by each platform was similar. ABI showed a slightly higher dynamic range in FC values, which might be associated with its detection system. 62/66 genes identified as differentially expressed by Microarray were confirmed by RT-PCR.ConclusionIn this study we present a cross-platform validation of two oligonucleotide-based technologies, AFFX and ABI. We found good reproducibility between replicates, and showed that both platforms can be used to select differentially expressed genes with substantial agreement. Pathway analysis of the affected functions identified themes well in agreement with those expected for a cell cycle inhibitor, suggesting that this procedure is appropriate to facilitate the identification of biologically relevant signatures associated with compound treatment. The high rate of confirmation found for both common and platform-specific genes suggests that the combination of platforms may overcome biases related to probe design and technical features, thereby accelerating the identification of trustworthy differentially expressed genes.

Dataset Information

The balance of reproducibility, sensitivity, and specificity of lists of differentially expressed genes in microarray studies.

Publications

The balance of reproducibility, sensitivity, and specificity of lists of differentially expressed genes in microarray studies.

Similar Datasets

OmicsDI is part of the ELIXIR infrastructure

Tweets