Browse
Submit Data
Databases
API
Help

Dataset Information

20 Views

0 Connections

0 Citations

0 Reanalyses

0 Downloads

Omics score: 0

Optimal Shrinkage of Eigenvalues in the Spiked Covariance Model.

ABSTRACT: We show that in a common high-dimensional covariance model, the choice of loss function has a profound effect on optimal estimation. In an asymptotic framework based on the Spiked Covariance model and use of orthogonally invariant estimators, we show that optimal estimation of the population covariance matrix boils down to design of an optimal shrinker ? that acts elementwise on the sample eigenvalues. Indeed, to each loss function there corresponds a unique admissible eigenvalue shrinker ?* dominating all other shrinkers. The shape of the optimal shrinker is determined by the choice of loss function and, crucially, by inconsistency of both eigenvalues and eigenvectors of the sample covariance matrix. Details of these phenomena and closed form formulas for the optimal eigenvalue shrinkers are worked out for a menagerie of 26 loss functions for covariance estimation found in the literature, including the Stein, Entropy, Divergence, Fréchet, Bhattacharya/Matusita, Frobenius Norm, Operator Norm, Nuclear Norm and Condition Number losses.

SUBMITTER: Donoho DL

PROVIDER: S-EPMC6152949 | biostudies-literature | 2018 Aug

REPOSITORIES: biostudies-literature

ACCESS DATA

Json Xml

Publications

Optimal Shrinkage of Eigenvalues in the Spiked Covariance Model.

Donoho David L DL Gavish Matan M Johnstone Iain M IM

Annals of statistics 20180627 4

We show that in a common high-dimensional covariance model, the choice of loss function has a profound effect on optimal estimation. In an asymptotic framework based on the Spiked Covariance model and use of orthogonally invariant estimators, we show that optimal estimation of the population covariance matrix boils down to design of an optimal shrinker <i>η</i> that acts elementwise on the sample eigenvalues. Indeed, to each loss function there corresponds a unique admissible eigenvalue shrinker ...[more]

PMID: 30258255

Similar Datasets

Asymptotics of empirical eigenstructure for high dimensional spiked covariance.

Project description:We derive the asymptotic distributions of the spiked eigenvalues and eigenvectors under a generalized and unified asymptotic regime, which takes into account the magnitude of spiked eigenvalues, sample size, and dimensionality. This regime allows high dimensionality and diverging eigenvalues and provides new insights into the roles that the leading eigenvalues, sample size, and dimensionality play in principal component analysis. Our results are a natural extension of those in Paul (2007) to a more general setting and solve the rates of convergence problems in Shen et al. (2013). They also reveal the biases of estimating leading eigenvalues and eigenvectors by using principal component analysis, and lead to a new covariance estimator for the approximate factor model, called shrinkage principal orthogonal complement thresholding (S-POET), that corrects the biases. Our results are successfully applied to outstanding problems in estimation of risks of large portfolios and false discovery proportions for dependent test statistics and are illustrated by simulation studies.

| S-EPMC5563862 | biostudies-literature

Asymptotic properties of principal component analysis and shrinkage-bias adjustment under the generalized spiked population model.

Project description:With the development of high-throughput technologies, principal component analysis (PCA) in the high-dimensional regime is of great interest. Most of the existing theoretical and methodological results for high-dimensional PCA are based on the spiked population model in which all the population eigenvalues are equal except for a few large ones. Due to the presence of local correlation among features, however, this assumption may not be satisfied in many real-world datasets. To address this issue, we investigate the asymptotic behavior of PCA under the generalized spiked population model. Based on our theoretical results, we propose a series of methods for the consistent estimation of population eigenvalues, angles between the sample and population eigenvectors, correlation coefficients between the sample and population principal component (PC) scores, and the shrinkage bias adjustment for the predicted PC scores. Using numerical experiments and real data examples from the genetics literature, we show that our methods can greatly reduce bias and improve prediction accuracy.

| S-EPMC7441582 | biostudies-literature

Eigenvalues of the covariance matrix as early warning signals for critical transitions in ecological systems.

Project description:Many ecological systems are subject critical transitions, which are abrupt changes to contrasting states triggered by small changes in some key component of the system. Temporal early warning signals such as the variance of a time series, and spatial early warning signals such as the spatial correlation in a snapshot of the system's state, have been proposed to forecast critical transitions. However, temporal early warning signals do not take the spatial pattern into account, and past spatial indicators only examine one snapshot at a time. In this study, we propose the use of eigenvalues of the covariance matrix of multiple time series as early warning signals. We first show theoretically why these indicators may increase as the system moves closer to the critical transition. Then, we apply the method to simulated data from several spatial ecological models to demonstrate the method's applicability. This method has the advantage that it takes into account only the fluctuations of the system about its equilibrium, thus eliminating the effects of any change in equilibrium values. The eigenvector associated with the largest eigenvalue of the covariance matrix is helpful for identifying the regions that are most vulnerable to the critical transition.

| S-EPMC6385210 | biostudies-literature

NONPARAMETRIC COVARIANCE MODEL.

Project description:There has been considerable attention on estimation of conditional variance function in the literature. We propose here a nonparametric model for conditional covariance matrix. A kernel estimator is developed accordingly, its asymptotic bias and variance are derived, and its asymptotic normality is established. A real data example is used to illustrate the proposed estimation procedure.

| S-EPMC3002111 | biostudies-literature

Another look at the eigenvalues of a population matrix model.

Project description:Population matrix models are important tools in resource management, in part because they are used to calculate the finite rate of growth ("dominant eigenvalue"). But understanding how a population matrix model converts life history traits into the finite rate of growth can be tricky. We introduce interactive software ("IsoPOPd") that uses the characteristic equation to display how vital rates (survival and fertility) contribute to the finite rate of growth. Higher-order interactions among vital rates complicate the linkage between a management intervention and a population's growth rate. We illustrate the use of the software for investigating the consequences of three management interventions in a 3-stage model of white-tailed deer (Odocoileus virginianus). The software is applicable to any species with 2- or 3-stages, but the mathematical concepts underlying the software are applicable to a population matrix model of any size. The IsoPOPd software is available at: https://cwhl.vet.cornell.edu/tools/isopopd.

| S-EPMC6855200 | biostudies-literature

A volumetric prediction model for postoperative cyst shrinkage.

Project description:ObjectivesWith only limited information available on dimensional changes after jaw cyst surgery, postoperative cyst shrinkage remains largely unpredictable. We aimed to propose a model for volumetric shrinkage based on time elapsed since cyst surgery.Material and methodsWe used data from patients that underwent cyst enucleation or decompression between 2007 and 2017 and had at least three computed tomography (CT) scans per patient. We fitted one simple exponential decay model [V(t) = V0 · e-ɑt] and one model with a patient-specific decay rate [Vk(t) = V0 · e-βt + γkt].ResultsBased on 108 CT scans from 36 patients (median age at surgery: 45.5 years, IQR: 32.3-55.3, 44% female), our simple exponential decay model is V(t) = V0 · e-0.0035t where V(t) is the residual cyst volume after time t elapsed since surgery, V0 is the initial cyst volume, and e is the base of the natural logarithm. Considering a patient-specific decay rate, the model is Vk(t) = V0 · e-0.0049t + γkt where γk is normally distributed, with expectation 0 and standard deviation 0.0041.ConclusionsUsing an exponential regression model, we were able to reliably estimate volumetric shrinkage after jaw cyst surgery. The patient-specific decay rate substantially improved the fit of the model, whereas adding specific covariates as interaction effects to model the decay rate did not provide any significant improvement.Clinical relevanceEstimating postoperative cyst shrinkage is relevant for both treatment planning of jaw cyst surgery as well as evaluating the clinical success of the surgical approach.

| S-EPMC8531058 | biostudies-literature

EDGEWORTH CORRECTION FOR THE LARGEST EIGENVALUE IN A SPIKED PCA MODEL.

Project description:We study improved approximations to the distribution of the largest eigenvalue ℓ^ of the sample covariance matrix of n zero-mean Gaussian observations in dimension p + 1. We assume that one population principal component has variance ℓ > 1 and the remaining 'noise' components have common variance 1. In the high-dimensional limit p/n → γ > 0, we study Edgeworth corrections to the limiting Gaussian distribution of ℓ^ in the supercritical case ℓ> 1 +γ . The skewness correction involves a quadratic polynomial, as in classical settings, but the coefficients reflect the high-dimensional structure. The methods involve Edgeworth expansions for sums of independent non-identically distributed variates obtained by conditioning on the sample noise eigenvalues, and the limiting bulk properties and fluctuations of these noise eigenvalues.

| S-EPMC6420228 | biostudies-literature

Penalized model-based clustering with unconstrained covariance matrices.

Project description:Clustering is one of the most useful tools for high-dimensional analysis, e.g., for microarray data. It becomes challenging in presence of a large number of noise variables, which may mask underlying clustering structures. Therefore, noise removal through variable selection is necessary. One effective way is regularization for simultaneous parameter estimation and variable selection in model-based clustering. However, existing methods focus on regularizing the mean parameters representing centers of clusters, ignoring dependencies among variables within clusters, leading to incorrect orientations or shapes of the resulting clusters. In this article, we propose a regularized Gaussian mixture model permitting a treatment of general covariance matrices, taking various dependencies into account. At the same time, this approach shrinks the means and covariance matrices, achieving better clustering and variable selection. To overcome one technical challenge in estimating possibly large covariance matrices, we derive an E-M algorithm utilizing the graphical lasso (Friedman et al 2007) for parameter estimation. Numerical examples, including applications to microarray gene expression data, demonstrate the utility of the proposed method.

| S-EPMC2867492 | biostudies-literature

Project description:There is a fundamental limit on the capacity of fibre optical communication system (Shannon Limit). This limit can be potentially overcome via using Nonlinear Frequency Division Multiplexing. Dealing with noises in these systems is one of the most critical parts in implementing a practical system. In this paper, we discover and characterize the correlations among the NFT channels. It is demonstrated that the correlation is universal (i.e., independent of types of system noises) and can be exploited to maximize transmission throughput. We propose and experimentally confirm a noise model showing that end-to-end noise can be modelled as the accumulation of noise associated with each segment of optical communication which can be dealt with independently. Also, each point noise can be further decomposed into different components, some of which are more significant (and even dominating) than others. Hence, one can further approximate and simplify the noise model by focusing on the significant component.

| S-EPMC6483980 | biostudies-literature

Dimensionality of social networks using motifs and eigenvalues.

Project description:We consider the dimensionality of social networks, and develop experiments aimed at predicting that dimension. We find that a social network model with nodes and links sampled from an m-dimensional metric space with power-law distributed influence regions best fits samples from real-world networks when m scales logarithmically with the number of nodes of the network. This supports a logarithmic dimension hypothesis, and we provide evidence with two different social networks, Facebook and LinkedIn. Further, we employ two different methods for confirming the hypothesis: the first uses the distribution of motif counts, and the second exploits the eigenvalue distribution.

| S-EPMC4154874 | biostudies-literature

OmicsDI is part of the ELIXIR infrastructure

OmicsDI is an Elixir interoperability service. Learn more ›

OmicsDI Databases

PRIDE
PeptideAtlas
MassIVE
JPOST Repository
Physiome Model Repository

EGA
EVA
ENA
LINCS
PAXDB
Cell Collective

MetaboLights
Metabolomics Workbench
MetabolomeExpress
GNPS
BioModels
FAIRDOMHub

ArrayExpress
dbGaP
ExpressionAtlas
GEO
NODE

Information

Databases
Help
API
Contact us
Code on GitHub
Terms of use
Submit Data