Dataset Information

A comparison of Bayesian and frequentist model selection methods for factor analysis models.

ABSTRACT: We compare the performances of well-known frequentist model fit indices (MFIs) and several Bayesian model selection criteria (MCC) as tools for cross-loading selection in factor analysis under low to moderate sample sizes, cross-loading sizes, and possible violations of distributional assumptions. The Bayesian criteria considered include the Bayes factor (BF), Bayesian Information Criterion (BIC), Deviance Information Criterion (DIC), a Bayesian leave-one-out with Pareto smoothed importance sampling (LOO-PSIS), and a Bayesian variable selection method using the spike-and-slab prior (SSP; Lu, Chow, & Loken, 2016). Simulation results indicate that of the Bayesian measures considered, the BF and the BIC showed the best balance between true positive rates and false positive rates, followed closely by the SSP. The LOO-PSIS and the DIC showed the highest true positive rates among all the measures considered, but with elevated false positive rates. In comparison, likelihood ratio tests (LRTs) are still the preferred frequentist model comparison tool, except for their higher false positive detection rates compared to the BF, BIC and SSP under violations of distributional assumptions. The root mean squared error of approximation (RMSEA) and the Tucker-Lewis index (TLI) at the conventional cut-off of approximate fit impose much more stringent "penalties" on model complexity under conditions with low cross-loading size, low sample size, and high model complexity compared with the LRTs and all other Bayesian MCC. Nevertheless, they provided a reasonable alternative to the LRTs in cases where the models cannot be readily constructed as nested within each other. (PsycINFO Database Record

SUBMITTER: Lu ZH

PROVIDER: S-EPMC5527973 | biostudies-literature | 2017 Jun

REPOSITORIES: biostudies-literature

ACCESS DATA

Publications

A comparison of Bayesian and frequentist model selection methods for factor analysis models.

Lu Zhao-Hua ZH Chow Sy-Miin SM Loken Eric E

Psychological methods 20170601 2

We compare the performances of well-known frequentist model fit indices (MFIs) and several Bayesian model selection criteria (MCC) as tools for cross-loading selection in factor analysis under low to moderate sample sizes, cross-loading sizes, and possible violations of distributional assumptions. The Bayesian criteria considered include the Bayes factor (BF), Bayesian Information Criterion (BIC), Deviance Information Criterion (DIC), a Bayesian leave-one-out with Pareto smoothed importance samp ...[more]

PMID: 28594228

Similar Datasets

Project description:In order to have confidence in model-based phylogenetic methods, such as maximum likelihood (ML) and Bayesian analyses, one must use an appropriate model of molecular evolution identified using statistically rigorous criteria. Although model selection methods such as the likelihood ratio test and Akaike information criterion are widely used in the phylogenetic literature, model selection methods lack the ability to reject all models if they provide an inadequate fit to the data. There are two methods, however, that assess absolute model adequacy, the frequentist Goldman-Cox (GC) test and Bayesian posterior predictive simulations (PPSs), which are commonly used in conjunction with the multinomial log likelihood test statistic. In this study, we use empirical and simulated data to evaluate the adequacy of common substitution models using both frequentist and Bayesian methods and compare the results with those obtained with model selection methods. In addition, we investigate the relationship between model adequacy and performance in ML and Bayesian analyses in terms of topology, branch lengths, and bipartition support. We show that tests of model adequacy based on the multinomial likelihood often fail to reject simple substitution models, especially when the models incorporate among-site rate variation (ASRV), and normally fail to reject less complex models than those chosen by model selection methods. In addition, we find that PPSs often fail to reject simpler models than the GC test. Use of the simplest substitution models not rejected based on fit normally results in similar but divergent estimates of tree topology and branch lengths. In addition, use of the simplest adequate substitution models can affect estimates of bipartition support, although these differences are often small with the largest differences confined to poorly supported nodes. We also find that alternative assumptions about ASRV can affect tree topology, tree length, and bipartition support. Our results suggest that using the simplest substitution models not rejected based on fit may be a valid alternative to implementing more complex models identified by model selection methods. However, all common substitution models may fail to recover the correct topology and assign appropriate bipartition support if the true tree shape is difficult to estimate regardless of model adequacy.

Project description:Background/aimsRegulatory approval of a drug or device involves an assessment of not only the benefits but also the risks of adverse events associated with the therapeutic agent. Although randomized controlled trials (RCTs) are the gold standard for evaluating effectiveness, the number of treated patients in a single RCT may not be enough to detect a rare but serious side effect of the treatment. Meta-analysis plays an important role in the evaluation of the safety of medical products and has advantage over analyzing a single RCT when estimating the rate of adverse events.MethodsIn this article, we compare 15 widely used meta-analysis models under both Bayesian and frequentist frameworks when outcomes are extremely infrequent or rare. We present extensive simulation study results and then apply these methods to a real meta-analysis that considers RCTs investigating the effect of rosiglitazone on the risks of myocardial infarction and of death from cardiovascular causes.ResultsOur simulation studies suggest that the beta hyperprior method modeling treatment group-specific parameters and accounting for heterogeneity performs the best. Most models ignoring between-study heterogeneity give poor coverage probability when such heterogeneity exists. In the data analysis, different methods provide a wide range of log odds ratio estimates between rosiglitazone and control treatments with a mixed conclusion on their statistical significance based on 95% confidence (or credible) intervals.ConclusionIn the rare event setting, treatment effect estimates obtained from traditional meta-analytic methods may be biased and provide poor coverage probability. This trend worsens when the data have large between-study heterogeneity. In general, we recommend methods that first estimate the summaries of treatment-specific risks across studies and then relative treatment effects based on the summaries when appropriate. Furthermore, we recommend fitting various methods, comparing the results and model performance, and investigating any significant discrepancies among them.

Project description:BackgroundFloating catchment methods have recently been applied to identify priority regions for Automated External Defibrillator (AED) deployment, to aid in improving Out of Hospital Cardiac Arrest (OHCA) survival. This approach models access as a supply-to-demand ratio for each area, targeting areas with high demand and low supply for AED placement. These methods incorporate spatial covariates on OHCA occurrence, but do not provide precise AED locations, which are critical to the initial intent of such location analysis research. Exact AED locations can be determined using optimisation methods, but they do not incorporate known spatial risk factors for OHCA, such as income and demographics. Combining these two approaches would evaluate AED placement impact, describe drivers of OHCA occurrence, and identify areas that may not be appropriately covered by AED placement strategies. There are two aims in this paper. First, to develop geospatial models of OHCA that account for and display uncertainty. Second, to evaluate the AED placement methods using geospatial models of accessibility. We first identify communities with the greatest gap between demand and supply for allocating AEDs. We then use this information to evaluate models for precise AED location deployment.MethodsCase study data set consisted of 2802 OHCA events and 719 AEDs. Spatial OHCA occurrence was described using a geospatial model, with possible spatial correlation accommodated by introducing a conditional autoregressive (CAR) prior on the municipality-level spatial random effect. This model was fit with Integrated Nested Laplacian Approximation (INLA), using covariates for population density, proportion male, proportion over 65 years, financial strength, and the proportion of land used for transport, commercial, buildings, recreation, and urban areas. Optimisation methods for AED locations were applied to find the top 100 AED placement locations. AED access was calculated for current access and 100 AED placements. Priority rankings were then given for each area based on their access score and predicted number of OHCA events.ResultsOf the 2802 OHCA events, 64.28% occurred in rural areas, and 35.72% in urban areas. Additionally, over 70% of individuals were aged over 65. Supply of AEDs was less than demand in most areas. Priority regions for AED placement were identified, and access scores were evaluated for AED placement methodology by ranking the access scores and the predicted OHCA count. AED placement methodology placed AEDs in areas with the highest priority, but placed more AEDs in areas with more predicted OHCA events in each grid cell.ConclusionThe methods in this paper incorporate OHCA spatial risk factors and OHCA coverage to identify spatial regions most in need of resources. These methods can be used to help understand how AED allocation methods affect OHCA accessibility, which is of significant practical value for communities when deciding AED placements.

Dataset Information

A comparison of Bayesian and frequentist model selection methods for factor analysis models.

Publications

A comparison of Bayesian and frequentist model selection methods for factor analysis models.

Similar Datasets

OmicsDI is part of the ELIXIR infrastructure