Comparison of Relative Fit Indices for Diagnostic Model Selection.
Ontology highlight
ABSTRACT:
SUBMITTER: Sen S
PROVIDER: S-EPMC5978522 | biostudies-literature | 2017 Sep
REPOSITORIES: biostudies-literature
Publications
Comparison of Relative Fit Indices for Diagnostic Model Selection.
Applied psychological measurement 20170308 6
The purpose of this study was to thoroughly examine the performance of three information-based fit indices-Akaike's Information Criterion (AIC), Bayesian Information Criterion (BIC), and sample-size-adjusted BIC (SABIC)-using the log-linear cognitive diagnosis model and a set of well-known item response theory (IRT) models. Two simulation studies were conducted to examine the extent to which relative fit indices can identify the generating model under a variety of data conditions and model missp ...[more]
Similar Datasets
Project description:In applications of cognitive diagnostic models (CDMs), practitioners usually face the difficulty of choosing appropriate CDMs and building accurate Q-matrices. However, functions of model-fit indices that are supposed to inform model and Q-matrix choices are not well understood. This study examines the performance of several promising model-fit indices in selecting model and Q-matrix under different sample size conditions. Relative performance between Akaike information criterion and Bayesian information criterion in model and Q-matrix selection appears to depend on the complexity of data generating models, Q-matrices, and sample sizes. Among the absolute fit indices, MX2 is least sensitive to sample size under correct model and Q-matrix specifications, and performs the best in power. Sample size is found to be the most influential factor on model-fit index values. Consequences of selecting inaccurate model and Q-matrix in classification accuracy of attribute mastery are also evaluated.
| S-EPMC5978496 | biostudies-literature
Project description:This study examined whether cutoffs in fit indices suggested for traditional formats with maximum likelihood estimators can be utilized to assess model fit and to test measurement invariance when a multiple group confirmatory factor analysis was employed for the Thurstonian item response theory (IRT) model. Regarding the performance of the evaluation criteria, detection of measurement non-invariance and Type I error rates were examined. The impact of measurement non-invariance on estimated scores in the Thurstonian IRT model was also examined through accuracy and efficiency in score estimation. The fit indices used for the evaluation of model fit performed well. Among six cutoffs for changes in model fit indices, only ΔCFI > .01 and ΔNCI > .02 detected metric non-invariance when the medium magnitude of non-invariance occurred and none of the cutoffs performed well to detect scalar non-invariance. Based on the generated sampling distributions of fit index differences, this study suggested ΔCFI > .001 and ΔNCI > .004 for scalar non-invariance and ΔCFI > .007 for metric non-invariance. Considering Type I error rate control and detection rates of measurement non-invariance, ΔCFI was recommended for measurement non-invariance tests for forced-choice format data. Challenges in measurement non-invariance tests in the Thurstonian IRT model were discussed along with the direction for future research to enhance the utility of forced-choice formats in test development for cross-cultural and international settings.
| S-EPMC7262996 | biostudies-literature
Project description:It is regarded as best practice in phylogenetic reconstruction to perform relative model selection to determine an appropriate evolutionary model for the data. This procedure ranks a set of candidate models according to their goodness of fit to the data, commonly using an information theoretic criterion. Users then specify the best-ranking model for inference. Although it is often assumed that better-fitting models translate to increase accuracy, recent studies have shown that the specific model employed may not substantially affect inferences. We examine whether there is a systematic relationship between relative model fit and topological inference accuracy in protein phylogenetics, using simulations and real sequences. Simulations employed site-heterogeneous mechanistic codon models that are distinct from protein-level phylogenetic inference models, allowing us to investigate how protein models performs when they are misspecified to the data, as will be the case for any real sequence analysis. We broadly find that phylogenies inferred across models with vastly different fits to the data produce highly consistent topologies. We additionally find that all models infer similar proportions of false-positive splits, raising the possibility that all available models of protein evolution are similarly misspecified. Moreover, we find that the parameter-rich GTR (general time reversible) model, whose amino acid exchangeabilities are free parameters, performs similarly to models with fixed exchangeabilities, although the inference precision associated with GTR models was not examined. We conclude that, although relative model selection may not hinder phylogenetic analysis on protein data, it may not offer specific predictable improvements and is not a reliable proxy for accuracy.
| S-EPMC7306691 | biostudies-literature
Project description:BACKGROUND: Vertigo and dizziness are symptoms which are reported frequently in clinical practice. We aimed to develop diagnostic indices for four prevalent vertiginous diseases: benign paroxysmal positional vertigo (BPPV), Menière's disease (MD), vestibular migraine (VM), and phobic postural vertigo (PPV). METHODS: Based on a detailed questionnaire handed out to consecutive patients presenting for the first time in our dizziness clinic we preselected a set of seven questions with desirable diagnostic properties when compared with the final diagnosis after medical workup. Using exact logistic regression analysis diagnostic scores, each comprising of four to six items that can simply be added up, were built for each of the four diagnoses. RESULTS: Of 193 patients 131 questionnaires were left after excluding those with missing consent or data. Applying the suggested cut-off points, sensitivity and specificity were 87.5 and 93.5% for BPPV, 100 and 87.4% for MD, 92.3 and 83.7% for VM, 73.7 and 84.1% for PPV, respectively. By changing the cut-off points sensitivity and specificity can be adjusted to meet diagnostic needs. CONCLUSIONS: The diagnostic indices showed promising diagnostic properties. Once further validated, they could provide an ease to use and yet flexible tool for screening vertigo in clinical practice and epidemiological research.
| S-EPMC2987857 | biostudies-other
Project description:Colorectal cancer (CRC) has high incidence and is associated with high case fatality. In France, the 5-year survival, pooled across all cancer stages at diagnosis, ranges from 57% in men to 60% in women. About one third of patients diagnosed with CRC will develop a metachronous recurrence during the following years. It is of paramount importance to accurately identify factors associated with the increased risk of progression and death, in order to develop effective follow-up and treatment strategies. However, to accurately assess the role of patients’ specific characteristics in the progression of cancer several methodological challenges need to be overcome.
One difficulty, common to prognostic studies of cancer, concerns the need to separate the effects of prognostic factors on different clinical endpoints, such as disease recurrence vs recurrence-free death. Another difficulty, encountered in prognostic studies, is that the cause of death is not available or not accurately coded. Yet, some patients are likely to die of causes not related to the disease of primary interest, especially in cancers with longer survival and in those that affect older subjects. Until recently, the existing statistical methodology was not able to simultaneously, deal with both difficulties, i.e. to account for (i) possibly different effects of prognostic factors on death vs recurrence, and (ii) unknown causes of death. However, this challenge has been addressed by the recent development of the Markov relative survival model (MRS) , which extends the Markov multi-state model to incorporate relative survival modelling. Simulations demonstrate that MRS is able to accurately estimate different effects of prognostic factors on the risk of each of several events, including separate effects on disease-specific vs other causes of death. To date, the MRS had not been applied in clinical or epidemiological studies.
The aim of this study was to assess the potential advantages of the new multi-state relative survival model (MRS), proposed by Huszti et al. (2012), in a prognostic cancer study. To this end, we compared the MRS results with those obtained with two more conventional analyses, based on Cox’s proportional hazards model, and the multi-state Markov model proposed by Alioum and Commenges (2001). The three models were applied to explore the impact of prognostic factors on cancer-specific mortality and recurrence, in a large population-based French registry of colorectal cancer, with up to 25 years of follow-up.
| 2224546 | ecrin-mdr-crc
Project description:BackgroundCo-expression measures are often used to define networks among genes. Mutual information (MI) is often used as a generalized correlation measure. It is not clear how much MI adds beyond standard (robust) correlation measures or regression model based association measures. Further, it is important to assess what transformations of these and other co-expression measures lead to biologically meaningful modules (clusters of genes).ResultsWe provide a comprehensive comparison between mutual information and several correlation measures in 8 empirical data sets and in simulations. We also study different approaches for transforming an adjacency matrix, e.g. using the topological overlap measure. Overall, we confirm close relationships between MI and correlation in all data sets which reflects the fact that most gene pairs satisfy linear or monotonic relationships. We discuss rare situations when the two measures disagree. We also compare correlation and MI based approaches when it comes to defining co-expression network modules. We show that a robust measure of correlation (the biweight midcorrelation transformed via the topological overlap transformation) leads to modules that are superior to MI based modules and maximal information coefficient (MIC) based modules in terms of gene ontology enrichment. We present a function that relates correlation to mutual information which can be used to approximate the mutual information from the corresponding correlation coefficient. We propose the use of polynomial or spline regression models as an alternative to MI for capturing non-linear relationships between quantitative variables.ConclusionThe biweight midcorrelation outperforms MI in terms of elucidating gene pairwise relationships. Coupled with the topological overlap matrix transformation, it often leads to more significantly enriched co-expression modules. Spline and polynomial networks form attractive alternatives to MI in case of non-linear relationships. Our results indicate that MI networks can safely be replaced by correlation networks when it comes to measuring co-expression relationships in stationary data.
| S-EPMC3586947 | biostudies-literature
Project description:Network models are applied in numerous domains where data arise from systems of interactions among pairs of actors. Both statistical and mechanistic network models are increasingly capable of capturing various dependencies among these actors. Yet, these dependencies pose statistical challenges for analyzing such data, especially when the data set comprises only a single observation of one network, often leading to intractable likelihoods regardless of the modeling paradigm and limiting the application of existing statistical methods for networks. We explore a subsampling bootstrap procedure to serve as the basis for goodness of fit and model selection with a single observed network that circumvents the intractability of such likelihoods. Our approach is based on flexible resampling distributions formed from the single observed network, allowing for more nuanced and higher dimensional comparisons than point estimates of quantities of interest. We include worked examples for model selection, with simulation, and assessment of goodness of fit, with duplication-divergence model fits for yeast (S.cerevisiae) protein-protein interaction data from the literature. The proposed approach produces a flexible resampling distribution that can be based on any network statistics of one's choosing and can be employed for both statistical and mechanistic network models.
| S-EPMC6854093 | biostudies-literature
Project description:Visual attention can be tuned to specific features to aid in visual search. The way in which these search strategies are established and maintained is flexible, reflecting goal-directed attentional control, but can exert a persistent effect on selection that remains even when these strategies are no longer advantageous, reflecting an attentional bias driven by selection history. Apart from feature-specific search, recent studies have shown that attention can be tuned to target-nontarget relationships. Here we tested whether a relational search strategy continues to bias attention in a subsequent task, where the relationally better color and former target color both serve as distractors (Experiment 1) or as potential targets (Experiment 2). We demonstrate that a relational bias can persist in a subsequent task in which color serves as a task-irrelevant feature, both impairing and facilitating visual search performance. Our findings extend our understanding of the relational account of attentional control and the nature of selection history effects on attention.
| S-EPMC7484361 | biostudies-literature
Project description:Researchers contribute to the frontiers of knowledge by establishing facts and reaching new conclusions through systematic investigations, and by subsequently publishing the outcomes of their research findings in the form of research papers. These research publications are indicative of researchers' scientific impact. Different bibliometric indices have been proposed to measure the impact or productivity of a researcher. These indices include publication count, citation count, number of coauthors, h-index, etc. The h-index, since its inception, has been ranked as the foremost impact indicator by many studies. However, as a consequence of the various short comings identified in h-index, some variants of h-index have been proposed. For instance, one dimension which requires significant attention is determining the ability of exceptional performers in a particular research area. In our study, we have compared effectiveness of h-index and some of its recent variants in identifying the exceptional performers of a field. We have also found correlation of h-index with recently proposed indices. A high correlation indicates same effect of these indices as of h-index and low correlation means these indices make non-redundant contribution while ranking potential researchers of a field of study. So far, effectiveness of these indices has not been explored/validated on real data sets of same field. We have considered these variants/modifications of h-index along with h-index and tested on comprehensive data set for the field of Computer Science. The Award winners' data set is considered as the benchmark for the evaluation of these indices for individual researchers. Results show that there is a low correlation of these indices with h-index, and in identifying exceptional performers of a field these indices perform better than h-index.
| S-EPMC7259586 | biostudies-literature
Project description:We introduce an approach for global fitting of the recently published high-throughput and high accuracy clonogenic cell-survival data for therapeutic scanned proton beams. Our fitting procedure accounts for the correlation between the cell-survival, the absorbed (physical) dose and the proton linear energy transfer (LET). The fitting polynomials and constraints have been constructed upon generalization of the microdosimetric kinetic model (gMKM) adapted to account for the low energy and high lineal-energy spectrum of the beam where the current radiobiological models may underestimate the reported relative biological effectiveness (RBE). The parameters (?, ?) of the linear-quadratic (LQ) model calculated by the presented method reveal a smooth transition from low to high LETs which is an advantage of the current method over methods previously employed to fit the same clonogenic data. Finally, the presented approach provides insight into underlying microscopic mechanisms which, with future study, may help to elucidate radiobiological responses along the Bragg curve and resolve discrepancies between experimental data and current RBE models.
| S-EPMC5567137 | biostudies-literature