On the Performance of Semi- and Nonparametric Item Response Functions in Computer Adaptive Tests.
Ontology highlight
ABSTRACT:
SUBMITTER: Falk CF
PROVIDER: S-EPMC8721622 | biostudies-literature |
REPOSITORIES: biostudies-literature
Similar Datasets
Project description:When dealing with classical spike train analysis, the practitioner often performs goodness-of-fit tests to test whether the observed process is a Poisson process, for instance, or if it obeys another type of probabilistic model (Yana et al. in Biophys. J. 46(3):323-330, 1984; Brown et al. in Neural Comput. 14(2):325-346, 2002; Pouzat and Chaffiol in Technical report, http://arxiv.org/abs/arXiv:0909.2785, 2009). In doing so, there is a fundamental plug-in step, where the parameters of the supposed underlying model are estimated. The aim of this article is to show that plug-in has sometimes very undesirable effects. We propose a new method based on subsampling to deal with those plug-in issues in the case of the Kolmogorov-Smirnov test of uniformity. The method relies on the plug-in of good estimates of the underlying model that have to be consistent with a controlled rate of convergence. Some nonparametric estimates satisfying those constraints in the Poisson or in the Hawkes framework are highlighted. Moreover, they share adaptive properties that are useful from a practical point of view. We show the performance of those methods on simulated data. We also provide a complete analysis with these tools on single unit activity recorded on a monkey during a sensory-motor task.Electronic Supplementary MaterialThe online version of this article (doi:10.1186/2190-8567-4-3) contains supplementary material.
| S-EPMC4021619 | biostudies-literature
Project description:PurposeTo compare the performance of the Patient-Reported Outcomes Measurement Information System (PROMIS) physical function (PF), pain interference (PIF), and depression computer adaptive tests (CAT) relative to legacy instruments in patients undergoing meniscal surgery.MethodsPatients scheduled to undergo meniscal surgery completed legacy knee function PROMs (International Knee Documentation Committee [IKDC], Knee Injury and Osteoarthritis Outcome Score [KOOS] subscores), Marx Activity Rating Scale (MARS), Veterans-Rand 12 (VR12), Short Form 12 (SF12), and the Brief Resilience Scale (BRS) alongside PROMIS PF, PIF, and Depression preoperatively. Spearman rank correlations were calculated, and score distributions were examined for floor and ceiling effects.Results152 patients (46.6 ± 14.9 years, 67.1% male) completed PROMs for appropriate inclusion. PROMIS PF yielded high-moderate to high correlations with the IKDC and KOOS subscales (r = 0.61 to 0.73), demonstrating similar performance to the IKDC. PROMIS PIF demonstrated moderately high-moderate to high correlations with the IKDC, KOOS subscales, VR-12 Physical Component Score (PCS), and SF12 PCS (r = 0.62 to 0.71), performing comparably to KOOS Pain (r = 0.55 to 0.92). PROMIS Depression demonstrated moderate to high-moderate correlations with the mental health legacies (r = 0.46 to 0.66). Significant ceiling effects were observed for MARS (n = 29, 18.8%), and significant floor effects were exhibited by PROMIS Depression (n = 38, 25%) and MARS (n = 27, 17.6%).ConclusionThe PROMIS PF, PIF CAT, and Depression instruments exhibit comparable performance profiles relative to legacy knee PROMs. PROMIS PF and PIF demonstrated no floor and ceiling effects, whereas PROMIS Depression exhibited a significant relative floor effect. PROMIS PF and PIF may be appropriately used to establish functional baselines preoperatively.Level of evidenceIV, diagnostic case series.
| S-EPMC7588601 | biostudies-literature
Project description:Currently, there are two predominant approaches in adaptive testing. One, referred to as cognitive diagnosis computerized adaptive testing (CD-CAT), is based on cognitive diagnosis models, and the other, the traditional CAT, is based on item response theory. The present study evaluates the performance of two item selection rules (ISRs) originally developed in the CD-CAT framework, the double Kullback-Leibler information (DKL) and the generalized deterministic inputs, noisy "and" gate model discrimination index (GDI), in the context of traditional CAT. The accuracy and test security associated with these two ISRs are compared to those of the point Fisher information and weighted KL using a simulation study. The impact of the trait level estimation method is also investigated. The results show that the new ISRs, particularly DKL, could be used to improve the accuracy of CAT. Better accuracy for DKL is achieved at the expense of higher item overlap rate. Differences among the item selection rules become smaller as the test gets longer. The two CD-CAT ISRs select different types of items: items with the highest possible a parameter with DKL, and items with the lowest possible c parameter with GDI. Regarding the trait level estimator, expected a posteriori method is generally better in the first stages of the CAT, and converges with the maximum likelihood method when a medium to large number of items are involved. The use of DKL can be recommended in low-stakes settings where test security is less of a concern.
| S-EPMC6953845 | biostudies-literature
Project description:PurposePrevious studies using the WHOQOL measures have demonstrated that the relationship between individual items and the underlying quality of life (QoL) construct may differ between cultures. If unaccounted for, these differing relationships can lead to measurement bias which, in turn, can undermine the reliability of results.MethodsWe used item response theory (IRT) to assess differential item functioning (DIF) in WHOQOL data from diverse language versions collected in UK, Zimbabwe, Russia, and India (total N = 1332). Data were fitted to the partial credit 'Rasch' model. We used four item banks previously derived from the WHOQOL-100 measure, which provided excellent measurement for physical, psychological, social, and environmental quality of life domains (40 items overall). Cross-cultural differential item functioning was assessed using analysis of variance for item residuals and post hoc Tukey tests. Simulated computer-adaptive tests (CATs) were conducted to assess the efficiency and precision of the four items banks.ResultsSplitting item parameters by DIF results in four linked item banks without DIF or other breaches of IRT model assumptions. Simulated CATs were more precise and efficient than longer paper-based alternatives.DiscussionAssessing differential item functioning using item response theory can identify measurement invariance between cultures which, if uncontrolled, may undermine accurate comparisons in computer-adaptive testing assessments of QoL. We demonstrate how compensating for DIF using item anchoring allowed data from all four countries to be compared on a common metric, thus facilitating assessments which were both sensitive to cultural nuance and comparable between countries.
| S-EPMC5874271 | biostudies-literature
Project description:In dichotomous item response theory (IRT) framework, the asymptotic standard error (ASE) is the most common statistic to evaluate the precision of various ability estimators. Easy-to-use ASE formulas are readily available; however, the accuracy of some of these formulas was recently questioned and new ASE formulas were derived from a general asymptotic theory framework. Furthermore, exact standard errors were suggested to better evaluate the precision of ability estimators, especially with short tests for which the asymptotic framework is invalid. Unfortunately, the accuracy of exact standard errors was assessed so far only in a very limiting setting. The purpose of this article is to perform a global comparison of exact versus (classical and new formulations of) asymptotic standard errors, for a wide range of usual IRT ability estimators, IRT models, and with short tests. Results indicate that exact standard errors globally outperform the ASE versions in terms of reduced bias and root mean square error, while the new ASE formulas are also globally less biased than their classical counterparts. Further discussion about the usefulness and practical computation of exact standard errors are outlined.
| S-EPMC7221492 | biostudies-literature
Project description:The use of patient-reported outcome instruments (PRO) in clinical trials in order to capture the impacts of treatment on patients is widespread. However, regulatory agencies have over the past decade highlighted the need for PROs that are fit for purpose and target relevant aspects of the patient's condition. Many legacy PROs were developed with little patient input, are lengthy, and may lack relevance having not been modified or adapted as medical treatments have advanced. Computer-adaptive test (CAT) systems provide the possibility of targeted approaches to capturing patient-centric data, while minimising patient burden. Coupled with greater patient input in the development of PROs, CAT offers the opportunity of overcoming the shortcomings of the previous generation of PROs. This paper describes the some of the issues facing legacy PROs, current regulatory guidance, and initiatives, such as the Patient-Reported Outcome Measurement Information System (PROMIS), as well as the early signs of use of CAT to capture PRO data in clinical trials.
| S-EPMC6299156 | biostudies-literature
Project description:How to detect faking on personality measures has been investigated using various methods and procedures. As previous findings are mixed and rarely based on ideal point item response theory models, additional research is needed for further exploration. This study modeled the responses of personality tests using ideal point method across instructed faking and honest responding conditions. A sample of undergraduate students participated the within-subjects measures to examine how the item location parameter derived from the generalized graded unfolding model changed, and how individuals' perception about items changed when faked. The mean test scores of faking group was positively correlated to the magnitude of within-subjects score change. The item-level analysis revealed both conscientiousness items (18.8%) and neuroticism items (50.0%) appeared significant shifts on item parameters, suggesting that response pattern changed from honest to faking conditions. The direction of the change appeared both in positive and negative way, demonstrating that faking could increase or decrease personality factor scores. The results indicated that the changes of perceptions on items could be operated by faking, offering some support for the ideal point model to be an adequate measure for detecting faking. However, the findings of diagnostic accuracy analysis also implied that the appropriateness of ideal point models for detecting faking should be under consideration, also be used with caution. Implications, further research directions, and limitations are discussed.
| S-EPMC6987465 | biostudies-literature
Project description:Current use of multidimensional computerized adaptive testing (MCAT) has been developed in conjunction with compensatory multidimensional item response theory (MIRT) models rather than with non-compensatory ones. In recognition of the usefulness of MCAT and the complications associated with non-compensatory data, this study aimed to develop MCAT algorithms using non-compensatory MIRT models and to evaluate their performance. For the purpose of the study, three item selection methods were adapted and compared, namely, the Fisher information method, the mutual information method, and the Kullback-Leibler information method. The results of a series of simulations showed that the Fisher information and mutual information methods performed similarly, and both outperformed the Kullback-Leibler information method. In addition, it was found that the more stringent the termination criterion and the higher the correlation between the latent traits, the higher the resulting measurement precision and test reliability. Test reliability was very similar across the dimensions, regardless of the correlation between the latent traits and termination criterion. On average, the difficulties of the administered items were found to be at a lower level than the examinees' abilities, which shed light on item bank construction for non-compensatory items.
| S-EPMC6696872 | biostudies-literature
Project description:Differential item functioning (DIF) occurs when people with the same proficiency have different probabilities of giving a certain response to an item. The present study focused on an assumption implicit in popular methods for DIF testing that has received little attention in published literature (item residual homogeneity). The assumption is explained, a strategy for detecting violations of it (i.e., item residual heterogeneity) is illustrated with empirical data, and simulations are carried out to evaluate the performance of binary logistic regression, two-group item response theory (IRT), and the Mantel-Haenszel (MH) test in the presence of item residual heterogeneity. Results indicated that heterogeneity inflated Type I error and attenuated power for logistic regression, and attenuated power and produced biased estimates of the latent focal group mean and standard deviation for two-group IRT. The MH test was robust to item residual heterogeneity, probably because it does not use the logistic function.
| S-EPMC5978544 | biostudies-literature
Project description:Modern psychometric theory provides many useful tools for ability testing, such as item response theory, computerised adaptive testing, and automatic item generation. However, these techniques have yet to be integrated into mainstream psychological practice. This is unfortunate, because modern psychometric techniques can bring many benefits, including sophisticated reliability measures, improved construct validity, avoidance of exposure effects, and improved efficiency. In the present research we therefore use these techniques to develop a new test of a well-studied psychological capacity: melodic discrimination, the ability to detect differences between melodies. We calibrate and validate this test in a series of studies. Studies 1 and 2 respectively calibrate and validate an initial test version, while Studies 3 and 4 calibrate and validate an updated test version incorporating additional easy items. The results support the new test's viability, with evidence for strong reliability and construct validity. We discuss how these modern psychometric techniques may also be profitably applied to other areas of music psychology and psychological science in general.
| S-EPMC5472621 | biostudies-literature