Browse
Submit Data
Databases
API
Help

Dataset Information

0 Views

0 Connections

0 Citations

0 Reanalyses

0 Downloads

Omics score: 0

On the Performance of Semi- and Nonparametric Item Response Functions in Computer Adaptive Tests.

ABSTRACT: Large-scale assessments often use a computer adaptive test (CAT) for selection of items and for scoring respondents. Such tests often assume a parametric form for the relationship between item responses and the underlying construct. Although semi- and nonparametric response functions could be used, there is scant research on their performance in a CAT. In this work, we compare parametric response functions versus those estimated using kernel smoothing and a logistic function of a monotonic polynomial. Monotonic polynomial items can be used with traditional CAT item selection algorithms that use analytical derivatives. We compared these approaches in CAT simulations with a variety of item selection algorithms. Our simulations also varied the features of the calibration and item pool: sample size, the presence of missing data, and the percentage of nonstandard items. In general, the results support the use of semi- and nonparametric item response functions in a CAT.

SUBMITTER: Falk CF

PROVIDER: S-EPMC8721622 | biostudies-literature | 2022 Feb

REPOSITORIES: biostudies-literature

ACCESS DATA

Json Xml

Publications

On the Performance of Semi- and Nonparametric Item Response Functions in Computer Adaptive Tests.

Falk Carl F CF Feuerstahler Leah M LM

Educational and psychological measurement 20210518 1

Large-scale assessments often use a computer adaptive test (CAT) for selection of items and for scoring respondents. Such tests often assume a parametric form for the relationship between item responses and the underlying construct. Although semi- and nonparametric response functions could be used, there is scant research on their performance in a CAT. In this work, we compare parametric response functions versus those estimated using kernel smoothing and a logistic function of a monotonic polyn ...[more]

PMID: 34987268

Similar Datasets

Goodness-of-Fit Tests and Nonparametric Adaptive Estimation for Spike Train Analysis.

Project description:When dealing with classical spike train analysis, the practitioner often performs goodness-of-fit tests to test whether the observed process is a Poisson process, for instance, or if it obeys another type of probabilistic model (Yana et al. in Biophys. J. 46(3):323-330, 1984; Brown et al. in Neural Comput. 14(2):325-346, 2002; Pouzat and Chaffiol in Technical report, http://arxiv.org/abs/arXiv:0909.2785, 2009). In doing so, there is a fundamental plug-in step, where the parameters of the supposed underlying model are estimated. The aim of this article is to show that plug-in has sometimes very undesirable effects. We propose a new method based on subsampling to deal with those plug-in issues in the case of the Kolmogorov-Smirnov test of uniformity. The method relies on the plug-in of good estimates of the underlying model that have to be consistent with a controlled rate of convergence. Some nonparametric estimates satisfying those constraints in the Poisson or in the Hawkes framework are highlighted. Moreover, they share adaptive properties that are useful from a practical point of view. We show the performance of those methods on simulated data. We also provide a complete analysis with these tools on single unit activity recorded on a monkey during a sensory-motor task.Electronic Supplementary MaterialThe online version of this article (doi:10.1186/2190-8567-4-3) contains supplementary material.

| S-EPMC4021619 | biostudies-literature

Semi-Parametric Item Response Theory With O'Sullivan Splines for Item Responses and Response Time.

Project description:Response time (RT) has been an essential resource for supplementing the estimation accuracy of latent traits and item parameters in educational testing. Most item response theory (IRT) approaches are based on parametric RT models. However, since test takers may alter their behaviors during a test due to motivation or strategy shifts, fatigue, or other causes, parametric IRT models are unlikely to capture such subtle and nonlinear information. In this work, we propose a novel semi-parametric IRT model with O'Sullivan splines to accommodate the flexible mean RT shapes and explore the underlying nonlinear relationships between latent traits and RT. A simulation study was conducted to demonstrate the substantial improvement in parameter estimation achieved by the new model, as well as the detriment of using parametric models in terms of biases and measurement errors. Using this model, a dataset of mathematics test scores and RT from the Programme for International Student Assessment was analyzed to demonstrate the evident nonlinearity and to compare the proposed model with existing models in terms of model fitting. The findings presented in this study indicate the promising nature of the new approach, suggesting its potential as an additional psychometric tool to enhance test reliability and reduce measurement errors.

| S-EPMC11789044 | biostudies-literature

Performance of PROMIS Physical Function, Pain Interference, and Depression Computer Adaptive Tests Instruments in Patients Undergoing Meniscal Surgery.

Project description:PurposeTo compare the performance of the Patient-Reported Outcomes Measurement Information System (PROMIS) physical function (PF), pain interference (PIF), and depression computer adaptive tests (CAT) relative to legacy instruments in patients undergoing meniscal surgery.MethodsPatients scheduled to undergo meniscal surgery completed legacy knee function PROMs (International Knee Documentation Committee [IKDC], Knee Injury and Osteoarthritis Outcome Score [KOOS] subscores), Marx Activity Rating Scale (MARS), Veterans-Rand 12 (VR12), Short Form 12 (SF12), and the Brief Resilience Scale (BRS) alongside PROMIS PF, PIF, and Depression preoperatively. Spearman rank correlations were calculated, and score distributions were examined for floor and ceiling effects.Results152 patients (46.6 ± 14.9 years, 67.1% male) completed PROMs for appropriate inclusion. PROMIS PF yielded high-moderate to high correlations with the IKDC and KOOS subscales (r = 0.61 to 0.73), demonstrating similar performance to the IKDC. PROMIS PIF demonstrated moderately high-moderate to high correlations with the IKDC, KOOS subscales, VR-12 Physical Component Score (PCS), and SF12 PCS (r = 0.62 to 0.71), performing comparably to KOOS Pain (r = 0.55 to 0.92). PROMIS Depression demonstrated moderate to high-moderate correlations with the mental health legacies (r = 0.46 to 0.66). Significant ceiling effects were observed for MARS (n = 29, 18.8%), and significant floor effects were exhibited by PROMIS Depression (n = 38, 25%) and MARS (n = 27, 17.6%).ConclusionThe PROMIS PF, PIF CAT, and Depression instruments exhibit comparable performance profiles relative to legacy knee PROMs. PROMIS PF and PIF demonstrated no floor and ceiling effects, whereas PROMIS Depression exhibited a significant relative floor effect. PROMIS PF and PIF may be appropriately used to establish functional baselines preoperatively.Level of evidenceIV, diagnostic case series.

| S-EPMC7588601 | biostudies-literature

Adapting cognitive diagnosis computerized adaptive testing item selection rules to traditional item response theory.

Project description:Currently, there are two predominant approaches in adaptive testing. One, referred to as cognitive diagnosis computerized adaptive testing (CD-CAT), is based on cognitive diagnosis models, and the other, the traditional CAT, is based on item response theory. The present study evaluates the performance of two item selection rules (ISRs) originally developed in the CD-CAT framework, the double Kullback-Leibler information (DKL) and the generalized deterministic inputs, noisy "and" gate model discrimination index (GDI), in the context of traditional CAT. The accuracy and test security associated with these two ISRs are compared to those of the point Fisher information and weighted KL using a simulation study. The impact of the trait level estimation method is also investigated. The results show that the new ISRs, particularly DKL, could be used to improve the accuracy of CAT. Better accuracy for DKL is achieved at the expense of higher item overlap rate. Differences among the item selection rules become smaller as the test gets longer. The two CD-CAT ISRs select different types of items: items with the highest possible a parameter with DKL, and items with the lowest possible c parameter with GDI. Regarding the trait level estimator, expected a posteriori method is generally better in the first stages of the CAT, and converges with the maximum likelihood method when a medium to large number of items are involved. The use of DKL can be recommended in low-stakes settings where test security is less of a concern.

| S-EPMC6953845 | biostudies-literature

Nonparametric optimal designs for degradation tests.

Project description:This paper discusses the optimal design of a degradation test within the nonparametric framework by assuming the underlying process to be an empirical Lévy process with heterogeneity. The optimization of the experiment relies on the design variables including the sample size, the measurement frequency and total operation time. It is necessary to investigate the effect of the design variables on the estimation of first passage time (FPT). Subject to total experimental cost not exceeding a pre-specified budget, the values of design variables are determined such that the bootstrap mean square error of the 100p th percentile of the FPT distribution is minimized.

| S-EPMC9041718 | biostudies-literature

Adjusting for cross-cultural differences in computer-adaptive tests of quality of life.

Project description:PurposePrevious studies using the WHOQOL measures have demonstrated that the relationship between individual items and the underlying quality of life (QoL) construct may differ between cultures. If unaccounted for, these differing relationships can lead to measurement bias which, in turn, can undermine the reliability of results.MethodsWe used item response theory (IRT) to assess differential item functioning (DIF) in WHOQOL data from diverse language versions collected in UK, Zimbabwe, Russia, and India (total N = 1332). Data were fitted to the partial credit 'Rasch' model. We used four item banks previously derived from the WHOQOL-100 measure, which provided excellent measurement for physical, psychological, social, and environmental quality of life domains (40 items overall). Cross-cultural differential item functioning was assessed using analysis of variance for item residuals and post hoc Tukey tests. Simulated computer-adaptive tests (CATs) were conducted to assess the efficiency and precision of the four items banks.ResultsSplitting item parameters by DIF results in four linked item banks without DIF or other breaches of IRT model assumptions. Simulated CATs were more precise and efficient than longer paper-based alternatives.DiscussionAssessing differential item functioning using item response theory can identify measurement invariance between cultures which, if uncontrolled, may undermine accurate comparisons in computer-adaptive testing assessments of QoL. We demonstrate how compensating for DIF using item anchoring allowed data from all four countries to be compared on a common metric, thus facilitating assessments which were both sensitive to cultural nuance and comparable between countries.

| S-EPMC5874271 | biostudies-literature

Efficient Standard Errors in Item Response Theory Models for Short Tests.

Project description:In dichotomous item response theory (IRT) framework, the asymptotic standard error (ASE) is the most common statistic to evaluate the precision of various ability estimators. Easy-to-use ASE formulas are readily available; however, the accuracy of some of these formulas was recently questioned and new ASE formulas were derived from a general asymptotic theory framework. Furthermore, exact standard errors were suggested to better evaluate the precision of ability estimators, especially with short tests for which the asymptotic framework is invalid. Unfortunately, the accuracy of exact standard errors was assessed so far only in a very limiting setting. The purpose of this article is to perform a global comparison of exact versus (classical and new formulations of) asymptotic standard errors, for a wide range of usual IRT ability estimators, IRT models, and with short tests. Results indicate that exact standard errors globally outperform the ASE versions in terms of reduced bias and root mean square error, while the new ASE formulas are also globally less biased than their classical counterparts. Further discussion about the usefulness and practical computation of exact standard errors are outlined.

| S-EPMC7221492 | biostudies-literature

A Bayesian nonparametric approach to dynamic item-response modeling: An application to the GUSTO cohort study.

Project description:Statistical analysis of questionnaire data is often performed employing techniques from item-response theory. In this framework, it is possible to differentiate respondent profiles and characterize the questions (items) included in the questionnaire via interpretable parameters. These models are often crosssectional and aim at evaluating the performance of the respondents. The motivating application of this work is the analysis of psychometric questionnaires taken by a group of mothers at different time points and by their children at one later time point. The data are available through the GUSTO cohort study. To this end, we propose a Bayesian semiparametric model and extend the current literature by: (i) introducing temporal dependence among questionnaires taken at different time points; (ii) jointly modeling the responses to questionnaires taken from different, but related, groups of subjects (in our case mothers and children), introducing a further dependency structure and therefore sharing of information; (iii) allowing clustering of subjects based on their latent response profile. The proposed model is able to identify three main groups of mother/child pairs characterized by their response profiles. Furthermore, we report an interesting maternal reporting bias effect strongly affecting the clustering structure of the mother/child dyads.

| S-EPMC9546363 | biostudies-literature

Item banking and computer-adaptive testing in clinical trials: Standing in sight of the PROMISed land.

Project description:The use of patient-reported outcome instruments (PRO) in clinical trials in order to capture the impacts of treatment on patients is widespread. However, regulatory agencies have over the past decade highlighted the need for PROs that are fit for purpose and target relevant aspects of the patient's condition. Many legacy PROs were developed with little patient input, are lengthy, and may lack relevance having not been modified or adapted as medical treatments have advanced. Computer-adaptive test (CAT) systems provide the possibility of targeted approaches to capturing patient-centric data, while minimising patient burden. Coupled with greater patient input in the development of PROs, CAT offers the opportunity of overcoming the shortcomings of the previous generation of PROs. This paper describes the some of the issues facing legacy PROs, current regulatory guidance, and initiatives, such as the Patient-Reported Outcome Measurement Information System (PROMIS), as well as the early signs of use of CAT to capture PRO data in clinical trials.

| S-EPMC6299156 | biostudies-literature

Efficiency Analysis of Item Response Theory Kernel Equating for Mixed-Format Tests.

Project description:This study aims to evaluate the performance of Item Response Theory (IRT) kernel equating in the context of mixed-format tests by comparing it to IRT observed score equating and kernel equating with log-linear presmoothing. Comparisons were made through both simulations and real data applications, under both equivalent groups (EG) and non-equivalent groups with anchor test (NEAT) sampling designs. To prevent bias towards IRT methods, data were simulated with and without the use of IRT models. The results suggest that the difference between IRT kernel equating and IRT observed score equating is minimal, both in terms of the equated scores and their standard errors. The application of IRT models for presmoothing yielded smaller standard error of equating than the log-linear presmoothing approach. When test data were generated using IRT models, IRT-based methods proved less biased than log-linear kernel equating. However, when data were simulated without IRT models, log-linear kernel equating showed less bias. Overall, IRT kernel equating shows great promise when equating mixed-format tests.

| S-EPMC10664743 | biostudies-literature

OmicsDI is part of the ELIXIR infrastructure

OmicsDI is an Elixir interoperability service. Learn more ›

Tweets

OmicsDI Databases

PRIDE
PeptideAtlas
MassIVE
JPOST Repository
Physiome Model Repository

EGA
EVA
ENA
LINCS
PAXDB
Cell Collective

MetaboLights
Metabolomics Workbench
MetabolomeExpress
GNPS
BioModels
FAIRDOMHub

ArrayExpress
dbGaP
ExpressionAtlas
GEO
NODE

Information

Databases
Help
API
Contact us
Code on GitHub
Terms of use
Submit Data