Browse
Submit Data
Databases
API
Help

Dataset Information

14 Views

0 Connections

0 Citations

0 Reanalyses

0 Downloads

Omics score: 0

Evaluating Robust Scale Transformation Methods With Multiple Outlying Common Items Under IRT True Score Equating.

ABSTRACT: Item parameter estimates of a common item on a new test form may change abnormally due to reasons such as item overexposure or change of curriculum. A common item, whose change does not fit the pattern implied by the normally behaved common items, is defined as an outlier. Although improving equating accuracy, detecting and eliminating of outliers may cause a content imbalance among common items. Robust scale transformation methods have recently been proposed to solve this problem when only one outlier is present in the data, although it is not uncommon to see multiple outliers in practice. In this simulation study, the authors examined the robust scale transformation methods under conditions where there were multiple outlying common items. Results indicated that the robust scale transformation methods could reduce the influences of multiple outliers on scale transformation and equating. The robust methods performed similarly to a traditional outlier detection and elimination method in terms of reducing the influence of outliers while keeping adequate content balance.

SUBMITTER: He Y

PROVIDER: S-EPMC7262993 | biostudies-literature |

REPOSITORIES: biostudies-literature

ACCESS DATA

Json Xml

Similar Datasets

A Comparison of IRT Observed Score Kernel Equating and Several Equating Methods.

Project description:Item response theory (IRT) observed score kernel equating was evaluated and compared with equipercentile equating, IRT observed score equating, and kernel equating methods by varying the sample size and test length. Considering that IRT data simulation might unequally favor IRT equating methods, pseudo tests and pseudo groups were also constructed to make equating results comparable with those from the IRT data simulation. Identity equating and the large sample single group rule were both set as criterion equating (or true equating) on which local and global indices were based. Results show that in random equivalent groups design, IRT observed score kernel equating is more accurate and stable than others. In non-equivalent groups with anchor test design, IRT observed score equating shows lowest systematic and random errors among equating methods. Those errors decrease as a shorter test and a larger sample are used in equating; nevertheless, effect of the latter one is ignorable. No clear preference for data simulation method is found, though still affecting equating results. Preferences for true equating are spotted in random Equivalent Groups design. Finally, recommendations and further improvements are discussed.

| S-EPMC7069341 | biostudies-literature

What Affects the Quality of Score Transformations? Potential Issues in True-Score Equating Using the Partial Credit Model.

Project description:This simulation study investigated to what extent departures from construct similarity as well as differences in the difficulty and targeting of scales impact the score transformation when scales are equated by means of concurrent calibration using the partial credit model with a common person design. Practical implications of the simulation results are discussed with a focus on scale equating in health-related research settings. The study simulated data for two scales, varying the number of items and the sample sizes. The factor correlation between scales was used to operationalize construct similarity. Targeting of the scales was operationalized through increasing departure from equal difficulty and by varying the dispersion of the item and person parameters in each scale. The results show that low similarity between scales goes along with lower transformation precision. In cases with equal levels of similarity, precision improves in settings where the range of the item parameters is encompassing the person parameters range. With decreasing similarity, score transformation precision benefits more from good targeting. Difficulty shifts up to two logits somewhat increased the estimation bias but without affecting the transformation precision. The observed robustness against difficulty shifts supports the advantage of applying a true-score equating methods over identity equating, which was used as a naive baseline method for comparison. Finally, larger sample size did not improve the transformation precision in this study, longer scales improved only marginally the quality of the equating. The insights from the simulation study are used in a real-data example.

| S-EPMC10638984 | biostudies-literature

Asymptotic Standard Errors of Generalized Partial Credit Model True Score Equating Using Characteristic Curve Methods.

Project description:In this study, the delta method was applied to estimate the standard errors of the true score equating when using the characteristic curve methods with the generalized partial credit model in test equating under the context of the common-item nonequivalent groups equating design. Simulation studies were further conducted to compare the performance of the delta method with that of the bootstrap method and the multiple imputation method. The results indicated that the standard errors produced by the delta method were very close to the criterion empirical standard errors as well as those yielded by the bootstrap method and the multiple imputation method under all the manipulated conditions.

| S-EPMC8361376 | biostudies-literature

Can IRT Solve the Missing Data Problem in Test Equating?

Project description:In this paper test equating is considered as a missing data problem. The unobserved responses of the reference population to the new test must be imputed to specify a new cutscore. The proportion of students from the reference population that would have failed the new exam and those having failed the reference exam are made approximately the same. We investigate whether item response theory (IRT) makes it possible to identify the distribution of these missing responses and the distribution of test scores from the observed data without parametric assumptions for the ability distribution. We show that while the score distribution is not fully identifiable, the uncertainty about the score distribution on the new test due to non-identifiability is very small. Moreover, ignoring the non-identifiability issue and assuming a normal distribution for ability may lead to bias in test equating, which we illustrate in simulated and empirical data examples.

| S-EPMC4700285 | biostudies-literature

Unfolding IRT Models for Likert-Type Items With a Don't Know Option.

Project description:Attitude surveys are widely used in the social sciences. It has been argued that the underlying response process to attitude items may be more aligned with the ideal-point (unfolding) process than with the cumulative (dominance) process, and therefore, unfolding item response theory (IRT) models are more appropriate than dominance IRT models for these surveys. Missing data and don't know (DK) responses are common in attitude surveys, and they may not be ignorable in the likelihood for parameter estimation. Existing unfolding IRT models often treat missing data or DK as missing at random. In this study, a new class of unfolding IRT models for nonignorable missing data and DK were developed, in which the missingness and DK were assumed to measure a hierarchy of latent traits, which may be correlated with the latent attitude that a test intended to measure. The Bayesian approach with Markov chain Monte Carlo methods was used to estimate the parameters of the new models. Simulation studies demonstrated that the parameters were recovered fairly well, and ignoring nonignorable missingness or DK resulted in poor parameter estimates. An empirical example of a religious belief scale about health was given.

| S-EPMC5978636 | biostudies-literature

Measuring individual true change with PROMIS using IRT-based plausible values.

Project description:AimsA primary advantage of IRT-based patient-reported outcome measures such as PROMIS short forms and computer-adaptive tests is that each estimate of the latent trait comes with a standard error. Such measurement error needs to be acknowledged, in particular when monitoring individual patients over time. In this study, we use plausible values to account for measurement error and analyze the probability of true within-individual change.MethodsWe use a longitudinal, observational study of stable and exacerbated COPD patients (N = 185), providing PROMIS Physical Function and Fatigue T-scores over 3 months. At each measurement, we imputed 1000 plausible values from the scores' posterior distribution. These were then used to calculate probability of true change using a pre-specified threshold such as minimally important difference supported by the literature, or [Formula: see text] > 0. We demonstrate assessment of change in individuals and in groups, across different measures (Short Forms and CATs), and at various levels of confidence.ResultsUsing plausible value imputation and with 95% certainty, 47.5% of participants in the exacerbated group reported less fatigue, compared with 26.5% of participants in the stable group. Comparison of Short Forms and CATs suggests that CATs have better ability to detect change compared to short forms. We also illustrate this method using an individual's probability of change at different time points.ConclusionPlausible values offer a flexible way to include measurement error in analysis of individuals and on sample level. Assessment of probability of true change can complement existing distribution-based approaches and facilitates interpretation of improvement or decline.

| S-EPMC10849110 | biostudies-literature

Estimating True Student Growth Percentile Distributions Using Latent Regression Multidimensional IRT Models.

Project description:Student Growth Percentiles (SGPs) increasingly are being used in the United States for inferences about student achievement growth and educator effectiveness. Emerging research has indicated that SGPs estimated from observed test scores have large measurement errors. As such, little is known about "true" SGPs, which are defined in terms of nonlinear functions of latent achievement attributes for individual students and their distributions across students. We develop a novel framework using latent regression multidimensional item response theory models to study distributional properties of true SGPs. We apply these methods to several cohorts of longitudinal item response data from more than 330,000 students in a large urban metropolitan area to provide new empirical information about true SGPs. We find that true SGPs are correlated 0.3 to 0.5 across mathematics and English language arts, and that they have nontrivial relationships with individual student characteristics, particularly student race/ethnicity and absenteeism. We evaluate the potential of using these relationships to improve the accuracy of SGPs estimated from observed test scores, finding that accuracy gains even under optimal circumstances are modest. We also consider the properties of SGPs averaged to the teacher level, widely used for teacher evaluations. We find that average true SGPs for individual teachers vary substantially as a function of the characteristics of the students they teach. We discuss implications of our findings for the estimation and interpretation of SGPs at both the individual and aggregate levels.

| S-EPMC5965647 | biostudies-literature

Asymptotic Standard Errors of Parameter Scale Transformation Coefficients in Test Equating Under the Nominal Response Model.

Project description:Researchers have developed a characteristic curve procedure to estimate the parameter scale transformation coefficients in test equating under the nominal response model. In the study, the delta method was applied to derive the standard error expressions for computing the standard errors for the estimates of the parameter scale transformation coefficients. This brief report presents the results of a simulation study that examined the accuracy of the derived formulas and compared the performance of this analytical method with that of the multiple imputation method. The results indicated that the standard errors produced by the delta method were very close to the criterion standard errors as well as those yielded by the multiple imputation method under all the simulation conditions.

| S-EPMC7876637 | biostudies-literature

A Comparative Evaluation of Kernel Equating and Test Characteristic Curve Equating.

Project description:This study compares the kernel equating (KE) and test characteristic curve (TCC) equating methods using the nonequivalent anchor test equating design. In this Monte Carlo study, four independent variables were examined: sample size, test length, average form discrimination, anchor test reliability, and the percentage of anchor items. For each condition, there were 100 replications. To assess the performance of TCC equating and KE, the differences between the examinee parametric true scores and the equated estimated expected true scores were examined. The equated scores were based on the average across replications for each condition. Generally speaking, both KE and TCC equating produced accurate results, although KE tended to perform better than TCC on the parametric true score scale across conditions. Past research and the current study's results seem to indicate that KE should be strongly considered for most equating situations, particularly in light of its flexibility.

| S-EPMC5978648 | biostudies-literature

Evaluation of Candidate Items for Severe PTSD Screening for Patients With Chronic Pain: Pilot Data Analysis With the IRT Approach.

Project description:ObjectivesPost-traumatic stress disorder (PTSD) commonly co-occurs with chronic pain. Although PTSD symptoms are associated with negative health outcomes in patients with chronic pain, PTSD is typically under-detected and under-treated in outpatient pain settings. There is a need for rapid, brief screening tools to identify those at greatest risk for severe PTSD symptoms. To achieve that goal, our aim was to use item response theory (IRT) to identify the most informative PTSD symptoms characterizing severe PTSD in patients with chronic pain.MethodsFifty-six patients (71% female, 61% White) with mixed etiology chronic pain completed the PTSD Checklist-Civilian Version (PCL-C) as part of their appointment with a pain psychologist at a tertiary outpatient pain clinic. We used an IRT approach to evaluate each item's discriminant (a) and severity (b) parameters.ResultsFindings revealed that "feeling upset at reminders" (a = 3.67, b = 2.44) and "avoid thinking or talking about it" (a = 3.61, b = 2.17) as being highly discriminant for severe PTSD.ConclusionsWe identified 2 candidate items for a brief PTSD screener as they were associated with severe PTSD symptoms. These 2 items may provide clinical utility in outpatient pain treatment settings to identify those suffering from severe PTSD, enabling physicians to refer them to trauma-specific evaluation or therapy. Future research is needed to further validate and confirm these candidate PTSD items in a larger clinic sample.Lay summaryThe current study used the IRT approach to identify candidate items for a brief screener for severe PTSD. We examined 17 items of the PCL-C, and identified 2 items that were highly discriminant for severe PTSD. The 2 items were "feeling upset at reminders" and "avoid thinking or talking about it." These 2 items may provide clinical utility, since they may enable physicians to screen and make a referral for further assessment or treatment for PTSD.

| S-EPMC7083588 | biostudies-literature

OmicsDI is part of the ELIXIR infrastructure

OmicsDI is an Elixir interoperability service. Learn more ›

Tweets

OmicsDI Databases

PRIDE
PeptideAtlas
MassIVE
JPOST Repository
Physiome Model Repository

EGA
EVA
ENA
LINCS
PAXDB
Cell Collective

MetaboLights
Metabolomics Workbench
MetabolomeExpress
GNPS
BioModels
FAIRDOMHub

ArrayExpress
dbGaP
ExpressionAtlas
GEO
NODE

Information

Databases
Help
API
Contact us
Code on GitHub
Terms of use
Submit Data