Evaluations of the sum-score-based and item response theory-based tests of group mean differences under various simulation conditions.
Ontology highlight
ABSTRACT:
SUBMITTER: Wang M
PROVIDER: S-EPMC8649417 | biostudies-literature |
REPOSITORIES: biostudies-literature
Similar Datasets
Project description:In dichotomous item response theory (IRT) framework, the asymptotic standard error (ASE) is the most common statistic to evaluate the precision of various ability estimators. Easy-to-use ASE formulas are readily available; however, the accuracy of some of these formulas was recently questioned and new ASE formulas were derived from a general asymptotic theory framework. Furthermore, exact standard errors were suggested to better evaluate the precision of ability estimators, especially with short tests for which the asymptotic framework is invalid. Unfortunately, the accuracy of exact standard errors was assessed so far only in a very limiting setting. The purpose of this article is to perform a global comparison of exact versus (classical and new formulations of) asymptotic standard errors, for a wide range of usual IRT ability estimators, IRT models, and with short tests. Results indicate that exact standard errors globally outperform the ASE versions in terms of reduced bias and root mean square error, while the new ASE formulas are also globally less biased than their classical counterparts. Further discussion about the usefulness and practical computation of exact standard errors are outlined.
| S-EPMC7221492 | biostudies-literature
Project description:How to detect faking on personality measures has been investigated using various methods and procedures. As previous findings are mixed and rarely based on ideal point item response theory models, additional research is needed for further exploration. This study modeled the responses of personality tests using ideal point method across instructed faking and honest responding conditions. A sample of undergraduate students participated the within-subjects measures to examine how the item location parameter derived from the generalized graded unfolding model changed, and how individuals' perception about items changed when faked. The mean test scores of faking group was positively correlated to the magnitude of within-subjects score change. The item-level analysis revealed both conscientiousness items (18.8%) and neuroticism items (50.0%) appeared significant shifts on item parameters, suggesting that response pattern changed from honest to faking conditions. The direction of the change appeared both in positive and negative way, demonstrating that faking could increase or decrease personality factor scores. The results indicated that the changes of perceptions on items could be operated by faking, offering some support for the ideal point model to be an adequate measure for detecting faking. However, the findings of diagnostic accuracy analysis also implied that the appropriateness of ideal point models for detecting faking should be under consideration, also be used with caution. Implications, further research directions, and limitations are discussed.
| S-EPMC6987465 | biostudies-literature
Project description:Applying item response theory (IRT) true score equating to multidimensional IRT models is not straightforward due to the one-to-many relationship between a true score and latent variables. Under the common-item nonequivalent groups design, the purpose of the current study was to introduce two IRT true score equating procedures that adopted different dimension reduction strategies for the bifactor model. The first procedure, which was referred to as the integration procedure, linked the latent variable scales for the bifactor model and integrated out the specific factors from the item response function of the bifactor model. Then, IRT true score equating was applied to the marginalized bifactor model. The second procedure, which was referred to as the PIRT-based procedure, projected the specific dimensions onto the general dimension to obtain a locally dependent unidimensional IRT (UIRT) model and linked the scales of the UIRT model, followed by the application of IRT true score equating to the locally dependent UIRT model. Equating results obtained with the two equating procedures along with those obtained with the unidimensional three-parameter logistic (3PL) model were compared using both simulated and real data. In general, the integration and PIRT-based procedures provided equating results that were not practically different. Furthermore, the equating results produced by the two bifactor-based procedures became more accurate than the results returned by the 3PL model as tests became more multidimensional.
| S-EPMC9382090 | biostudies-literature
Project description:BackgroundHigh-fidelity simulation (HFS) has become a widely used and established pedagogy for teaching clinical nursing skills. Nevertheless, there are few evidence-based instruments that validate the effectiveness of simulation learning in mainland China.MethodsThe Simulation Learning Effectiveness Inventory (SLEI) was adapted and validated for use in this study. Psychometric evaluation, incorporating classical test theory and item response theory (IRT) methods, was performed with 533 third-year undergraduate nursing students who were recruited from May 2017 to July 2018.ResultsThe findings of exploratory and confirmatory factor analyses revealed that the simplified Chinese version of the SLEI (SLEI-SC) was composed of six factors, namely, course arrangement, equipment resource, debriefing, clinical ability, problem solving, and confidence, which explained 60.84% of the total variance. The Cronbach's α, MIIC, marginal reliability, and test-retest reliability values obtained for the total scale were 0.88, 0.38, 0.96, and 0.88, respectively. Furthermore, the difference between the total scores for learning effectiveness pre- and post-course was statistically significant (t = 2.59, p < 0.05, Cohen's d = 0.60). IRT analysis showed that the SLEI-SC discriminates well between students with high and low levels of learning effectiveness and offers information about a broad range of learning effectiveness measures. The relationship between final course grade and total score on the SLEI-SC was statistically significant (r = 0.63, p < 0.05).ConclusionWe demonstrated initial psychometric evidence and support for the 31-item SLEI-SC as a developmentally appropriate instrument for assessing the learning effectiveness of all phases of HFS use with nursing students.
| S-EPMC7042430 | biostudies-literature
Project description:Three widely used interpersonal trust measurement scales [Interpersonal Trust Scale (ITS), Philosophies of Human Nature Scale (RPHNS), Company Trust Scale (CTS)] have seldom been applied in non-Western contexts. Different social environments may lead to variation in the level or structure of trust. Therefore, it is necessary to compare the applicability of these scales to different levels of trust-related traits in Eastern cultures so that researchers can choose appropriate scales for relevant studies. This study attempted to conduct a comparative analysis of the ITS, RPHNS, and CTS. A sample of 725 Chinese college students was analyzed. Total score correlations and latent factor correlations estimated by confirmatory factor analysis (CFA) for a first-order three-factor model were assessed, and then the quality of the item parameters, test reliability and standard errors, and test information were assessed. The results are as follows: (1) the ITS and the RPHNS assessed almost the same trust traits; therefore, only the ITS and the RPHNS are compared in the next sections; (2) the original structure of only the RPHNS is verified; (3) some items on the ITS do not work well, while the RPHNS has higher overall test reliability; and (4) the average item information provided by the RPHNS is higher across all trait levels. In most cases, the RPHNS is the better choice in the Chinese cultural context.
| S-EPMC6989430 | biostudies-literature
Project description:BackgroundThe Neuropsychiatric Inventory (NPI) is a widely used scale for the assessment of the behavioral and psychological symptoms of dementia (BPSD). We previously developed a novel dementia scale, the ABC dementia scale, in the TRIAD1412 trial and we compared the BPSD domain scores with the NPI scores. We, therefore, considered that we should investigate the quality of the NPI items using statistical approaches.ObjectiveWe investigated the statistical characteristics of the 12 questions or items in the Japanese version of the NPI using the item response theory. This theory is the standard approach for the development of a new assessment scale and we used it to evaluate the quality of the items in the NPI.MethodsFirst, we performed factor analysis with Promax rotation to identify latent constructs in the data from 312 patients obtained in TRIAD1412. Second, following the result of the factor analysis, we divided the 12 items into domains and then investigated the characteristics of the sub-syndromes in each domain using item response category characteristic curves.ResultsWe found three latent constructs or domains: "hyperactivity," "psychosis and apathy," and "affect" (Cronbach's α= 0.68) in the 12 items. Further, the items on euphoria, apathy, and appetite and eating abnormalities did not provide sufficient information to estimate BPSD severity.ConclusionThe NPI item characteristics indicate that while the scale can distinguish whether patients have severe BPSD or not, it cannot estimate the degree of severity in a suspected case with a mild or unknown level of BPSD.
| S-EPMC7306920 | biostudies-literature
Project description:This article introduces three new variants of entropy to detect person misfit (Ei, EMi , and EMRi ), and provides preliminary evidence that these measures are worthy of further investigation. Previously, entropy has been used as a measure of approximate data-model fit to quantify how well individuals are classified into latent classes, and to quantify the quality of classification and separation between groups in logistic regression models. In the current study, entropy is explored through conceptual examples and Monte Carlo simulation comparing entropy with established measures of person fit in item response theory (IRT) such as lz, lz*, U, and W. Simulation results indicated that EMi and EMRi were successfully able to detect aberrant response patterns when comparing contaminated and uncontaminated subgroups of persons. In addition, EMi and EMRi performed similarly in showing separation between the contaminated and uncontaminated subgroups. However, EMRi may be advantageous over other measures when subtests include a small number of items. EMi and EMRi are recommended for use as approximate person-fit measures for IRT models. These measures of approximate person fit may be useful in making relative judgments about potential persons whose response patterns do not fit the theoretical model.
| S-EPMC5978476 | biostudies-literature
Project description:Differential item functioning (DIF) occurs when people with the same proficiency have different probabilities of giving a certain response to an item. The present study focused on an assumption implicit in popular methods for DIF testing that has received little attention in published literature (item residual homogeneity). The assumption is explained, a strategy for detecting violations of it (i.e., item residual heterogeneity) is illustrated with empirical data, and simulations are carried out to evaluate the performance of binary logistic regression, two-group item response theory (IRT), and the Mantel-Haenszel (MH) test in the presence of item residual heterogeneity. Results indicated that heterogeneity inflated Type I error and attenuated power for logistic regression, and attenuated power and produced biased estimates of the latent focal group mean and standard deviation for two-group IRT. The MH test was robust to item residual heterogeneity, probably because it does not use the logistic function.
| S-EPMC5978544 | biostudies-literature
Project description:The Connectedness to Nature Scale (CNS) is used as a measure of the subjective cognitive connection between individuals and nature. However, to date, it has not been analyzed at the item level to confirm its quality. In the present study, we conduct such an analysis based on Item Response Theory. We employed data from previous studies using the Spanish-language version of the CNS, analyzing a sample of 1008 participants. The results show that seven items presented appropriate indices of discrimination and difficulty, in addition to a good fit. The remaining six have inadequate discrimination indices and do not present a good fit. A second study with 321 participants shows that the seven-item scale has adequate levels of reliability and validity. Therefore, it would be appropriate to use a reduced version of the scale after eliminating the items that display inappropriate behavior, since they may interfere with research results on connectedness to nature.
| S-EPMC5543030 | biostudies-other
Project description:PurposeIn a sequential objective structured clinical examination (OSCE), all students initially take a short screening OSCE. Examinees who pass are excused from further testing, but an additional OSCE is administered to the remaining examinees. Previous investigations of sequential OSCE were based on classical test theory. We aimed to design and evaluate screening OSCEs based on item response theory (IRT).MethodsWe carried out a retrospective observational study. At each station of a 10-station OSCE, the students' performance was graded on a Likert-type scale. Since the data were polytomous, the difficulty parameters, discrimination parameters, and students' ability were calculated using a graded response model. To design several screening OSCEs, we identified the 5 most difficult stations and the 5 most discriminative ones. For each test, 5, 4, or 3 stations were selected. Normal and stringent cut-scores were defined for each test. We compared the results of each of the 12 screening OSCEs to the main OSCE and calculated the positive and negative predictive values (PPV and NPV), as well as the exam cost.ResultsA total of 253 students (95.1%) passed the main OSCE, while 72.6% to 94.4% of examinees passed the screening tests. The PPV values ranged from 0.98 to 1.00, and the NPV values ranged from 0.18 to 0.59. Two tests effectively predicted the results of the main exam, resulting in financial savings of 34% to 40%.ConclusionIf stations with the highest IRT-based discrimination values and stringent cut-scores are utilized in the screening test, sequential OSCE can be an efficient and convenient way to conduct an OSCE.
| S-EPMC5676019 | biostudies-literature