An Item Response Model for True-False Exams Based on Signal Detection Theory.
Ontology highlight
ABSTRACT:
SUBMITTER: DeCarlo LT
PROVIDER: S-EPMC7174808 | biostudies-literature |
REPOSITORIES: biostudies-literature
Similar Datasets
Project description:INTRODUCTION:Models from signal detection theory are commonly used to score neuropsychological test data, especially tests of recognition memory. Here we show that certain item response theory models can be formulated as signal detection theory models, thus linking two complementary but distinct methodologies. We then use the approach to evaluate the validity (construct representation) of commonly used research measures, demonstrate the impact of conditional error on neuropsychological outcomes, and evaluate measurement bias. METHOD:Signal detection-item response theory (SD-IRT) models were fitted to recognition memory data for words, faces, and objects. The sample consisted of U.S. Infantry Marines and Navy Corpsmen participating in the Marine Resiliency Study. Data comprised item responses to the Penn Face Memory Test (PFMT; N = 1,338), Penn Word Memory Test (PWMT; N = 1,331), and Visual Object Learning Test (VOLT; N = 1,249), and self-report of past head injury with loss of consciousness. RESULTS:SD-IRT models adequately fitted recognition memory item data across all modalities. Error varied systematically with ability estimates, and distributions of residuals from the regression of memory discrimination onto self-report of past head injury were positively skewed towards regions of larger measurement error. Analyses of differential item functioning revealed little evidence of systematic bias by level of education. CONCLUSIONS:SD-IRT models benefit from the measurement rigor of item response theory-which permits the modeling of item difficulty and examinee ability-and from signal detection theory-which provides an interpretive framework encompassing the experimentally validated constructs of memory discrimination and response bias. We used this approach to validate the construct representation of commonly used research measures and to demonstrate how nonoptimized item parameters can lead to erroneous conclusions when interpreting neuropsychological test data. Future work might include the development of computerized adaptive tests and integration with mixture and random-effects models.
| S-EPMC6050112 | biostudies-literature
Project description:BackgroundComputer and paper examinations in our days are constructed from an item pool which is regularly updated. Given the way that exams are created, one of the major concerns is the security of the items that are being used in order to ensure a good estimation of abilities. The aim of this study is to measure the prevalence of item pre-knowledge in our medical school.MethodsThe Deterministic, Gated Item Response Theory Model (DGM) was applied to estimate the prevalence of students who have had item pre-knowledge from six multiple choice examinations of the Clinical Anatomy course at the Faculty of Medicine of University of Porto. Each examination consisted of 100 items with an average of 200 students and 20% repeated items per examination. The estimation of the sensitivity and specificity was based on a simulation study. The sensitivity and specificity estimates, and apparent prevalence were used to estimate true prevalence of cheating students in the examinations under study.ResultsThe specificity in the DGM for different simulation scenarios was between 68 and 98%, while the sensitivity ranged from 60 to 91%. The apparent prevalence was between 0.0 and 3.4%, while the true prevalence ranged from 1.2 to 3.7%.ConclusionsThe true prevalence was much lower compared to the students self-reported copying of responses from other students; however, it is important to keep monitoring the pre-knowledge prevalence in order to enforce measures in case an increase occurs.
| S-EPMC6659220 | biostudies-literature
Project description:Applying item response theory (IRT) true score equating to multidimensional IRT models is not straightforward due to the one-to-many relationship between a true score and latent variables. Under the common-item nonequivalent groups design, the purpose of the current study was to introduce two IRT true score equating procedures that adopted different dimension reduction strategies for the bifactor model. The first procedure, which was referred to as the integration procedure, linked the latent variable scales for the bifactor model and integrated out the specific factors from the item response function of the bifactor model. Then, IRT true score equating was applied to the marginalized bifactor model. The second procedure, which was referred to as the PIRT-based procedure, projected the specific dimensions onto the general dimension to obtain a locally dependent unidimensional IRT (UIRT) model and linked the scales of the UIRT model, followed by the application of IRT true score equating to the locally dependent UIRT model. Equating results obtained with the two equating procedures along with those obtained with the unidimensional three-parameter logistic (3PL) model were compared using both simulated and real data. In general, the integration and PIRT-based procedures provided equating results that were not practically different. Furthermore, the equating results produced by the two bifactor-based procedures became more accurate than the results returned by the 3PL model as tests became more multidimensional.
| S-EPMC9382090 | biostudies-literature
Project description:Advancements in item response theory (IRT) have led to models for dual dependence, which control for cluster and method effects during a psychometric analysis. Currently, however, this class of models does not include one that controls for when the method effects stem from two method sources in which one source functions differently across the aspects of another source (i.e., a nested method-source interaction). For this study, then, a Bayesian IRT model is proposed, one that accounts for such interaction among method sources while controlling for the clustering of individuals within the sample. The proposed model accomplishes these tasks by specifying a multilevel trifactor structure for the latent trait space. Details of simulations are also reported. These simulations demonstrate that this model can identify when item response data represent a multilevel trifactor structure, and it does so in data from samples as small as 250 cases nested within 50 clusters. Additionally, the simulations show that misleading estimates for the item discriminations could arise when the trifactor structure reflected in the data is not correctly accounted for. The utility of the model is also illustrated through the analysis of empirical data.
| S-EPMC6506989 | biostudies-literature
Project description:BackgroundA new patient-reported health measurement model has been developed to quantify descriptions of health states. Known as the multi-attribute preference response (MAPR) model, it is based on item response theory. The response task in the MAPR is for a patient to judge whether hypothetical health-state descriptions are better or worse than his/her own health status.MethodsIn its most simple form MAPR is a Rasch model where for each respondent on the same unidimensional health scale values are estimated of their own health status and values of the hypothetical comparator health states. These values reflect the quality or severity of the health states. Alternatively, the respondents are offered health-state descriptions that are based on a classification system (e.g., multi-attribute) with a fixed number of health attributes, each with a limited number of levels. In the latter variant, the weights of the levels of the attributes in the descriptive system, which represents the range of the health states, are estimated. The results of a small empirical study are presented to illustrate the procedures of the MAPR model and possible extensions of the model are discussed.ResultsThe small study that we conducted to illustrate the procedure and results of our proposed method to measure the quality of health states and patients' own health status showed confirming results.ConclusionsThis paper introduces the typical MAPR model and shows how it extends the basic Rasch model with a regression function for the attributes of the health-state classification system.
| S-EPMC6013962 | biostudies-literature
Project description:Memory is reconstructive and error-prone, which make memory illusions very common in everyday life. However, studying memory illusions can provide valuable insights into how memory works. Pupil response has emerged, in recent years, as an indicator of memory encoding and retrieval, however its validity as a measure of memory success is debated. In this study, we explored whether pupil response patterns can differentiate true from false memories and whether variations in the temporal dynamics of pupil response can elucidate the mechanisms underlying false memory creation. The Deese-Roediger-McDermott (DRM) paradigm was employed to generate false memories in two separate experiments involving visual and auditory stimuli. Pupil responses effectively differentiated true from false memories based on variations in pupil amplitude at different temporal components. This discrimination remained consistent across both experiments, with slightly stronger effects in the auditory condition, aligning with the more pronounced false memory effects in this condition. Notably, differential pupil responses between true and false memories varied based on the type of memory involved at recognition. These findings provide valuable insights into the cognitive processes underlying memory distortions, with implications for theoretical frameworks and real-world contexts.
| S-EPMC10567773 | biostudies-literature
Project description:A mixture extension of Samejima's continuous response model for continuous measurement outcomes and its estimation through a heuristic approach based on limited-information factor analysis is introduced. Using an empirical data set, it is shown that two groups of respondents that differ both qualitatively and quantitatively in their response behavior can be revealed. In addition to the real data application, the effectiveness of the heuristic estimation approach under real data analytic conditions was examined through a Monte Carlo simulation study. The results showed that the heuristic estimation approach provided reliable parameter estimates and the model successfully converged above 80% when the sample size was 250 and above 90% when the sample size was 500 or 1,000 for most conditions.
| S-EPMC7047264 | biostudies-literature
Project description:Using the bifactor item response theory model to analyze data arising from educational and psychological studies has gained popularity over the years. Unfortunately, using this model in practice comes with challenges. One such challenge is an empirical identification issue that is seldom discussed in the literature, and its impact on the estimates of the bifactor model's parameters has not been demonstrated. This issue occurs when an item's discriminations on the general and specific dimensions are approximately equal (i.e., the within-item discriminations are similar in strength), leading to difficulties in obtaining unique estimates for those discriminations. We conducted three simulation studies to demonstrate that within-item discriminations being similar in strength creates problems in estimation stability. The results suggest that a large sample could alleviate but not resolve the problems, at least when considering sample sizes up to 4,000. When the discriminations within items were made clearly different, the estimates of these discriminations were more consistent across the data replicates than that observed when the discriminations within the items were similar. The results also show that the similarity of an item's discriminatory magnitudes on different dimensions has direct implications on the sample size needed in order to consistently obtain accurate parameter estimates. Although our goal was to provide evidence of the empirical identification issue, the study further reveals that the extent of similarity of within-item discriminations, the magnitude of discriminations, and how well the items are targeted to the respondents also play factors in the estimation of the bifactor model's parameters.
| S-EPMC9574084 | biostudies-literature
Project description:Three widely used interpersonal trust measurement scales [Interpersonal Trust Scale (ITS), Philosophies of Human Nature Scale (RPHNS), Company Trust Scale (CTS)] have seldom been applied in non-Western contexts. Different social environments may lead to variation in the level or structure of trust. Therefore, it is necessary to compare the applicability of these scales to different levels of trust-related traits in Eastern cultures so that researchers can choose appropriate scales for relevant studies. This study attempted to conduct a comparative analysis of the ITS, RPHNS, and CTS. A sample of 725 Chinese college students was analyzed. Total score correlations and latent factor correlations estimated by confirmatory factor analysis (CFA) for a first-order three-factor model were assessed, and then the quality of the item parameters, test reliability and standard errors, and test information were assessed. The results are as follows: (1) the ITS and the RPHNS assessed almost the same trust traits; therefore, only the ITS and the RPHNS are compared in the next sections; (2) the original structure of only the RPHNS is verified; (3) some items on the ITS do not work well, while the RPHNS has higher overall test reliability; and (4) the average item information provided by the RPHNS is higher across all trait levels. In most cases, the RPHNS is the better choice in the Chinese cultural context.
| S-EPMC6989430 | biostudies-literature
Project description:BackgroundThe Neuropsychiatric Inventory (NPI) is a widely used scale for the assessment of the behavioral and psychological symptoms of dementia (BPSD). We previously developed a novel dementia scale, the ABC dementia scale, in the TRIAD1412 trial and we compared the BPSD domain scores with the NPI scores. We, therefore, considered that we should investigate the quality of the NPI items using statistical approaches.ObjectiveWe investigated the statistical characteristics of the 12 questions or items in the Japanese version of the NPI using the item response theory. This theory is the standard approach for the development of a new assessment scale and we used it to evaluate the quality of the items in the NPI.MethodsFirst, we performed factor analysis with Promax rotation to identify latent constructs in the data from 312 patients obtained in TRIAD1412. Second, following the result of the factor analysis, we divided the 12 items into domains and then investigated the characteristics of the sub-syndromes in each domain using item response category characteristic curves.ResultsWe found three latent constructs or domains: "hyperactivity," "psychosis and apathy," and "affect" (Cronbach's α= 0.68) in the 12 items. Further, the items on euphoria, apathy, and appetite and eating abnormalities did not provide sufficient information to estimate BPSD severity.ConclusionThe NPI item characteristics indicate that while the scale can distinguish whether patients have severe BPSD or not, it cannot estimate the degree of severity in a suspected case with a mild or unknown level of BPSD.
| S-EPMC7306920 | biostudies-literature