Introducing a Computerized Figural Memory Test Based on Automatic Item Generation: An Analysis With the Rasch Poisson Counts Model.
Ontology highlight
ABSTRACT:
SUBMITTER: Jendryczko D
PROVIDER: S-EPMC7298330 | biostudies-literature | 2020
REPOSITORIES: biostudies-literature
Publications
Introducing a Computerized Figural Memory Test Based on Automatic Item Generation: An Analysis With the Rasch Poisson Counts Model.
Jendryczko David D Berkemeyer Laura L Holling Heinz H
Frontiers in psychology 20200610
An automatic item generator for figural memory test items called <i>figumem</i> was developed. It is available in <i>R</i>. A cognitive model allowed the generation of hypothetically parallel items within three difficulty levels determined by visual information load. In a pilot study, participants solved three items for each level of visual load. Within an item response theory approach, the Rasch Poisson counts model and modifications of it were fitted to the data. Results showed overall satisfy ...[more]
Similar Datasets
Project description:The Ruff Figural Fluency Test (RFFT) is a sensitive test for nonverbal fluency suitable for all age groups. However, assessment of performance on the RFFT is time-consuming and may be affected by interrater differences. Therefore, we developed computer software specifically designed to analyze performance on the RFFT by automated pattern recognition. The aim of this study was to compare assessment by the new software with conventional assessment by human raters. The software was developed using data from the Lifelines Cohort Study and validated in an independent cohort of the Prevention of Renal and Vascular End Stage Disease (PREVEND) study. The total study population included 1,761 persons: 54% men; mean age (SD), 58 (10) years. All RFFT protocols were assessed by the new software and two independent human raters (criterion standard). The mean number of unique designs (SD) was 81 (29) and the median number of perseverative errors (interquartile range) was 9 (4 to 16). The intraclass correlation coefficient (ICC) between the computerized and human assessment was 0.994 (95%CI, 0.988 to 0.996; p<0.001) and 0.991 (95%CI, 0.990 to 0.991; p<0.001) for the number of unique designs and perseverative errors, respectively. The mean difference (SD) between the computerized and human assessment was -1.42 (2.78) and +0.02 (1.94) points for the number of unique designs and perseverative errors, respectively. This was comparable to the agreement between two independent human assessments: ICC, 0.995 (0.994 to 0.995; p<0.001) and 0.985 (0.982 to 0.988; p<0.001), and mean difference (SD), -0.44 (2.98) and +0.56 (2.36) points for the number of unique designs and perseverative errors, respectively. We conclude that the agreement between the computerized and human assessment was very high and comparable to the agreement between two independent human assessments. Therefore, the software is an accurate tool for the assessment of performance on the RFFT.
| S-EPMC5035016 | biostudies-literature
Project description:Computer adaptive tests provide important measurement advantages over traditional fixed-item tests, but research on the psychological reactions of test takers to adaptive tests is lacking. In particular, it has been suggested that test-taker engagement, and possibly test performance as a consequence, could benefit from the control that adaptive tests have on the number of test items examinees answer correctly. However, previous research on this issue found little support for this possibility. This study expands on previous research by examining this issue in the context of a mathematical ability assessment and by considering the possible effect of immediate feedback of response correctness on test engagement, test anxiety, time on task, and test performance. Middle school students completed a mathematics assessment under one of three test type conditions (fixed, adaptive, or easier adaptive) and either with or without immediate feedback about the correctness of responses. Results showed little evidence for test type effects. The easier adaptive test resulted in higher engagement and lower anxiety than either the adaptive or fixed-item tests; however, no significant differences in performance were found across test types, although performance was significantly higher across all test types when students received immediate feedback. In addition, these effects were not related to ability level, as measured by the state assessment achievement levels. The possibility that test experiences in adaptive tests may not in practice be significantly different than in fixed-item tests is raised and discussed to explain the results of this and previous studies.
| S-EPMC5978472 | biostudies-literature
Project description:The assessment of motor proficiency is essential across childhood to identify children's strengths and difficulties and to provide adequate instruction and opportunities; assessment is a powerful tool to promote children's development. This study aimed to investigate the hierarchal order of the Test of Gross Motor Development-Third Edition (TGMD-3) items regarding difficulty levels and the differential item functioning across gender and age group (3 to 5, 6 to 8, and 9 to 10 years old). Participants are 989 children (3 to 10.9 years; girls n = 491) who were assessed using TGMD-3. For locomotor skills, appropriate results reliability (alpha = 1.0), infit (M = 0.99; SD = 0.17), outfit (M = 1.18; SD = 0.64), and point-biserial correlations (rpb values from 0.14 to 0.58) were found; the trend was similar for ball skills: reliability (alpha = 1.0), infit (M = 0.99; SD = 0.13), outfit (M = 1.08; SD = 0.52); point-biserial correlations (rpb values from 0.06 to 0.59) were obtained. Two motor criteria: gallop, item-1, and one-hand forehand strike, item-4, were the most difficult items; in contrast, run, item-2, and two-hand catch, item-2, were the easiest items. Differential item functioning for age was observed in nine locomotor and ten ball skills items. These items were easier for older children compared to younger ones. The TGMD-3 has items with different difficulty levels capable of differential functioning across age groups.
| S-EPMC9322710 | biostudies-literature
Project description:This study developed a battery of computerized working memory (WM) tests and a scoring system suitable for young adult users. The tests comprised five classic tasks derived from Baddeley's model of WM, reflecting each of the five WM functions. We recruited 115 undergraduate and graduate students from various academic fields and constructed a preliminary WM scoring norm for young adults. The scoring norm was used as a basis for developing a computerized assessment system. The results of correlation analysis show that the fluid intelligence of young adults is related to the memory function of WM, but not to the central executive system. The proposed working memory test battery for young adults comprehensively reflects the WM capacity of adults.
| S-EPMC5376327 | biostudies-literature
Project description:Automatic Item Generation (AIG) techniques are offering innovative ways to produce test items as they overcome many disadvantages involving standard item writing, such as time-consuming work and resource-intensive demands. Although this field is relatively new, it is progressing at a high speed, and several contributions have been accomplished. Nevertheless, a scarce amount of AIG software evidencing favorable psychometric properties of the generated items has been made accessible to the broad scientific community. This research had two goals: first, to present an empirical study of items produced with the aid of the Item Maker (IMak) package available online and, second, to present IMak itself for the automatic generation of figural analogies. We were particularly interested in assessing whether automatically created figural analogy rules could predict item psychometric difficulty. A total of 23 items were generated and administered to 307 participants, 49.51% from Germany. The mean age was 28.61 (SD = 10.19) and 57.65% of the participants were female. Results reveal adequate psychometric properties including convergent validity, that most of the manipulated rules contribute to item difficulty, and that rule-based difficulty prediction is possible to some extent. In other words, psychometric quality of the generated items is supported, which reveals the utility of the IMak package in assessment settings. Finally, the package is presented and its functions for figural analogy item generation are further described.
| S-EPMC6087760 | biostudies-literature
Project description:BackgroundThe 26-item Eating Attitudes Test (EAT-26) is a commonly used tool to assess eating disorder risk. The purpose of this study was to examine the psychometric properties of the EAT-26 with a combined sample: (1) of adults with overweight and obesity enrolled in a behavioral weight loss program and (2) general adult sample (n = 469; age = 36.17 ± 17.83 years; female = 72.5%; white = 66.3%; obese BMI category = 58%).MethodsRasch modeling was used to assess model-data fit, create an item-person map to evaluate relative distribution items and persons, item difficulty, and person's eating disorder (ED) risk level of the EAT-26. Differential item functioning (DIF) and rating scale functioning of the EAT-26 were also evaluated using Rasch analysis.ResultsA total of 7 misfit items were removed from the final analysis due to unacceptable Infit and Outfit mean square residual values. The item-person map showed that the items were biased toward participants with moderate to high levels of ED risk and did not cover those who had low risk for having an ED (< - 1 logits). The DIF analyses results showed that none of the items functioned differently across sex, but 5 items were flagged based on obesity status. The six-category Likert-type rating scale did not function well indicating a different response format may be needed.ConclusionSeveral concerns were identified with the psychometric evaluation of the EAT-26 that may question its utility in assessing ED risk in individuals at low risk for ED, within samples of people who have overweight and obesity seeking weight loss treatment. The 26-item Eating Attitudes Test is a self-rated measure of eating attitudes that measures symptoms and concerns of eating disorders (ED). Very little is known about how this instrument performs differently based on individual factors like body mass index (BMI) and sex (male/female). We used an advanced measurement theory (i.e., Rasch analysis) to determine if the EAT-26 is an adequate measure to detect disordered eating in men and women of different BMIs. Results indicated that the EAT-26 was biased toward participants with moderate to high levels of disordered eating risk and did not adequately detect individuals at low risk for disordered eating. The EAT-26 did not function differently based on sex (male/female). However, five questions did function differently based on obesity status (those without obesity/ those with obesity). Finally, we observed the six-category rating scale did not function appropriately and that a new response format may be warranted. In sum, there were several issues (e.g., poor rating scale and different item functioning) with the EAT-26 and future work should develop screening tools that detect low risk of disordered eating as well as function well in adults with overweight and obesity.
| S-EPMC9069796 | biostudies-literature
Project description:BackgroundThe Ruff Figural Fluency Test (RFFT; a pencil and paper test) and the CogState (a computerized cognitive test battery) are well-validated and suitable tests to evaluate cognitive functioning in large observational studies at the population level. The LifeLines Cohort Study includes the RFFT as baseline measurement and incorporated the CogState as replacement for the RFFT at follow-up. It is unknown how these two tests relate to each other. Therefore, the aim of this study is to examine the correlation between the RFFT and the CogState and the impact of demographic characteristics on this association.MethodsA subcohort of the LifeLines Cohort Study, a large population based cohort study, participated in this study. Correlations between the RFFT and six subtasks of the CogState were examined. Subgroup analyses were performed to investigate the influence of age, education, and gender on the results. With sensitivity analyses we investigated the influence of computer experience and (physical) impairments.ResultsA total of 509 participants (mean age (SD): 53 years (14.6); range 18-87 years) participated in this study. All correlations between the RFFT and the CogState were statistically significant (except for the correlation between the RFFT error ratio and the CogState One Back Task), ranging from -0.39 to 0.28. Stratifying the analyses for age, education, and gender did not substantially affect our conclusions. Sensitivity analyses showed no substantial influence of level of computer experience or (physical) impairments.ConclusionsCorrelations found in the present study were only weak to moderate, indicating that cognitive functioning measured by the RFFT does not measure the same components of cognitive functioning as six subtasks of the CogState. Computerized testing such as the CogState may be very well suited for large cohort studies to assess cognitive functioning in the general population and to identify cognitive changes as early as possible, as it is a less time- and labor intensive tool.
| S-EPMC5427615 | biostudies-literature
Project description:PurposeExamining item usage is an important step in evaluating the performance of a computerized adaptive test (CAT). We study item usage for a newly developed multidimensional CAT which draws items from three PROMIS domains, as well as a disease-specific one.MethodsThe multidimensional item bank used in the current study contained 194 items from four domains: the PROMIS domains fatigue, physical function, and ability to participate in social roles and activities, and a disease-specific domain (the COPD-SIB). The item bank was calibrated using the multidimensional graded response model and data of 795 patients with chronic obstructive pulmonary disease. To evaluate the item usage rates of all individual items in our item bank, CAT simulations were performed on responses generated based on a multivariate uniform distribution. The outcome variables included active bank size and item overuse (usage rate larger than the expected item usage rate).ResultsFor average θ-values, the overall active bank size was 9-10%; this number quickly increased as θ-values became more extreme. For values of -2 and +2, the overall active bank size equaled 39-40%. There was 78% overlap between overused items and active bank size for average θ-values. For more extreme θ-values, the overused items made up a much smaller part of the active bank size: here the overlap was only 35%.ConclusionsOur results strengthen the claim that relatively short item banks may suffice when using polytomous items (and no content constraints/exposure control mechanisms), especially when using MCAT.
| S-EPMC5655597 | biostudies-literature
Project description:Most current models of recognition memory fail to separately model item and person heterogeneity which makes it difficult to assess ability at the latent construct level and prevents the administration of adaptive tests. Here we propose to employ a General Condorcet Model for Recognition (GCMR) in order to estimate ability, response bias and item difficulty in dichotomous recognition memory tasks. Using a Bayesian modeling framework and MCMC inference, we perform 3 separate validation studies comparing GCMR to the Rasch model from IRT and the 2-High-Threshold (2HT) recognition model. First, two simulations demonstrate that recovery of GCMR ability estimates with varying sparsity and test difficulty is more robust and that estimates improve from the two other models under common test scenarios. Then, using a real dataset, face validity is confirmed by replicating previous findings of general and domain-specific age effects (Güsten et al. in Cortex 137:138-148, https://doi.org/10.1016/j.cortex.2020.12.017 , 2021). Using cross-validation we show better out-of-sample prediction for the GCMR as compared to Rasch and 2HT model. In addition, we present a hierarchical extension of the model that is able to estimate age- and domain-specific effects directly, without recurring to a two-stage procedure. Finally, an adaptive test using the GCMR is simulated, showing that the test length necessary to obtain reliable ability estimates can be significantly reduced compared to a non-adaptive procedure. The GCMR allows to model trial-by-trial performance and to increase the efficiency and reliability of recognition memory assessments.
| S-EPMC8786965 | biostudies-literature
Project description:Item calibration is a technique to estimate characteristics of questions (called items) for achievement tests. In computerized tests, item calibration is an important tool for maintaining, updating and developing new items for an item bank. To efficiently sample examinees with specific ability levels for this calibration, we use optimal design theory assuming that the probability to answer correctly follows an item response model. Locally optimal unrestricted designs have usually a few design points for ability. In practice, it is hard to sample examinees from a population with these specific ability levels due to unavailability or limited availability of examinees. To counter this problem, we use the concept of optimal restricted designs and show that this concept naturally fits to item calibration. We prove an equivalence theorem needed to verify optimality of a design. Locally optimal restricted designs provide intervals of ability levels for optimal calibration of an item. When assuming a two-parameter logistic model, several scenarios with D-optimal restricted designs are presented for calibration of a single item and simultaneous calibration of several items. These scenarios show that the naive way to sample examinees around unrestricted design points is not optimal.
| S-EPMC6820328 | biostudies-literature