Browse
Submit Data
Databases
API
Help

Dataset Information

25 Views

0 Connections

0 Citations

0 Reanalyses

0 Downloads

Omics score: 0

Measurement Efficiency for Fixed-Precision Multidimensional Computerized Adaptive Tests: Comparing Health Measurement and Educational Testing Using Example Banks.

ABSTRACT: It is currently not entirely clear to what degree the research on multidimensional computerized adaptive testing (CAT) conducted in the field of educational testing can be generalized to fields such as health assessment, where CAT design factors differ considerably from those typically used in educational testing. In this study, the impact of a number of important design factors on CAT performance is systematically evaluated, using realistic example item banks for two main scenarios: health assessment (polytomous items, small to medium item bank sizes, high discrimination parameters) and educational testing (dichotomous items, large item banks, small- to medium-sized discrimination parameters). Measurement efficiency is evaluated for both between-item multidimensional CATs and separate unidimensional CATs for each latent dimension. In this study, we focus on fixed-precision (variable-length) CATs because it is both feasible and desirable in health settings, but so far most research regarding CAT has focused on fixed-length testing. This study shows that the benefits associated with fixed-precision multidimensional CAT hold under a wide variety of circumstances.

SUBMITTER: Paap MCS

PROVIDER: S-EPMC6295884 | biostudies-literature |

REPOSITORIES: biostudies-literature

ACCESS DATA

Json Xml

Similar Datasets

Making Fixed-Precision Between-Item Multidimensional Computerized Adaptive Tests Even Shorter by Reducing the Asymmetry Between Selection and Stopping Rules.

Project description:Fixed-precision between-item multidimensional computerized adaptive tests (MCATs) are becoming increasingly popular. The current generation of item-selection rules used in these types of MCATs typically optimize a single-valued objective criterion for multivariate precision (e.g., Fisher information volume). In contrast, when all dimensions are of interest, the stopping rule is typically defined in terms of a required fixed marginal precision per dimension. This asymmetry between multivariate precision for selection and marginal precision for stopping, which is not present in unidimensional computerized adaptive tests, has received little attention thus far. In this article, we will discuss this selection-stopping asymmetry and its consequences, and introduce and evaluate three alternative item-selection approaches. These alternatives are computationally inexpensive, easy to communicate and implement, and result in effective fixed-marginal-precision MCATs that are shorter in test length than with the current generation of item-selection approaches.

| S-EPMC7495795 | biostudies-literature

Comparison of Patient-Reported Outcomes Measurement Information System Computerized Adaptive Testing Versus Fixed Short Forms in Juvenile Myositis.

Project description:ObjectivePatient-Reported Outcomes Measurement Information System (PROMIS) measures can be administered via computerized adaptive testing (CAT) or fixed short forms (FSFs), but the empirical benefits of CAT versus FSFs are unknown in juvenile myositis (JM). The present study was undertaken to assess whether PROMIS CAT is feasible, precise, correlated with FSFs, and less prone to respondent burden and floor/ceiling effects than FSFs in JM.MethodsPatients 8-17 years of age (self-report and parent proxy) and parents of patients 5-7 years of age (only parent proxy) completed PROMIS fatigue, pain interference, upper extremity function, mobility, anxiety, and depressive symptoms measures. Pearson correlations, paired t-tests, and Cohen's d were calculated between PROMIS CAT and FSFs. McNemar's test assessed floor/ceiling effects between CAT and FSFs. Precision and respondent burden were examined across the T score range.ResultsData from 67 patient-parent dyads were analyzed. CAT and FSF mean scores did not significantly differ except in parent proxy anxiety and fatigue (effect size 0.23 and 0.19, respectively). CAT had less pronounced floor/ceiling effects at the less symptomatic extreme in all domains except self-report anxiety. Increased item burden and higher SEs were seen in less symptomatic scorers for CAT. Modified stopping rules limiting CAT item administration did not decrease precision.ConclusionPROMIS CAT appears to be feasible and correlated with FSFs. CAT had less pronounced floor/ceiling effects, allowing detection of individual differences in less symptomatic patients. Modified stopping rules for CAT may decrease respondent burden. CAT can be considered for long-term follow-up of JM patients.

| S-EPMC8800940 | biostudies-literature

Efficiency, Precision, Validity, and Reliability of GlauCAT-Asian Computerized Adaptive Tests in Measuring Glaucoma-Related Quality of Life.

Project description:PurposeTo determine the efficiency, precision, and agreement of GlauCAT-Asian and its corresponding validity and reliability.MethodsIn this cross-sectional study, 219 participants (mean ± standard deviation age, 66.59 ± 8.61 years; 34% female) across the spectrum of glaucoma severity and 50 glaucoma suspects were recruited from glaucoma clinics in Singapore. Participants answered seven computerized adaptive testing (CAT) evaluations (Ocular Comfort, Activity Limitation, Lighting, Mobility, Concerns, Psychosocial, Glaucoma Management) and underwent eye examinations. Efficiency (mean number of items required for each CAT and time taken for CAT versus full item banks [IBs]), agreement (concordance between CATs and full IB person measures, henceforth referred to as scores), and precision (standard error of measurement [SE]) were evaluated. Other validity and reliability metrics were also assessed.ResultsThe mean number of items administered ranged from 9 (Mobility/Glaucoma Management) to 12 (Ocular Comfort). Compared to answering the full IBs, CATs provided an average time saving of 38.3% (range, 10% to 70.6% for Lighting and Activity Limitation, respectively). Agreement between scores obtained by CAT versus full IB was high (intracorrelation coefficient ≥0.75), as was precision of score estimates (mean SE range: 0.35 for Psychosocial to 0.29 for Mobility). Scores from Activity Limitation, Mobility, Lighting, and Concerns decreased significantly as glaucoma severity increased (criterion validity; P-trend <0.05). All tests displayed good convergent/divergent validity and test-retest reliability.ConclusionsGlauCAT-Asian provides efficient, precise, accurate, valid, and reliable measurement of the patient-centered impact of glaucoma.Translational relevanceGlauCAT-Asian may provide a valuable clinical tool for ophthalmologists to monitor impact of disease progression and the effectiveness of therapies.

| S-EPMC10860685 | biostudies-literature

Validity and reliability of patient reported outcomes measurement information system computerized adaptive tests in systemic lupus erythematous.

Project description:BackgroundThe evaluation of Patient Reported Outcomes Measurement Information System (PROMIS) computerized adaptive test (CAT) in adults with systemic lupus erythematous (SLE) is an emerging field of research. We aimed to examine the test-retest reliability and construct validity of the PROMIS CAT in a Canadian cohort of patients with SLE.MethodsTwo hundred twenty-seven patients completed 14 domains of PROMIS CAT and seven legacy instruments during their clinical visits. Test-retest reliability of PROMIS was evaluated 7-10 days from baseline using intraclass correlation coefficient (ICC (2; 1)). The construct validity of the PROMIS CAT domains was evaluated against the commonly used legacy instruments, and also in comparison to disease activity and disease damage using Spearman correlations. A multitrait-multimethod matrix (MMM) approach was used to further assess construct validity comparing selected 10 domains of PROMIS and SF-36 domains.ResultsModerate to excellent reliability was found for all domains (ICC [2;1] ranging from lowest, 0.66 for Sleep Disturbance and highest, 0.93 for the Mobility domain). Comparing seven legacy instruments with 14 domains of PROMIS CAT, moderate to strong correlations (0.51-0.91) were identified. The average time to complete all PROMIS CAT domains was 11.7 min. The MMM further established construct validity by showing moderate to strong correlations (0.55-0.87) between select PROMIS and SF-36 domains; the average correlations from similar traits (convergent validity) were significantly greater than the average correlations from different traits.ConclusionsThese results provide evidence on the reliability and validity of PROMIS CAT in SLE in a Canadian cohort.

| S-EPMC8649426 | biostudies-literature

Optimal Item Calibration for Computerized Achievement Tests.

Project description:Item calibration is a technique to estimate characteristics of questions (called items) for achievement tests. In computerized tests, item calibration is an important tool for maintaining, updating and developing new items for an item bank. To efficiently sample examinees with specific ability levels for this calibration, we use optimal design theory assuming that the probability to answer correctly follows an item response model. Locally optimal unrestricted designs have usually a few design points for ability. In practice, it is hard to sample examinees from a population with these specific ability levels due to unavailability or limited availability of examinees. To counter this problem, we use the concept of optimal restricted designs and show that this concept naturally fits to item calibration. We prove an equivalence theorem needed to verify optimality of a design. Locally optimal restricted designs provide intervals of ability levels for optimal calibration of an item. When assuming a two-parameter logistic model, several scenarios with D-optimal restricted designs are presented for calibration of a single item and simultaneous calibration of several items. These scenarios show that the naive way to sample examinees around unrestricted design points is not optimal.

| S-EPMC6820328 | biostudies-literature

The Physicians' Competence in Substance Abuse Test (P-CSAT): a multidimensional educational measurement tool for substance abuse training programs.

Project description:BackgroundEfforts to enhance physician substance abuse (SA) management skills lack tools to assess skills and training effectiveness. We sought to develop an easily administered survey to assess SA fact-based skills and clinical decision-making.MethodsWe prepared 60 fact-based items dealing with SA knowledge, attitudes, and behaviors (KAB) and 53 script concordance test (SCT) items assessing SA decision-making. We used expert review and standard psychometric criteria to eliminate discordant or non-contributory survey items. We tested 92 draft items in 117 physicians, including 13 with additional SA training (trained), and 17 recognized SA experts. We assessed final survey internal consistency with Cronbach's alpha and differences in scores between experts, trained physicians, and physicians without SA training (novices) with the Kruskal-Wallis test.ResultsFollowing refinement, the draft survey was reduced to 30 KAB and 33 SCT items. Alpha was 0.901 for the final 63-item survey and 0.887 and 0.797 for the KAB and SCT subscales, respectively. Novices, trained physicians, and experts scored means of 196, 213, and 261 respectively out of 315 possible points on the final survey. The KAB and SCT subscale results showed similar patterns. Score differences for the overall survey and its subscales were highly significant (p<0.001).ConclusionsThis survey, which we have named the Physicians' Competence in Substance Abuse Test (P-CSAT) and placed in the public domain, meets baseline criteria for reliability and validity. Future studies should determine the extent to which the P-CSAT provides consistent results in other practitioner populations and responds to SA educational efforts.

| S-EPMC3288760 | biostudies-literature

Multidimensional Computerized Adaptive Testing Using Non-Compensatory Item Response Theory Models.

Project description:Current use of multidimensional computerized adaptive testing (MCAT) has been developed in conjunction with compensatory multidimensional item response theory (MIRT) models rather than with non-compensatory ones. In recognition of the usefulness of MCAT and the complications associated with non-compensatory data, this study aimed to develop MCAT algorithms using non-compensatory MIRT models and to evaluate their performance. For the purpose of the study, three item selection methods were adapted and compared, namely, the Fisher information method, the mutual information method, and the Kullback-Leibler information method. The results of a series of simulations showed that the Fisher information and mutual information methods performed similarly, and both outperformed the Kullback-Leibler information method. In addition, it was found that the more stringent the termination criterion and the higher the correlation between the latent traits, the higher the resulting measurement precision and test reliability. Test reliability was very similar across the dimensions, regardless of the correlation between the latent traits and termination criterion. On average, the difficulties of the administered items were found to be at a lower level than the examinees' abilities, which shed light on item bank construction for non-compensatory items.

| S-EPMC6696872 | biostudies-literature

The Block Item Pocket Method for Reviewable Multidimensional Computerized Adaptive Testing.

Project description:Most computerized adaptive testing (CAT) programs do not allow item review due to a decrease in estimation precision and aberrant manipulation strategies. In this article, a block item pocket (BIP) method that combines the item pocket method with the successive block method to realize reviewable CAT was proposed. A worst-case but still reasonable answering strategy and the Wainer-like manipulation strategy were simulated to evaluate the estimation precision of reviewable unidimensional computerized adaptive testing (UCAT) and multidimensional computerized adaptive testing (MCAT) under a series of BIP settings. For both UCAT and MCAT, it was found that the estimation precision of the BIP method improved as the number of blocks increased or the item pocket size decreased under the reasonable strategy. The BIP method was more effective in handling the Wainer-like strategy. With the help of block design, the BIP method can still maintain acceptable estimation precision under slightly large total IP size conditions. These results suggested that the BIP method was a reliable solution for both reviewable UCAT and MCAT.

| S-EPMC7711249 | biostudies-literature

Comparing diagnostic tests on benefit-risk.

Project description:Comparing diagnostic tests on accuracy alone can be inconclusive. For example, a test may have better sensitivity than another test yet worse specificity. Comparing tests on benefit risk may be more conclusive because clinical consequences of diagnostic error are considered. For benefit-risk evaluation, we propose diagnostic yield, the expected distribution of subjects with true positive, false positive, true negative, and false negative test results in a hypothetical population. We construct a table of diagnostic yield that includes the number of false positive subjects experiencing adverse consequences from unnecessary work-up. We then develop a decision theory for evaluating tests. The theory provides additional interpretation to quantities in the diagnostic yield table. It also indicates that the expected utility of a test relative to a perfect test is a weighted accuracy measure, the average of sensitivity and specificity weighted for prevalence and relative importance of false positive and false negative testing errors, also interpretable as the cost-benefit ratio of treating non-diseased and diseased subjects. We propose plots of diagnostic yield, weighted accuracy, and relative net benefit of tests as functions of prevalence or cost-benefit ratio. Concepts are illustrated with hypothetical screening tests for colorectal cancer with test positive subjects being referred to colonoscopy.

| S-EPMC5471848 | biostudies-other

Item Selection Methods in Multidimensional Computerized Adaptive Testing With Polytomously Scored Items.

Project description:Multidimensional computerized adaptive testing (MCAT) has been developed over the past decades, and most of them can only deal with dichotomously scored items. However, polytomously scored items have been broadly used in a variety of tests for their advantages of providing more information and testing complicated abilities and skills. The purpose of this study is to discuss the item selection algorithms used in MCAT with polytomously scored items (PMCAT). Several promising item selection algorithms used in MCAT are extended to PMCAT, and two new item selection methods are proposed to improve the existing selection strategies. Two simulation studies are conducted to demonstrate the feasibility of the extended and proposed methods. The simulation results show that most of the extended item selection methods for PMCAT are feasible and the new proposed item selection methods perform well. Combined with the security of the pool, when two dimensions are considered (Study 1), the proposed modified continuous entropy method (MCEM) is the ideal of all in that it gains the lowest item exposure rate and has a relatively high accuracy. As for high dimensions (Study 2), results show that mutual information (MUI) and MCEM keep relatively high estimation accuracy, and the item exposure rates decrease as the correlation increases.

| S-EPMC6291894 | biostudies-literature

OmicsDI is part of the ELIXIR infrastructure

OmicsDI is an Elixir interoperability service. Learn more ›

Tweets

OmicsDI Databases

PRIDE
PeptideAtlas
MassIVE
JPOST Repository
Physiome Model Repository

EGA
EVA
ENA
LINCS
PAXDB
Cell Collective

MetaboLights
Metabolomics Workbench
MetabolomeExpress
GNPS
BioModels
FAIRDOMHub

ArrayExpress
dbGaP
ExpressionAtlas
GEO
NODE

Information

Databases
Help
API
Contact us
Code on GitHub
Terms of use
Submit Data