Dataset Information

Comparison of Classical Test Theory and Item Response Theory in Individual Change Assessment.

ABSTRACT: Clinical psychologists are advised to assess clinical and statistical significance when assessing change in individual patients. Individual change assessment can be conducted using either the methodologies of classical test theory (CTT) or item response theory (IRT). Researchers have been optimistic about the possible advantages of using IRT rather than CTT in change assessment. However, little empirical evidence is available to support the alleged superiority of IRT in the context of individual change assessment. In this study, the authors compared the CTT and IRT methods with respect to their Type I error and detection rates. Preliminary results revealed that IRT is indeed superior to CTT in individual change detection, provided that the tests consist of at least 20 items. For shorter tests, however, CTT is generally better at correctly detecting change in individuals. The results and their implications are discussed.

SUBMITTER: Jabrayilov R

PROVIDER: S-EPMC5978722 | biostudies-literature | 2016 Nov

REPOSITORIES: biostudies-literature

ACCESS DATA

Publications

Comparison of Classical Test Theory and Item Response Theory in Individual Change Assessment.

Jabrayilov Ruslan R Emons Wilco H M WHM Sijtsma Klaas K

Applied psychological measurement 20160924 8

Clinical psychologists are advised to assess clinical and statistical significance when assessing change in individual patients. Individual change assessment can be conducted using either the methodologies of classical test theory (CTT) or item response theory (IRT). Researchers have been optimistic about the possible advantages of using IRT rather than CTT in change assessment. However, little empirical evidence is available to support the alleged superiority of IRT in the context of individual ...[more]

PMID: 29881070

Similar Datasets

Project description:BackgroundThis study compares classical test theory and item response theory frameworks to determine reliable change. Reliable change followed by anchoring to the change in categorically distinct responses on a criterion measure is a useful method to detect meaningful change on a target measure.MethodsAdult cancer patients were recruited from five cancer centers. Baseline and follow-up assessments at 6 weeks were administered. We investigated short forms derived from PROMIS® item banks on anxiety, depression, fatigue, pain intensity, pain interference, and sleep disturbance. We detected reliable change using reliable change index (RCI). We derived the T-scores corresponding to the RCI calculated under IRT and CTT frameworks using PROMIS® short forms. For changes that were reliable, meaningful change was identified using patient-reported change in PRO-CTCAE by at least one level. For both CTT and IRT approaches, we applied one-sided tests to detect reliable improvement or worsening using RCI. We compared the percentages of patients with reliable change and reliable/meaningful change.ResultsThe amount of change in T score corresponding to RCICTT of 1.65 ranged from 5.1 to 9.2 depending on domains. The amount of change corresponding to RCIIRT of 1.65 varied across the score range, and the minimum change ranged from 3.0 to 8.2 depending on domains. Across domains, the RCICTT and RCIIRT classified 80% to 98% of the patients consistently. When there was disagreement, the RCIIRT tended to identify more patients as having reliably changed compared to RCICTT if scores at both timepoints were in the range of 43 to 78 in anxiety, 45 to 70 in depression, 38 to 80 in fatigue, 35 to 78 in sleep disturbance, and 48 to 74 in pain interference, due to smaller standard errors in these ranges using the IRT method. The CTT method found more changes compared to IRT for the pain intensity domain that was shorter in length. Using RCICTT, 22% to 66% had reliable change in either direction depending on domains, and among these patients, 62% to 83% had meaningful change. Using RCIIRT, 37% to 68% had reliable change in either direction, and among these patients, 62% to 81% had meaningful change.ConclusionApplying the two-step criteria demonstrated in this study, we determined how much change is needed to declare reliable change at different levels of baseline scores. We offer reference values for percentage of patients who meaningfully change for investigators using the PROMIS instruments in oncology.

Project description:BackgroundThe International Classification of Functioning, Disability and Health (ICF) proposes three main health outcomes, Impairment (I), Activity Limitation (A) and Participation Restriction (P), but good measures of these constructs are needed The aim of this study was to use both Classical Test Theory (CTT) and Item Response Theory (IRT) methods to carry out an item analysis to improve measurement of these three components in patients having joint replacement surgery mainly for osteoarthritis (OA).MethodsA geographical cohort of patients about to undergo lower limb joint replacement was invited to participate. Five hundred and twenty four patients completed ICF items that had been previously identified as measuring only a single ICF construct in patients with osteoarthritis. There were 13 I, 26 A and 20 P items. The SF-36 was used to explore the construct validity of the resultant I, A and P measures. The CTT and IRT analyses were run separately to identify items for inclusion or exclusion in the measurement of each construct. The results from both analyses were compared and contrasted.ResultsOverall, the item analysis resulted in the removal of 4 I items, 9 A items and 11 P items. CTT and IRT identified the same 14 items for removal, with CTT additionally excluding 3 items, and IRT a further 7 items. In a preliminary exploration of reliability and validity, the new measures appeared acceptable.ConclusionNew measures were developed that reflect the ICF components of Impairment, Activity Limitation and Participation Restriction for patients with advanced arthritis. The resulting Aberdeen IAP measures (Ab-IAP) comprising I (Ab-I, 9 items), A (Ab-A, 17 items), and P (Ab-P, 9 items) met the criteria of conventional psychometric (CTT) analyses and the additional criteria (information and discrimination) of IRT. The use of both methods was more informative than the use of only one of these methods. Thus combining CTT and IRT appears to be a valuable tool in the development of measures.

Project description:PURPOSE:The association of chronic lymphocytic leukemia (CLL) with health-related quality of life (HRQoL) is rarely studied globally. This study evaluated the psychometric properties of the EORTC-Chronic Lymphocytic Leukaemia (CLL17 [phase III]) module, a newly developed assessment on CLL patients' HRQoL, among Chinese CLL patients. METHODS:The Chinese CLL17, comprised of three subscales (symptom burden [SB], physical condition [PC] and worries/fears [WF]), was provided by the developer team through EORTC. A cross-sectional online survey was conducted to collect data. The classical traditional theory (CTT) and the item response theory (IRT) were used to evaluate the psychometric properties of CLL17. Internal consistency reliability was determined by the Cronbach's alpha and item-total correlation. Dimensionality was verified through confirmatory factor analysis (CFA). Convergent validity was also assessed. The generalized partial credit model was used for the IRT. The difficulty, discrimination, item fit, and differential item functioning (DIF) were calculated to assess the instrument's psychometric properties. RESULTS:In all, 318 patients, aged between 26 and 82 years, completed the questionnaire. A good level of internal reliability was achieved (Cronbach's alpha = 0.92). The item-total correlation coefficient ranged from 0.46 to 0.72. There was a mid-to-high correlation between CLL17 and domains of EQ-5D and QLQ-C30. The IRT model showed a satisfactory homogeneity, item fit and good discrimination of items, except for item 4, 6 and 16 (< 1.0). low information provided by item 16 and 17. SB and PC provided more information with theta > 0, whereas WF provided more information with theta < 0. Item 17 perform inconsistently for respondents from different age groups (DIF). CONCLUSION:The EORTC-CLL17 Chinese version shows acceptable reliability and validity, making it a valuable instrument to evaluate the impact on the HRQoL of Chinese CLL patients.

Project description:BackgroundEarly prelingual auditory development (EPLAD) is a fundamental and important process in the speech and language development of infants and toddlers. The Infant-Toddler Meaningful Auditory Integration Scale (ITMAIS) is a widely used measurement tool for EPLAD, however it has not yet undergone a comprehensive psychometric analysis. The aim of this research was to modify and verify the psychometric properties of ITMAIS using a combination of Item Response Theory (IRT) and Classical Test Theory (CTT).MethodsStage 1-1730 children were retrospectively recruited to enable the application of an IRT model, specifically the graded response model, to modify the ITMAIS. Stage 2-another 450 infants and toddlers with normal hearing or permanent hearing loss before auditory intervention were recruited to verify the psychometric properties of the modified ITMAIS (ITMAIS-m) using the CTT method.ResultsUsing the metric of the graded response model, by removing item 2 from the ITMAIS, ITMAIS-m demonstrated discrimination parameters ranging from 3.947 to 5.431, difficulty parameters from - 1.146 to 1.150, item information distributed between 4.798 and 9.259 and a test information score of 48.061. None of the items showed differential item functioning. ITMAIS-m was further verified in Stage 2, showing Cronbach's α of 0.919 and item-total correlations ranging from 0.693 to 0.851. There was good convergent validity of ITMAIS-m with other auditory outcome measure (r = 0.932) and pure tone average thresholds (r ranging from - 0.670 to - 0.909), as well as a high ability to discriminate between different hearing grades (Cohen d ranging from 0.41 to 5.83).ConclusionsThe ITMAIS-m is a reliable and valid tool for evaluating EPLAD in infants and toddlers, which can be efficiently and precisely applied in clinical practice. The combined use of IRT and CTT provides a powerful means to modify psychometrically robust scales aimed at childhood auditory outcome measurements.

Dataset Information

Comparison of Classical Test Theory and Item Response Theory in Individual Change Assessment.

Publications

Comparison of Classical Test Theory and Item Response Theory in Individual Change Assessment.

Similar Datasets

OmicsDI is part of the ELIXIR infrastructure

Tweets