Modeling Composite Assessment Data Using Item Response Theory.
Ontology highlight
ABSTRACT:
SUBMITTER: Ueckert S
PROVIDER: S-EPMC5915608 | biostudies-literature | 2018 Apr
REPOSITORIES: biostudies-literature
Publications
Modeling Composite Assessment Data Using Item Response Theory.
CPT: pharmacometrics & systems pharmacology 20180301 4
Composite assessments aim to combine different aspects of a disease in a single score and are utilized in a variety of therapeutic areas. The data arising from these evaluations are inherently discrete with distinct statistical properties. This tutorial presents the framework of the item response theory (IRT) for the analysis of this data type in a pharmacometric context. The article considers both conceptual (terms and assumptions) and practical questions (modeling software, data requirements, ...[more]
Similar Datasets
Project description:PurposeThis work investigates improved utilization of ADAS-cog data (the primary outcome in Alzheimer's disease (AD) trials of mild and moderate AD) by combining pharmacometric modeling and item response theory (IRT).MethodsA baseline IRT model characterizing the ADAS-cog was built based on data from 2,744 individuals. Pharmacometric methods were used to extend the baseline IRT model to describe longitudinal ADAS-cog scores from an 18-month clinical study with 322 patients. Sensitivity of the ADAS-cog items in different patient populations as well as the power to detect a drug effect in relation to total score based methods were assessed with the IRT based model.ResultsIRT analysis was able to describe both total and item level baseline ADAS-cog data. Longitudinal data were also well described. Differences in the information content of the item level components could be quantitatively characterized and ranked for mild cognitively impairment and mild AD populations. Based on clinical trial simulations with a theoretical drug effect, the IRT method demonstrated a significantly higher power to detect drug effect compared to the traditional method of analysis.ConclusionA combined framework of IRT and pharmacometric modeling permits a more effective and precise analysis than total score based methods and therefore increases the value of ADAS-cog data.
| S-EPMC4153970 | biostudies-literature
Project description:Responses to survey data are determined not only by item characteristics and respondents' trait standings but also by response styles. Recently, methods for modeling response style with personality and attitudinal data have turned toward the use of anchoring vignettes, which provide fixed rating targets. Although existing research is promising, a few outstanding questions remain. First, it is not known how many vignettes and vignette ratings are necessary to identify response style parameters. Second, the comparative accuracy of these models is largely unexplored. Third, it remains unclear whether correcting for response style improves criterion validity. Both simulated data and data observed from a population-representative sample responding to a measure of personality pathology (the Personality Inventory for DSM-5 [PID-5]) were modeled using an array of response style models. In simulations, most models estimating response styles outperformed the graded response model (GRM), and in observed data, all response style models were superior to the GRM. Correcting for response style had a small, but in some cases significant, effect on the prediction of self-reported social dysfunction.
| S-EPMC6297915 | biostudies-literature
Project description:The Tucker-Lewis index (TLI; Tucker & Lewis, 1973), also known as the non-normed fit index (NNFI; Bentler & Bonett, 1980), is one of the numerous incremental fit indices widely used in linear mean and covariance structure modeling, particularly in exploratory factor analysis, tools popular in prevention research. It augments information provided by other indices such as the root-mean-square error of approximation (RMSEA). In this paper, we develop and examine an analogous index for categorical item level data modeled with item response theory (IRT). The proposed Tucker-Lewis index for IRT (TLIRT) is based on Maydeu-Olivares and Joe's (2005) [Formula: see text] family of limited-information overall model fit statistics. The limited-information fit statistics have significantly better Chi-square approximation and power than traditional full-information Pearson or likelihood ratio statistics under realistic situations. Building on the incremental fit assessment principle, the TLIRT compares the fit of model under consideration along a spectrum of worst to best possible model fit scenarios. We examine the performance of the new index using simulated and empirical data. Results from a simulation study suggest that the new index behaves as theoretically expected, and it can offer additional insights about model fit not available from other sources. In addition, a more stringent cutoff value is perhaps needed than Hu and Bentler's (1999) traditional cutoff criterion with continuous variables. In the empirical data analysis, we use a data set from a measurement development project in support of cigarette smoking cessation research to illustrate the usefulness of the TLIRT. We noticed that had we only utilized the RMSEA index, we could have arrived at qualitatively different conclusions about model fit, depending on the choice of test statistics, an issue to which the TLIRT is relatively more immune.
| S-EPMC10115722 | biostudies-literature
Project description:Allostatic load is commonly operationalized using a sum-score of high-risk biomarkers. However, this method implies that biomarkers contribute equally to allostatic load, as each is given equal weight. Our goal in this methodological paper is to evaluate this, and complementarily, to identify biomarkers that are most informative and least informative for developing an allostatic load index. Item response theory models provide an alternate approach to calculating the allostatic load score, by treating individual biomarkers (e.g. “items”) as indicators of a latent allostatic load construct. Item response theory scores account for the data-driven discriminating power of each biomarker, and an individual’s pattern of biomarker responses. To demonstrate feasibility of this approach, we used data from the 2015–2016 National Health Examination and Nutrition Survey (NHANES; N = 3751), with twelve allostatic load biomarkers representing immune response, metabolic function and cardiovascular health. Item response theory models revealed that body-mass-index and C-reactive protein were the most informative biomarkers for allostatic load. Both higher allostatic load sum-score and allostatic load item response theory score were associated with lower socio-economic status (p = 0.008; p<0.001, respectively). Further, both formulations of allostatic load were positively associated with a nine-item depression screener (p<0.001 for both), but only the item response theory score was also positively associated with the impact of depressive symptoms on daily life (p = 0.045). Item response theory scores may be more finely tuned to tease out effects, compared to sum-scores, and also provide more flexibility when there are missing biomarker measurements. Supplemental R code for our approach are included. Highlights • Methodological paper to introduce item response theory for calculating allostatic load.• Biomarker data from NHANES 2015–2016 representative of United States adults.• Body-mass-index and C-reactive protein most informative for allostatic load.• Item response theory captures more variability in allostatic load compared to sum-scores.• Future work - item response theory can standardize allostatic load across datasets.
| S-EPMC9216382 | biostudies-literature
Project description:PurposeMeaningful thresholds are needed to interpret patient-reported outcome measure (PROM) results. This paper introduces a new method, based on item response theory (IRT), to estimate such thresholds. The performance of the method is examined in simulated datasets and two real datasets, and compared with other methods.MethodsThe IRT method involves fitting an IRT model to the PROM items and an anchor item indicating the criterion state of interest. The difficulty parameter of the anchor item represents the meaningful threshold on the latent trait. The latent threshold is then linked to the corresponding expected PROM score. We simulated 4500 item response datasets to a 10-item PROM, and an anchor item. The datasets varied with respect to the mean and standard deviation of the latent trait, and the reliability of the anchor item. The real datasets consisted of a depression scale with a clinical depression diagnosis as anchor variable and a pain scale with a patient acceptable symptom state (PASS) question as anchor variable.ResultsThe new IRT method recovered the true thresholds accurately across the simulated datasets. The other methods, except one, produced biased threshold estimates if the state prevalence was smaller or greater than 0.5. The adjusted predictive modeling method matched the new IRT method (also in the real datasets) but showed some residual bias if the prevalence was smaller than 0.3 or greater than 0.7.ConclusionsThe new IRT method perfectly recovers meaningful (interpretational) thresholds for multi-item questionnaires, provided that the data satisfy the assumptions for IRT analysis.
| S-EPMC10172229 | biostudies-literature
Project description:Clinical psychologists are advised to assess clinical and statistical significance when assessing change in individual patients. Individual change assessment can be conducted using either the methodologies of classical test theory (CTT) or item response theory (IRT). Researchers have been optimistic about the possible advantages of using IRT rather than CTT in change assessment. However, little empirical evidence is available to support the alleged superiority of IRT in the context of individual change assessment. In this study, the authors compared the CTT and IRT methods with respect to their Type I error and detection rates. Preliminary results revealed that IRT is indeed superior to CTT in individual change detection, provided that the tests consist of at least 20 items. For shorter tests, however, CTT is generally better at correctly detecting change in individuals. The results and their implications are discussed.
| S-EPMC5978722 | biostudies-literature
Project description:Three widely used interpersonal trust measurement scales [Interpersonal Trust Scale (ITS), Philosophies of Human Nature Scale (RPHNS), Company Trust Scale (CTS)] have seldom been applied in non-Western contexts. Different social environments may lead to variation in the level or structure of trust. Therefore, it is necessary to compare the applicability of these scales to different levels of trust-related traits in Eastern cultures so that researchers can choose appropriate scales for relevant studies. This study attempted to conduct a comparative analysis of the ITS, RPHNS, and CTS. A sample of 725 Chinese college students was analyzed. Total score correlations and latent factor correlations estimated by confirmatory factor analysis (CFA) for a first-order three-factor model were assessed, and then the quality of the item parameters, test reliability and standard errors, and test information were assessed. The results are as follows: (1) the ITS and the RPHNS assessed almost the same trust traits; therefore, only the ITS and the RPHNS are compared in the next sections; (2) the original structure of only the RPHNS is verified; (3) some items on the ITS do not work well, while the RPHNS has higher overall test reliability; and (4) the average item information provided by the RPHNS is higher across all trait levels. In most cases, the RPHNS is the better choice in the Chinese cultural context.
| S-EPMC6989430 | biostudies-literature
Project description:Item response theory (IRT) is moving to the forefront of methodologies used to develop, evaluate, and score clinical measures. Funding agencies and test developers are routinely supporting IRT work, and the theory has become closely tied to technological advances within the field. As a result, familiarity with IRT has grown increasingly relevant to mental health research and practice. But to what end? This article reviews advances in applications of IRT to clinical measurement in an effort to identify tangible improvements that can be attributed to the methodology. Although IRT shares similarities with classical test theory and factor analysis, the approach has certain practical benefits, but also limitations, when applied to measurement challenges. Major opportunities include the use of computerized adaptive tests to prevent conditional measurement error, multidimensional models to prevent misinterpretation of scores, and analyses of differential item functioning to prevent bias. Whereas these methods and technologies were once only discussed as future possibilities, they are now accessible because of recent support of IRT-focused clinical research. Despite this, much work still remains in widely disseminating methods and technologies from IRT into mental health research and practice. Clinicians have been reluctant to fully embrace the approach, especially in terms or prospective test development and adaptive item administration. Widespread use of IRT technologies will require continued cooperation among psychometricians, clinicians, and other stakeholders. There are also many opportunities to expand the methodology, especially with respect to integrating modern measurement theory with models from personality and cognitive psychology as well as neuroscience. (PsycINFO Database Record (c) 2019 APA, all rights reserved).
| S-EPMC6745011 | biostudies-literature
Project description:Some large-scale testing requires examinees to select and answer a fixed number of items from given items (e.g., select one out of the three items). Usually, they are constructed-response items that are marked by human raters. In this examinee-selected item (ESI) design, some examinees may benefit more than others from choosing easier items to answer, and so the missing data induced by the design become missing not at random (MNAR). Although item response theory (IRT) models have recently been developed to account for MNAR data in the ESI design, they do not consider the rater effect; thus, their utility is seriously restricted. In this study, two methods are developed: the first one is a new IRT model to account for both MNAR data and rater severity simultaneously, and the second one adapts conditional maximum likelihood estimation and pairwise estimation methods to the ESI design with the rater effect. A series of simulations was then conducted to compare their performance with those of conventional IRT models that ignored MNAR data or rater severity. The results indicated a good parameter recovery for the new model. The conditional maximum likelihood estimation and pairwise estimation methods were applicable when the Rasch models fit the data, but the conventional IRT models yielded biased parameter estimates. An empirical example was given to illustrate these new initiatives.
| S-EPMC6696873 | biostudies-literature
Project description:Item-level data from composite scales can be analyzed with pharmacometric item response theory (IRT) models to improve the quantification of disease severity compared with the use of total composite scores. However, regular IRT models assume unidimensionality, which is violated in the scale measuring iatrogenic withdrawal in children because some items are also affected by pain, undersedation, or delirium. Here, we compare regular IRT modelling of pediatric iatrogenic withdrawal symptom data with two new analysis approaches in which the latent variable is guided towards the condition of interest using numerical withdrawal severity scored by nurses as a "supervising variable:" supervised IRT (sIRT) and supervised multi-dimensional (smIRT) modelling. In this example, in which the items scores are affected by multiple conditions, regular IRT modeling is worse to quantify disease severity than the total composite score, whereas improved performance compared with the composite score is observed for the sIRT and smIRT models.
| S-EPMC6930857 | biostudies-literature