Dataset Information

Methods for Estimating Item-Score Reliability.

ABSTRACT: Reliability is usually estimated for a test score, but it can also be estimated for item scores. Item-score reliability can be useful to assess the item's contribution to the test score's reliability, for identifying unreliable scores in aberrant item-score patterns in person-fit analysis, and for selecting the most reliable item from a test to use as a single-item measure. Four methods were discussed for estimating item-score reliability: the Molenaar-Sijtsma method (method MS), Guttman's method ?6 , the latent class reliability coefficient (method LCRC), and the correction for attenuation (method CA). A simulation study was used to compare the methods with respect to median bias, variability (interquartile range [IQR]), and percentage of outliers. The simulation study consisted of six conditions: standard, polytomous items, unequal ? parameters, two-dimensional data, long test, and small sample size. Methods MS and CA were the most accurate. Method LCRC showed almost unbiased results, but large variability. Method ?6 consistently underestimated item-score reliabilty, but showed a smaller IQR than the other methods.

SUBMITTER: Zijlmans EAO

PROVIDER: S-EPMC6140096 | biostudies-literature | 2018 Oct

REPOSITORIES: biostudies-literature

ACCESS DATA

Publications

Methods for Estimating Item-Score Reliability.

Zijlmans Eva A O EAO van der Ark L Andries LA Tijmstra Jesper J Sijtsma Klaas K

Applied psychological measurement 20180409 7

Reliability is usually estimated for a test score, but it can also be estimated for item scores. Item-score reliability can be useful to assess the item's contribution to the test score's reliability, for identifying unreliable scores in aberrant item-score patterns in person-fit analysis, and for selecting the most reliable item from a test to use as a single-item measure. Four methods were discussed for estimating item-score reliability: the Molenaar-Sijtsma method (method MS), Guttman's metho ...[more]

PMID: 30237646

Similar Datasets

Project description:The usual method for assessing the reliability of survey data has been to conduct reinterviews a short interval (such as one to two weeks) after an initial interview and to use these data to estimate relatively simple statistics, such as gross difference rates (GDRs). More sophisticated approaches have also been used to estimate reliability. These include estimates from multi-trait, multi-method experiments, models applied to longitudinal data, and latent class analyses. To our knowledge, no prior study has systematically compared these different methods for assessing reliability. The Population Assessment of Tobacco and Health Reliability and Validity (PATH-RV) Study, done on a national probability sample, assessed the reliability of answers to the Wave 4 questionnaire from the PATH Study. Respondents in the PATH-RV were interviewed twice about two weeks apart. We examined whether the classic survey approach yielded different conclusions from the more sophisticated methods. We also examined two ex ante methods for assessing problems with survey questions and item nonresponse rates and response times to see how strongly these related to the different reliability estimates. We found that kappa was highly correlated with both GDRs and over-time correlations, but the latter two statistics were less highly correlated, particularly for adult respondents; estimates from longitudinal analyses of the same items in the main PATH study were also highly correlated with the traditional reliability estimates. The latent class analysis results, based on fewer items, also showed a high level of agreement with the traditional measures. The other methods and indicators had at best weak relationships with the reliability estimates derived from the reinterview data. Although the Question Understanding Aid seems to tap a different factor from the other measures, for adult respondents, it did predict item nonresponse and response latencies and thus may be a useful adjunct to the traditional measures.

Project description:Background The Predicting Bleeding Complications in Patients Undergoing Stent Implantation and Subsequent Dual Antiplatelet Therapy (PRECISE-DAPT) score has been shown to predict out-of-hospital major bleeding after myocardial infarction treated with percutaneous coronary intervention and dual antiplatelet therapy (DAPT). However, large validation studies have been scarce and the discriminative ability for patients with a preexisting bleeding risk factor (elderly, underweight, women, anemia, kidney dysfunction, or cancer) in a real-world setting is unknown. Methods and Results Patients undergoing percutaneous coronary intervention for myocardial infarction between 2008 and 2017 were included from the SWEDEHEART (Swedish Web System for Enhancement of Evidence-Based Care in Heart Disease Evaluated According to Recommended Therapies) registry (n=66 295). The predictive value of the PRECISE-DAPT score for rehospitalization with major bleeding during dual antiplatelet therapy was evaluated using receiver operating characteristic analyses. A high PRECISE-DAPT score (≥25; n=13 894) was associated with increased risk of major bleeding (3.9% versus 1.8%; hazard ratio [HR], 2.2; 95% CI, 2.0-2.5; P<0.001) compared with a non-high score (<25; n=52 401). The score demonstrated a c-statistic of 0.64 (95% CI, 0.63-0.66). The discriminative ability of the score to further stratify bleeding risk in patients with preexisting bleeding risk factors was poor, especially in patients who are elderly (c-statistic=0.57; 95% CI, 0.55-0.60) or underweight (c-statistic=0.56; 95% CI, 0.51-0.61), for whom a non-high PRECISE-DAPT score was associated with similar bleeding risk as a high PRECISE-DAPT score in the general myocardial infarction population. Conclusions In this nationwide population-based study, the PRECISE-DAPT score performed moderately in the general myocardial infarction population and poorly in patients with preexisting bleeding risk factors, where its usefulness seems limited.

Project description:Background and purposeThe modified TICI score is the benchmark for quantifying reperfusion after mechanical thrombectomy. There has been limited investigation into the reliability of this score. We aim to identify intra-rater and inter-rater reliability of the mTICI score among endovascular neurosurgeons.Materials and methodsFour independent endovascular neurosurgeons (raters) reviewed angiograms of 67 patients at 2 time points. κ statistics assessed inter- and intrarater reliability and compared raters'-versus-proceduralists' scores. Reliability was also assessed for occlusion location and by dichotomizing modified TICI scores (0-2a versus 2b-3).ResultsInterrater reliability was moderate-to-substantial, weighted κ = 0.417-0.703, overall κ = 0.374 (P < .001). The dichotomized modified TICI score had moderate-to-substantial interrater agreement, κ statistics = 0.468-0.715, overall κ = 0.582 (P < .001). Intrarater reliability was moderate-to-almost perfect, weighted κ = 0.594-0.81. The dichotomized modified TICI score had substantial-to-almost perfect reliability, κ = 0.632-0.82. Proceduralists had fair-to-moderate agreement with raters, weighted κ = 0.348-0.574, and the dichotomized modified TICI score had fair-to-moderate agreement, κ = 0.365-0.544. When proceduralists and raters disagreed, proceduralists' scores were higher in 79.6% of cases. M1 followed by ICA occlusions had the highest agreement.ConclusionsThe modified TICI score is a practical metric for assessing reperfusion after mechanical thrombectomy, though not without limitations. Agreement improved when scores were dichotomized around the clinically relevant threshold of successful revascularization. Interrater reliability improved with time, suggesting that formal training of interventionalists may improve reporting reliability. Agreement of the modified TICI scale is best with M1 and ICA occlusion and becomes less reliable with more distal or posterior circulation occlusions. These findings should be considered when developing research trials.

Dataset Information

Methods for Estimating Item-Score Reliability.

Publications

Methods for Estimating Item-Score Reliability.

Similar Datasets

OmicsDI is part of the ELIXIR infrastructure

Tweets