Dataset Information

Interrater Reliability of the Unified Huntington's Disease Rating Scale-Total Motor Score Certification.

ABSTRACT:

Background

The clinical assessment of motor symptoms in Huntington's disease is usually performed with the Unified Huntington's Disease Rating Scale-Total Motor Score (UHDRS-TMS). A high interrater reliability is desirable to monitor symptom progression. Therefore, a teaching video and a system for annual online certification has been developed and implemented.

Objectives

The aim of this study is to investigate the interrater reliability of the UHDRS-TMS and of its subitems, and to examine the performance of raters in consecutive years.

Methods

Data from the online UHDRS-TMS certification were used. The interrater reliability was assessed for all first-time participants (n = 944) between 2009 and 2016. Intraclass correlation coefficients (ICC) were calculated for each year separately and the mean was taken as the total ICC.

Results

The UHDRS-TMS (ICC = 0.847), tandem walking (0.824), pronate/supinate hands left (0.713), and retropulsion pull test (0.706) showed good interrater reliability. Poor interrater reliability was found for maximal dystonia of the left and right upper extremity (0.187 and 0.322, respectively), maximal dystonia of the left and right lower extremity (0.200 and 0.256, respectively), and maximal dystonia of the trunk (0.389), tongue protrusion (0.266), and rigidity arms left (0.390). Raters performed significantly worse on follow-up certification compared to their first certification.

Conclusions

Our results suggest that the rating of dystonia (absent, slight, mild, moderate, or marked) is subjective and difficult to interpret, especially on video. Therefore, changing the dystonia items of the UHDRS-TMS should be explored. We also recommend that raters should watch the UHDRS-TMS teaching video before each certification.

SUBMITTER: Winder JY

PROVIDER: S-EPMC6174470 | biostudies-literature | 2018 May-Jun

REPOSITORIES: biostudies-literature

ACCESS DATA

Publications

Interrater Reliability of the Unified Huntington's Disease Rating Scale-Total Motor Score Certification.

Winder Jessica Y JY Roos Raymund A C RAC Burgunder Jean-Marc JM Marinus Johan J Reilmann Ralf R

Movement disorders clinical practice 20180415 3

<h4>Background</h4>The clinical assessment of motor symptoms in Huntington's disease is usually performed with the Unified Huntington's Disease Rating Scale-Total Motor Score (UHDRS-TMS). A high interrater reliability is desirable to monitor symptom progression. Therefore, a teaching video and a system for annual online certification has been developed and implemented.<h4>Objectives</h4>The aim of this study is to investigate the interrater reliability of the UHDRS-TMS and of its subitems, and t ...[more]

PMID: 30363437

Similar Datasets

Project description:BackgroundThe diagnosis of Parkinson's disease (PD) and evaluation of its symptoms require in-person clinical examination. Remote evaluation of PD symptoms is desirable, especially during a pandemic such as the coronavirus disease 2019 pandemic. One potential method to remotely evaluate PD motor impairments is video-based analysis. In this study, we aimed to assess the feasibility of predicting the Unified Parkinson's Disease Rating Scale (UPDRS) score from gait videos using a convolutional neural network (CNN) model.MethodsWe retrospectively obtained 737 consecutive gait videos of 74 patients with PD and their corresponding neurologist-rated UPDRS scores. We utilized a CNN model for predicting the total UPDRS part III score and four subscores of axial symptoms (items 27, 28, 29, and 30), bradykinesia (items 23, 24, 25, 26, and 31), rigidity (item 22) and tremor (items 20 and 21). We trained the model on 80% of the gait videos and used 10% of the videos as a validation dataset. We evaluated the predictive performance of the trained model by comparing the model-predicted score with the neurologist-rated score for the remaining 10% of videos (test dataset). We calculated the coefficient of determination (R2) between those scores to evaluate the model's goodness of fit.ResultsIn the test dataset, the R2 values between the model-predicted and neurologist-rated values for the total UPDRS part III score and subscores of axial symptoms, bradykinesia, rigidity, and tremor were 0.59, 0.77, 0.56, 0.46, and 0.0, respectively. The performance was relatively low for videos from patients with severe symptoms.ConclusionsDespite the low predictive performance of the model for the total UPDRS part III score, it demonstrated relatively high performance in predicting subscores of axial symptoms. The model approximately predicted the total UPDRS part III scores of patients with moderate symptoms, but the performance was low for patients with severe symptoms owing to limited data. A larger dataset is needed to improve the model's performance in clinical settings.

Project description:BackgroundInterrater reliability, aka intercoder reliability, is defined as true agreement between raters, aka coders, without chance agreement. It is used across many disciplines including medical and health research to measure the quality of ratings, coding, diagnoses, or other observations and judgements. While numerous indices of interrater reliability are available, experts disagree on which ones are legitimate or more appropriate. Almost all agree that percent agreement (ao), the oldest and the simplest index, is also the most flawed because it fails to estimate and remove chance agreement, which is produced by raters' random rating. The experts, however, disagree on which chance estimators are legitimate or better. The experts also disagree on which of the three factors, rating category, distribution skew, or task difficulty, an index should rely on to estimate chance agreement, or which factors the known indices in fact rely on. The most popular chance-adjusted indices, according to a functionalist view of mathematical statistics, assume that all raters conduct intentional and maximum random rating while typical raters conduct involuntary and reluctant random rating. The mismatches between the assumed and the actual rater behaviors cause the indices to rely on mistaken factors to estimate chance agreement, leading to the numerous paradoxes, abnormalities, and other misbehaviors of the indices identified by prior studies.MethodsWe conducted a 4 × 8 × 3 between-subject controlled experiment with 4 subjects per cell. Each subject was a rating session with 100 pairs of rating by two raters, totaling 384 rating sessions as the experimental subjects. The experiment tested seven best-known indices of interrater reliability against the observed reliabilities and chance agreements. Impacts of the three factors, i.e., rating category, distribution skew, and task difficulty, on the indices were tested.ResultsThe most criticized index, percent agreement (ao), showed as the most accurate predictor of reliability, reporting directional r2 = .84. It was also the third best approximator, overestimating observed reliability by 13 percentage points on average. The three most acclaimed and most popular indices, Scott's π, Cohen's κ and Krippendorff's α, underperformed all other indices, reporting directional r2 = .312 and underestimated reliability by 31.4 ~ 31.8 points. The newest index, Gwet's AC1, emerged as the second-best predictor and the most accurate approximator. Bennett et al's S ranked behind AC1, and Perreault and Leigh's Ir ranked the fourth both for prediction and approximation. The reliance on category and skew and failure to rely on difficulty explain why the six chance-adjusted indices often underperformed ao, which they were created to outperform. The evidence corroborated the notion that the chance-adjusted indices assume intentional and maximum random rating while the raters instead exhibited involuntary and reluctant random rating.ConclusionThe authors call for more empirical studies and especially more controlled experiments to falsify or qualify this study. If the main findings are replicated and the underlying theories supported, new thinking and new indices may be needed. Index designers may need to refrain from assuming intentional and maximum random rating, and instead assume involuntary and reluctant random rating. Accordingly, the new indices may need to rely on task difficulty, rather than distribution skew or rating category, to estimate chance agreement.

Dataset Information

Interrater Reliability of the Unified Huntington's Disease Rating Scale-Total Motor Score Certification.

Background

Objectives

Methods

Results

Conclusions

Publications

Interrater Reliability of the Unified Huntington's Disease Rating Scale-Total Motor Score Certification.

Similar Datasets

OmicsDI is part of the ELIXIR infrastructure

Tweets