Dataset Information

Low interrater reliability of examiners performing the prone instability test: a clinical test for lumbar shear instability.

ABSTRACT:

Objective

To test the interrater reliability of examiners performing the prone instability test (PIT), a clinical test proposed to identify lumbar shear instability.

Design

Cross-sectional test-retest design examining individuals with mechanical low back pain (LBP).

Setting

University-based musculoskeletal analysis laboratory.

Participants

Individuals (N=30) with mechanical LBP recruited from community sources in a metropolitan region.

Interventions

Not applicable.

Main outcome measures

Repeated measures of a clinical examination test proposed to identify lumbar shear instability.

Results

Interrater reliability of examiners' judgments of PIT results were indexed with percentage of agreement and ? statistic. Examiners obtained 63% agreement and ? of .10 (95% confidence interval, -.27 to .47). Adjusted ? values based on prevalence and bias indexes were calculated to evaluate the effect on ?. The prevalence index associated with examiner judgments of the PIT was .43, and bias index was .03. The prevalence-adjusted bias-adjusted ? value was slightly higher than the unadjusted ? value (?=.27; 95% confidence interval, -.08 to .61).

Conclusions

Results of our study are not consistent with those of previous studies examining the reliability of therapists performing the PIT. We conclude that examiners do not attain acceptable interrater reliability when performing procedures for the PIT based on the information currently provided in the literature. Based on our experience, we suggest further exploration, standardization, and clarification of procedural details to improve therapists' ability to conduct the PIT on individuals with LBP.

SUBMITTER: Ravenna MM

PROVIDER: S-EPMC3104326 | biostudies-literature | 2011 Jun

REPOSITORIES: biostudies-literature

ACCESS DATA

Publications

Low interrater reliability of examiners performing the prone instability test: a clinical test for lumbar shear instability.

Ravenna Margaret M MM Hoffman Shannon L SL Van Dillen Linda R LR

Archives of physical medicine and rehabilitation 20110601 6

<h4>Objective</h4>To test the interrater reliability of examiners performing the prone instability test (PIT), a clinical test proposed to identify lumbar shear instability.<h4>Design</h4>Cross-sectional test-retest design examining individuals with mechanical low back pain (LBP).<h4>Setting</h4>University-based musculoskeletal analysis laboratory.<h4>Participants</h4>Individuals (N=30) with mechanical LBP recruited from community sources in a metropolitan region.<h4>Interventions</h4>Not applic ...[more]

PMID: 21621668

Similar Datasets

Project description:BackgroundInterrater reliability, aka intercoder reliability, is defined as true agreement between raters, aka coders, without chance agreement. It is used across many disciplines including medical and health research to measure the quality of ratings, coding, diagnoses, or other observations and judgements. While numerous indices of interrater reliability are available, experts disagree on which ones are legitimate or more appropriate. Almost all agree that percent agreement (ao), the oldest and the simplest index, is also the most flawed because it fails to estimate and remove chance agreement, which is produced by raters' random rating. The experts, however, disagree on which chance estimators are legitimate or better. The experts also disagree on which of the three factors, rating category, distribution skew, or task difficulty, an index should rely on to estimate chance agreement, or which factors the known indices in fact rely on. The most popular chance-adjusted indices, according to a functionalist view of mathematical statistics, assume that all raters conduct intentional and maximum random rating while typical raters conduct involuntary and reluctant random rating. The mismatches between the assumed and the actual rater behaviors cause the indices to rely on mistaken factors to estimate chance agreement, leading to the numerous paradoxes, abnormalities, and other misbehaviors of the indices identified by prior studies.MethodsWe conducted a 4 × 8 × 3 between-subject controlled experiment with 4 subjects per cell. Each subject was a rating session with 100 pairs of rating by two raters, totaling 384 rating sessions as the experimental subjects. The experiment tested seven best-known indices of interrater reliability against the observed reliabilities and chance agreements. Impacts of the three factors, i.e., rating category, distribution skew, and task difficulty, on the indices were tested.ResultsThe most criticized index, percent agreement (ao), showed as the most accurate predictor of reliability, reporting directional r2 = .84. It was also the third best approximator, overestimating observed reliability by 13 percentage points on average. The three most acclaimed and most popular indices, Scott's π, Cohen's κ and Krippendorff's α, underperformed all other indices, reporting directional r2 = .312 and underestimated reliability by 31.4 ~ 31.8 points. The newest index, Gwet's AC1, emerged as the second-best predictor and the most accurate approximator. Bennett et al's S ranked behind AC1, and Perreault and Leigh's Ir ranked the fourth both for prediction and approximation. The reliance on category and skew and failure to rely on difficulty explain why the six chance-adjusted indices often underperformed ao, which they were created to outperform. The evidence corroborated the notion that the chance-adjusted indices assume intentional and maximum random rating while the raters instead exhibited involuntary and reluctant random rating.ConclusionThe authors call for more empirical studies and especially more controlled experiments to falsify or qualify this study. If the main findings are replicated and the underlying theories supported, new thinking and new indices may be needed. Index designers may need to refrain from assuming intentional and maximum random rating, and instead assume involuntary and reluctant random rating. Accordingly, the new indices may need to rely on task difficulty, rather than distribution skew or rating category, to estimate chance agreement.

Project description:BackgroundDespite widespread use by manual therapists, there is little evidence regarding the reliability of thoracic spine static palpation to test for a manipulable lesion using stiffness or tenderness as diagnostic markers. We aimed to determine the interrater agreement of thoracic spine static palpation for segmental tenderness and stiffness and determine the effect of standardised training for examiners. The secondary aim was to explore expert consensus on the level of segmental tenderness required to locate a "manipulable lesion".MethodsTwo experienced chiropractors used static palpation of thoracic vertebrae on two occasions (pragmatic and standardised approaches). Participants rated tenderness on an 11-point numerical pain rating scale (NPRS) and raters judged segmental stiffness based on their experience and perception of normal mobility with the requested outcomes of hypomobile or normal mobility. We calculated interrater agreement using percent agreement, Cohen's Kappa coefficients ( κ ) and prevalence-adjusted bias-adjusted Kappa coefficients (PABAK). In a preliminary study, an expert panel of 10 chiropractors took part in a Delphi process to identify the level of meaningful segmental tenderness required to locate a "manipulable lesion".ResultsThirty-six participants (20 female) were enrolled for the reliability study on the 13th March 2017. Mean (SD) age was 22.4 (3.4) years with an equal distribution of asymptomatic (n = 17) and symptomatic (n = 17) participants. Overall, the interrater agreement for spinal segmental stiffness had Kappa values indicating less than chance agreement [ κ range - 0.11, 0.53]. When adjusted for prevalence and bias, the PABAK ranged from slight to substantial agreement [0.12-0.76] with moderate or substantial agreement demonstrated at the majority of spinal levels (T1, T2 and T6 to T12). Generally, there was fair to substantial agreement for segmental tenderness [Kappa range 0.22-0.77]. Training did not significantly improve interrater agreement for stiffness or tenderness. The Delphi process indicated that an NPRS score of 2 out of 10 identified a potential "manipulable lesion".ConclusionStatic palpation was overall moderately reliable for the identification of segmental thoracic spine stiffness and tenderness, with tenderness demonstrating a higher reliability. Also, an increased agreement was found within the mid-thoracic spine. A brief training intervention failed to improve reliability.

Dataset Information

Low interrater reliability of examiners performing the prone instability test: a clinical test for lumbar shear instability.

Objective

Design

Setting

Participants

Interventions

Main outcome measures

Results

Conclusions

Publications

Low interrater reliability of examiners performing the prone instability test: a clinical test for lumbar shear instability.

Similar Datasets

OmicsDI is part of the ELIXIR infrastructure

Tweets