Dataset Information

The reproducibility of psychiatric evaluations of work disability: two reliability and agreement studies.

ABSTRACT: BACKGROUND:Expert psychiatrists conducting work disability evaluations often disagree on work capacity (WC) when assessing the same patient. More structured and standardised evaluations focusing on function could improve agreement. The RELY studies aimed to establish the inter-rater reproducibility (reliability and agreement) of 'functional evaluations' in patients with mental disorders applying for disability benefits and to compare the effect of limited versus intensive expert training on reproducibility. METHODS:We performed two multi-centre reproducibility studies on standardised functional WC evaluation (RELY 1 and 2). Trained psychiatrists interviewed 30 and 40 patients respectively and determined WC using the Instrument for Functional Assessment in Psychiatry (IFAP). Three psychiatrists per patient estimated WC from videotaped evaluations. We analysed reliability (intraclass correlation coefficients [ICC]) and agreement ('standard error of measurement' [SEM] and proportions of comparisons within prespecified limits) between expert evaluations of WC. Our primary outcome was WC in alternative work (WCalternative.work), 100-0%. Secondary outcomes were WC in last job (WClast.job), 100-0%; patients' perceived fairness of the evaluation, 10-0, higher is better; usefulness to psychiatrists. RESULTS:Inter-rater reliability for WCalternative.work was fair in RELY 1 (ICC 0.43; 95%CI 0.22-0.60) and RELY 2 (ICC 0.44; 0.25-0.59). Agreement was low in both studies, the 'standard error of measurement' for WCalternative.work was 24.6 percentage points (20.9-28.4) and 19.4 (16.9-22.0) respectively. Using a 'maximum acceptable difference' of 25 percentage points WCalternative.work between two experts, 61.6% of comparisons in RELY 1, and 73.6% of comparisons in RELY 2 fell within these limits. Post-hoc secondary analysis for RELY 2 versus RELY 1 showed a significant change in SEMalternative.work (-?5.2 percentage points WCalternative.work [95%CI -?9.7 to -?0.6]), and in the proportions on the differences ??25 percentage points WCalternative.work between two experts (p =?0.008). Patients perceived the functional evaluation as fair (RELY 1: mean 8.0; RELY 2: 9.4), psychiatrists as useful. CONCLUSIONS:Evidence from non-randomised studies suggests that intensive training in functional evaluation may increase agreement on WC between experts, but fell short to reach stakeholders' expectations. It did not alter reliability. Isolated efforts in training psychiatrists may not suffice to reach the expected level of agreement. A societal discussion about achievable goals and readiness to consider procedural changes in WC evaluations may deserve considerations.

SUBMITTER: Kunz R

PROVIDER: S-EPMC6607597 | biostudies-literature | 2019 Jul

REPOSITORIES: biostudies-literature

ACCESS DATA

Publications

The reproducibility of psychiatric evaluations of work disability: two reliability and agreement studies.

Kunz Regina R von Allmen David Y DY Marelli Renato R Hoffmann-Richter Ulrike U Jeger Joerg J Mager Ralph R Colomb Etienne E Schaad Heinz J HJ Bachmann Monica M Vogel Nicole N Busse Jason W JW Eichhorn Martin M Bänziger Oskar O Zumbrunn Thomas T de Boer Wout E L WEL Fischer Katrin K

BMC psychiatry 20190703 1

<h4>Background</h4>Expert psychiatrists conducting work disability evaluations often disagree on work capacity (WC) when assessing the same patient. More structured and standardised evaluations focusing on function could improve agreement. The RELY studies aimed to establish the inter-rater reproducibility (reliability and agreement) of 'functional evaluations' in patients with mental disorders applying for disability benefits and to compare the effect of limited versus intensive expert training ...[more]

PMID: 31266488

Similar Datasets

Project description:Objectives To explore agreement among healthcare professionals assessing eligibility for work disability benefits.Design Systematic review and narrative synthesis of reproducibility studies.Data sources Medline, Embase, and PsycINFO searched up to 16 March 2016, without language restrictions, and review of bibliographies of included studies.Eligibility criteria Observational studies investigating reproducibility among healthcare professionals performing disability evaluations using a global rating of working capacity and reporting inter-rater reliability by a statistical measure or descriptively. Studies could be conducted in insurance settings, where decisions on ability to work include normative judgments based on legal considerations, or in research settings, where decisions on ability to work disregard normative considerations. : Teams of paired reviewers identified eligible studies, appraised their methodological quality and generalisability, and abstracted results with pretested forms. As heterogeneity of research designs and findings impeded a quantitative analysis, a descriptive synthesis stratified by setting (insurance or research) was performed.Results From 4562 references, 101 full text articles were reviewed. Of these, 16 studies conducted in an insurance setting and seven in a research setting, performed in 12 countries, met the inclusion criteria. Studies in the insurance setting were conducted with medical experts assessing claimants who were actual disability claimants or played by actors, hypothetical cases, or short written scenarios. Conditions were mental (n=6, 38%), musculoskeletal (n=4, 25%), or mixed (n=6, 38%). Applicability of findings from studies conducted in an insurance setting to real life evaluations ranged from generalisable (n=7, 44%) and probably generalisable (n=3, 19%) to probably not generalisable (n=6, 37%). Median inter-rater reliability among experts was 0.45 (range intraclass correlation coefficient 0.86 to κ-0.10). Inter-rater reliability was poor in six studies (37%) and excellent in only two (13%). This contrasts with studies conducted in the research setting, where the median inter-rater reliability was 0.76 (range 0.91-0.53), and 71% (5/7) studies achieved excellent inter-rater reliability. Reliability between assessing professionals was higher when the evaluation was guided by a standardised instrument (23 studies, P=0.006). No such association was detected for subjective or chronic health conditions or the studies' generalisability to real world evaluation of disability (P=0.46, 0.45, and 0.65, respectively).Conclusions Despite their common use and far reaching consequences for workers claiming disabling injury or illness, research on the reliability of medical evaluations of disability for work is limited and indicates high variation in judgments among assessing professionals. Standardising the evaluation process could improve reliability. Development and testing of instruments and structured approaches to improve reliability in evaluation of disability are urgently needed.

Project description:ObjectiveInsurers frequently commission medical experts to estimate the degree of the remaining work capacity (RWC) in claimants for disability benefits. The social functioning scale Mini-ICF-APP allows for a rating of activity and participation limitations in thirteen capacity domains, considered as particularly relevant for work ability. The current study sought to evaluate the role of the Mini-ICF-APP ratings in psychiatric work disability evaluations, by examining how the capacity limitation ratings varied with the claimants' primary psychiatric diagnoses and how the ratings were related to RWC estimates.MethodsMedical experts estimated the RWC of 946 claimants with mental disorders and rated their activity and participation limitations using the Mini-ICF-APP, with higher ratings reflecting more severe limitations. The ratings were compared between claimants with different psychiatric diagnoses by analyses of variance. The mean Mini-ICF-APP rating across all capacity domains as well as all capacity-specific ratings were entered in simple or multiple regression models to predict the RWC in an alternative job.ResultsThe Mini-ICF-APP capacity limitation ratings in all domains but mobility were higher for claimants with personality and behavior disorders as compared to those with mood disorders or with neurotic, stress-related and somatoform disorders. The largest differences were observed in social capacities (e.g. group integration: F 2, 847 = 78.300, P < 0.001). In claimants with depression, all ratings increased with the severity of the diagnosis (all Fs 2, 203 > 16.393, all Ps < 0.001). In the overall sample, the mean Mini-ICF-APP rating showed a strong negative correlation with the estimated RWC (r = -.720, P < 0.001). Adding the capacity-specific ratings to the prediction model improved this prediction only marginally.DiscussionThe Mini-ICF-APP allows for documenting claimants' activity and participation limitations, which is likely to increase the transparency of medical experts' RWC estimates and enables them to check the plausibility of such estimates. However, our study showed that despite the strong association between RWC and Mini-ICF-APP ratings, half of the RWC variance was unrelated to the capacity limitations documented in the Mini-ICF-APP.

Project description:ObjectiveFunctional evaluations establish functional and work (in-)capacities in the context of disability assessments and are increasingly recommended as a modern technique for work disability assessments. The RELY (Reliable disability EvaLuation in psychiatrY)-studies introduced semi-structured functional interviews in real-life assessments of claimants with mental disorders for evaluating their self-perceived health-related limitations and for investigating the reproducibility of work capacity (WC) estimates. Functional interviews elicit claimants' self-perceptions about their work-related limitations and capacities in the labour market. This secondary data analysis explored the coverage of work-related key topics in these interviews and investigated whether interviews with high coverage (versus low coverage) of work-related topics resulted in better reproducibility of WC estimates among experts.MethodsThirty video-taped RELY-assessments underwent a content analysis along a predefined framework for functional interviewing, including the claimant's self-perceived work limitations and work-related health complaints as centrepieces of functional interviewing. Following transcription, interviews were segmented into coding units. Coding units were allocated to the five steps with 19 key topics of the framework. Enquiry into key topics was ascertained by summing the functional coding units per key topic. Median split grouped the interviews into high and low coverage of functional topics and compared them for inter-rater reliability (intraclass correlation coefficient, ICC) and inter-rater agreement (standard error of measurement, SEM).ResultsInterviews were broken down in 40,010 coding units, 31% of which addressed functional topics. Enquiries in self-perceived work limitations and work-related health complaints were sparse (coding units medianpsychiatrist between 0 and 1.5, medianpatients between 0 and 9.5). High coverage interviews enquired on more functional topics (68% vs. 42%, chi2(1, N = 38) = 5.32, p = 0.021) and in more depth (36% vs. 16% of functional coding units, chi2(1, N = 1,314) = 141.15, p < 0.001). Interviews with higher functional coverage reached significantly higher inter-rater agreement in WC ratings among experts (mean difference in SEM, low-high coverage, 7.5% WC, 95% CI 0.2 to 15.1%WC). Inter-rater reliability was low in both groups (ICC, 0.38 versus 0.40).ConclusionsContent analysis showed little enquiry by experts on claimants' self-perceived activity limitations and work-related capacity. The association between interviews with higher functional coverage and better expert agreement on the claimants' remaining WC requires confirmation in prospective studies.

Project description:BackgroundGiven the high prevalence of neck pain, the neck disability index (NDI) has been used to evaluate patient status and treatment outcomes. Modified versions were proposed as solutions to measurement deficits in the NDI. However, the original 10-item NDI was scored out of 50 and is still the most frequently administered version. Examining the extent of agreement between traditional and Rasch-based versions using Bland-Altman (B&A) plots will inform our understanding of score differences that might rise from using different versions. Therefore, the objective of current study was to describe the extent of agreement between different versions of NDI.MethodsThe current study was a secondary data analysis. The study data was compiled from two prospectively collected data sources. We performed a comprehensive literature search to identify Rasch approved NDI within four databases including Embase, Medline, PubMed, and Google Scholar. Alternate forms and scorings were compared to each other and to the standard NDI. We graphed B&A plots and calculated the mean difference and the 95% limits of agreement (LoA; ±1.96 times the standard deviation).ResultsTwo Rasch approved alternative versions (8 vs 5 items) were identified from 303 screened publications. We analyzed data from 201 (43 males and 158 females) patients attending community clinics for neck pain. We found that the mean difference was approximately 10% of the total score between the 10-item and 5-item (-?4.6 points), whereas the 10-item versus 8-item and 8-item versus 5-item had smaller mean differences (-?2.3 points). The B&A plots displayed wider 95% LoA for the agreement between 10-item and 8-item (LoA: -?12.0, 7.4) and 5-item (LoA: -?14.9, 5.8) compared with the LoA for the 8-item and 5-item (LoA: -?7.8, 3.3).ConclusionTwo Rasch-based NDI solutions (8 vs 5 items) which differ in number of items and conceptual construction are available to provide interval level scoring. They both provide scores that are substantially different from the ordinal NDI, which does not provide interval level scoring. Smaller differences between the two Rasch solutions exist and may relate to the items included. Due to the size and unpredictable nature of the bias between measures, they should not be used interchangeably.

Dataset Information

The reproducibility of psychiatric evaluations of work disability: two reliability and agreement studies.

Publications

The reproducibility of psychiatric evaluations of work disability: two reliability and agreement studies.

Similar Datasets

OmicsDI is part of the ELIXIR infrastructure

Tweets