[Interrater reliability of the criteria-based assessment of criminal responsibility in paraphilic disorders].
Ontology highlight
ABSTRACT: Background
For the assessment of criminal responsibility in the context of sexual delinquency, the evaluation of the severity of a paraphilic disorder and the ability to be controlled are of particular importance. Established assessment criteria are insufficiently operationalized.Aim and methods
The existing criteria were tested by experts of various professional groups with respect to the reliability. For this purpose, 14 experts assessed the criteria of 2 different sets of criteria based on 2 case vignettes (see electronic supplementary material).Results and conclusion
The interrater reliability (IRR) with reference to the criteria of Briken and Müller was higher than the previously established criteria of Boetticher et al. The evaluation of the subjective importance of the criteria of both assessment scales showed that all criteria were rated as at least of average importance. It could thus be useful to initially include both sets of criteria for a greater transparency in the assessment process and to investigate this procedure in further studies.
SUBMITTER: Dobbrunz S
PROVIDER: S-EPMC7808990 | biostudies-literature | 2021 Jan
REPOSITORIES: biostudies-literature
Publications
[Interrater reliability of the criteria-based assessment of criminal responsibility in paraphilic disorders].
Dobbrunz S S Brunner F F Müller J L JL Briken P P
Der Nervenarzt 20210101 1
<h4>Background</h4>For the assessment of criminal responsibility in the context of sexual delinquency, the evaluation of the severity of a paraphilic disorder and the ability to be controlled are of particular importance. Established assessment criteria are insufficiently operationalized.<h4>Aim and methods</h4>The existing criteria were tested by experts of various professional groups with respect to the reliability. For this purpose, 14 experts assessed the criteria of 2 different sets of crit ...[more]
Similar Datasets
Project description:ObjectiveTo evaluate the interrater reliability of the new International Behavioural Variant FTD Criteria Consortium (FTDC) criteria for behavioral variant frontotemporal dementia (bvFTD).MethodsTwenty standardized clinical case modules were developed for patients with a range of neurodegenerative diagnoses, including bvFTD, primary progressive aphasia (nonfluent, semantic, and logopenic variant), Alzheimer disease, and Lewy body dementia. Eighteen blinded raters reviewed the modules and 1) rated the presence or absence of core diagnostic features for the FTDC criteria, and 2) provided an overall diagnostic rating. Interrater reliability was determined by ? statistics for multiple raters with categorical ratings.ResultsThe mean ? value for diagnostic agreement was 0.81 for possible bvFTD and 0.82 for probable bvFTD ("almost perfect agreement"). Interrater reliability for 4 of the 6 core features had "substantial" agreement (behavioral disinhibition, perseverative/compulsive, sympathy/empathy, hyperorality; ? = 0.61-0.80), whereas 2 had "moderate" agreement (apathy/inertia, neuropsychological; ? = 0.41-0.6). Clinician years of experience did not significantly influence rater accuracy.ConclusionsThe FTDC criteria show promise for improving the diagnostic accuracy and reliability of clinicians and researchers. As disease-altering therapies are developed, accurate differential diagnosis between bvFTD and other neurodegenerative diseases will become increasingly important.
| S-EPMC3716343 | biostudies-literature
Project description:PURPOSE:Assess the impact of false-positives (FP), false-negatives (FN), fixation losses (FL), and test duration (TD) on visual field (VF) reliability at different stages of glaucoma severity. DESIGN:Retrospective. PARTICIPANTS:A total of 10?262 VFs from 1538 eyes of 909 subjects with suspect or manifest glaucoma and ?5 VF examinations. METHODS:Predicted mean deviation (MD) was calculated with multilevel modeling of longitudinal data. Differences between predicted and observed MD (?MD) were calculated as a reliability measure. The impact of FP, FN, FL, and TD on ?MD was assessed using multilevel modeling. MAIN OUTCOME MEASURES:?MD associated with a 10% increment in FP, FN, and FL, or a 1-minute increase in TD. RESULTS:FL had little impact on ?MD (<0.2 decibels [dB] per 10% abnormal catch trials), and no level of FL produced ?1 dB of ?MD at any disease stage. FP yielded greater than expected MD, with a 10% increment in abnormal catch trials associated with a ?MD = 0.42, 0.73, and 0.66 dB in mild (MD >-6 dB), moderate (-6 ?MD <-12 dB), and severe (-12 ?MD ?-20 dB) disease, respectively, up to 20% abnormal catch trials, and a ?MD = 1.57, 2.06, and 3.53 dB beyond 20% abnormal catch trials. FNs generally produced observed MDs below expected MDs. FN were minimally impactful up to 20% abnormal catch trials (?MD per 10% increment >-0.14 dB at all levels of severity). Beyond 20% abnormal catch trials, each 10% increment in abnormal catch trials was associated with a ?MD = -1.27, -0.53, and -0.51 dB in mild, moderate, and severe disease, respectively. |?MD| ?1 dB occurred with 22% FP and 26% FN in early, 14% FP and 34% FN in moderate, and 16% FP and 51% FN in severe disease. A 1-minute increment in TD produced ?MDs between -0.35 and -0.40 dB. CONCLUSIONS:FL have little impact on reliability in patients with established glaucoma. FP, and to a lesser extent FNs and TD, significantly affect reliability. The impact of FP and FN varies with disease severity and over the range of abnormal catch trials. On the basis of our findings, we present evidence-based, severity-specific standards for classifying VF reliability for clinical or research applications.
| S-EPMC5675138 | biostudies-literature
Project description:IntroductionClinical clerkships that rely on community preceptors as faculty and assessors are challenged by the variability in their assessments of student performance. This workshop was designed to improve the interrater reliability of preceptors' assessment of student performance.MethodsThis workshop uses a series of videos showing standardized student interactions with standardized patients and a preceptor. The videos feature a borderline satisfactory student-patient encounter and a superior student encounter. For each case, there is a video of the student providing an oral presentation of the encounter to a preceptor. The concept of a plus/delta card is also presented. Participants are asked to note positive elements of the interactions and elements needing improvement on the plus/delta card while watching the videos. Next, participants share their assessment of the students with the group using an audience response system. Group discussion allows for consensus and leads to decreased variation in preceptor assessments of each student's performance. This workshop has been delivered to preceptors in our institution as a 1-hour workshop. Additionally, a version has been presented twice at national medical teaching conferences as a faculty development workshop, instructing faculty how to facilitate the workshop.ResultsParticipant evaluations from the workshop show an increase in preceptors' awareness of specific student behaviors to observe as well as increased confidence with assessing more consistently across various student performances.DiscussionUsing videos with standardized patients, students, and preceptors demonstrating various qualities of performance is effective in improving precision in preceptors' assessment of students in clinical settings.
| S-EPMC6338197 | biostudies-literature
Project description:BackgroundInterrater reliability, aka intercoder reliability, is defined as true agreement between raters, aka coders, without chance agreement. It is used across many disciplines including medical and health research to measure the quality of ratings, coding, diagnoses, or other observations and judgements. While numerous indices of interrater reliability are available, experts disagree on which ones are legitimate or more appropriate. Almost all agree that percent agreement (ao), the oldest and the simplest index, is also the most flawed because it fails to estimate and remove chance agreement, which is produced by raters' random rating. The experts, however, disagree on which chance estimators are legitimate or better. The experts also disagree on which of the three factors, rating category, distribution skew, or task difficulty, an index should rely on to estimate chance agreement, or which factors the known indices in fact rely on. The most popular chance-adjusted indices, according to a functionalist view of mathematical statistics, assume that all raters conduct intentional and maximum random rating while typical raters conduct involuntary and reluctant random rating. The mismatches between the assumed and the actual rater behaviors cause the indices to rely on mistaken factors to estimate chance agreement, leading to the numerous paradoxes, abnormalities, and other misbehaviors of the indices identified by prior studies.MethodsWe conducted a 4 × 8 × 3 between-subject controlled experiment with 4 subjects per cell. Each subject was a rating session with 100 pairs of rating by two raters, totaling 384 rating sessions as the experimental subjects. The experiment tested seven best-known indices of interrater reliability against the observed reliabilities and chance agreements. Impacts of the three factors, i.e., rating category, distribution skew, and task difficulty, on the indices were tested.ResultsThe most criticized index, percent agreement (ao), showed as the most accurate predictor of reliability, reporting directional r2 = .84. It was also the third best approximator, overestimating observed reliability by 13 percentage points on average. The three most acclaimed and most popular indices, Scott's π, Cohen's κ and Krippendorff's α, underperformed all other indices, reporting directional r2 = .312 and underestimated reliability by 31.4 ~ 31.8 points. The newest index, Gwet's AC1, emerged as the second-best predictor and the most accurate approximator. Bennett et al's S ranked behind AC1, and Perreault and Leigh's Ir ranked the fourth both for prediction and approximation. The reliance on category and skew and failure to rely on difficulty explain why the six chance-adjusted indices often underperformed ao, which they were created to outperform. The evidence corroborated the notion that the chance-adjusted indices assume intentional and maximum random rating while the raters instead exhibited involuntary and reluctant random rating.ConclusionThe authors call for more empirical studies and especially more controlled experiments to falsify or qualify this study. If the main findings are replicated and the underlying theories supported, new thinking and new indices may be needed. Index designers may need to refrain from assuming intentional and maximum random rating, and instead assume involuntary and reluctant random rating. Accordingly, the new indices may need to rely on task difficulty, rather than distribution skew or rating category, to estimate chance agreement.
| S-EPMC9426226 | biostudies-literature
Project description:BACKGROUND:The potential for simulation-based learning in neurosurgical training has led the Congress of Neurosurgical Surgeons to develop a series of simulation modules. The Northwestern Objective Microanastomosis Assessment Tool (NOMAT) was created as the corresponding assessment tool for the Congress of Neurosurgical Surgeons Microanastomosis Module. The face and construct validity of the NOMAT have been previously established. OBJECTIVE:To further validate the NOMAT by determining its interrater reliability (IRR) between raters of varying levels of microsurgical expertise. METHODS:The NOMAT was used to assess residents' performance in a microanastomosis simulation module in 2 settings: Northwestern University and the Society of Neurological Surgeons 2014 Boot Camp at the University of Indiana. At Northwestern University, participants were scored by 2 experienced microsurgeons. At the University of Indiana, participants were scored by 2 postdoctoral fellows and an experienced microsurgeon. The IRR of NOMAT was estimated by computing the intraclass correlation coefficient using SPSS v22.0 (IBM, Armonk, New York). RESULTS:A total of 75 residents were assessed. At Northwestern University, 21 residents each performed microanastomosis on 2 model vessels of different sizes, one 3 mm and one 1 mm. At the University of Indiana, 54 residents performed a single microanastomosis procedure on 3-mm vessels. The intraclass correlation coefficient of the total NOMAT scores was 0.88 at Northwestern University and 0.78 at the University of Indiana. CONCLUSION:This study indicates high IRR for the NOMAT. These results suggest that the use of raters with varying levels of expertise does not compromise the precision or validity of the scale. This allows for a wider adoption of the scale and, hence, a greater potential educational impact.
| S-EPMC6312083 | biostudies-other
Project description:Criminal responsibility assessment is undertaken by psychologists or psychiatrists to assess offenders' legal capacities, which vary among countries or regional legislations. There are two psychometric tools (i.e., checklists) validated for criminal responsibility assessment: the Roger Criminal Responsibility Scale, and the rating scale of criminal responsibility for mentally disordered offenders. Despite the existence of psychometric tools structured in clinical vignettes for evaluating legal capacities, none serve the purpose of assessing criminal responsibility. This study aims to validate a novel psychometric tool structured in vignettes for the assessment of criminal responsibility called the "Criminal Responsibility Scale." We applied the tool to 88 defendants referred for criminal responsibility assessment in a forensic medical institute in the city of Rio de Janeiro, Brazil, from December 2017 to December 2018. The validity of the Criminal Responsibility Scale and subscales were evaluated using confirmatory factor analysis. The two-factor solution proved satisfactory and met the needs for practical application of the tool (Kaiser-Meyer-Oklin = 0.82; p < 0.001). Moreover, the inter-rater reliability was evaluated by comparing the tool's final score with that of the expert's conclusion in each case and was found to be satisfactory (k = 0.667-1.0), with a resulting cutoff point of 30.50 (±2) and a Youden index of 0.509. Hence, the Criminal Responsibility Scale is an effective psychometric tool for assessments of criminal responsibility that may encourage future research in assessments of legal capacity with clinical vignette-based psychometric instruments.
| S-EPMC7728607 | biostudies-literature
Project description:The Elements of Morphology Standard Terminology working group published standardized definitions for external ear morphology. The primary objective of our study was to use these descriptions to evaluate the interrater reliability for specific features associated with microtia. We invited six raters from three different subspecialities to rate 100 ear photographs on 32 features. We calculated overall and within specialty and professional experience intraclass correlation coefficients (ICC) and 95% confidence intervals. A total of 600 possible observations were recorded for each feature. The overall interrater reliability ranged from 0.04 (95% CI: 0.00-0.14) for the width of the antihelix inferior crus to 0.93 (95% CI: 0.91-0.95) for the presence of the inferior crux of the antihelix. The reliability for quantitative characteristics such as length or width of an ear structure was generally lower than the reliability for qualitative characteristics (e.g., presence or absence of an ear structure). Categories with very poor interrater reliability included anti-helix inferior crux width (0.04, 95% CI: 0.00-0.14), crux helix extension (0.17, 95% CI 0.00-0.37), and shape of the incisura (0.14, 95% CI: 0.01-0.27). There were no significant differences in reliability estimates by specialty or professional experience for most variables. Our study showed that it is feasible to systematically characterize many of structures of the ear that are affected in microtia. We incorporated these descriptions into a standardized phenotypic assessment tool (PAT-Microtia) that might be used in multicenter research studies to identify sub-phenotypes for future studies of microtia.
| S-EPMC3664118 | biostudies-literature
Project description:The current study evaluated the interrater reliability of the Child and Adolescent Services Assessment (CASA), a widely used structured interview measuring pediatric mental health service use. Interviews (N = 72) were randomly selected from a pediatric effectiveness trial, and audio was coded by an independent rater. Regressions were employed to identify predictors of rater disagreement. Interrater reliability was high for items (> 94%) and summary metrics (ICC > .79) across service sectors. Predictors of disagreement varied by domain; significant predictors indexed higher clinical severity or social disadvantage. Results support the CASA as a reliable and robust assessment of pediatric service use, but administrators should be alert when assessing vulnerable populations.
| S-EPMC6193845 | biostudies-literature
Project description:BackgroundLearning strategies are thinking tools that students can use to actively acquire information. Examples of learning strategies include mnemonics, charts, and maps. One strategy that may help students master the tsunami of information presented in medical school is the mind map learning strategy. Currently, there is no valid and reliable rubric to grade mind maps and this may contribute to their underutilization in medicine. Because concept maps and mind maps engage learners similarly at a metacognitive level, a valid and reliable concept map assessment scoring system was adapted to form the mind map assessment rubric (MMAR). The MMAR can assess mind map depth based upon concept-links, cross-links, hierarchies, examples, pictures, and colors. The purpose of this study was to examine interrater reliability of the MMAR.MethodsThis exploratory study was conducted at a US medical school as part of a larger investigation on learning strategies. Sixty-six (N = 66) first-year medical students were given a 394-word text passage followed by a 30-minute presentation on mind mapping. After the presentation, subjects were again given the text passage and instructed to create mind maps based upon the passage. The mind maps were collected and independently scored using the MMAR by 3 examiners. Interrater reliability was measured using the intraclass correlation coefficient (ICC) statistic. Statistics were calculated using SPSS version 12.0 (Chicago, IL).ResultsAnalysis of the mind maps revealed the following: concept-links ICC = .05 (95% CI, -.42 to .38), cross-links ICC = .58 (95% CI, .37 to .73), hierarchies ICC = .23 (95% CI, -.15 to .50), examples ICC = .53 (95% CI, .29 to .69), pictures ICC = .86 (95% CI, .79 to .91), colors ICC = .73 (95% CI, .59 to .82), and total score ICC = .86 (95% CI, .79 to .91).ConclusionThe high ICC value for total mind map score indicates strong MMAR interrater reliability. Pictures and colors demonstrated moderate to strong interrater reliability. We conclude that the MMAR may be a valid and reliable tool to assess mind maps in medicine. However, further research on the validity and reliability of the MMAR is necessary.
| S-EPMC2683832 | biostudies-literature
Project description:Nowadays, the clinical forensic medical management of migration flows comprises the age assessment of unaccompanied minors. The process of age estimation is a fundamental pillar for legally ensuring the minors' rights and their protection needs. The procedure is complex and involves different phases and actors, from medical doctors to law enforcement officers. The present study aimed to investigate the performance of Greulich and Pyle, Demirjian, and Mincer methods when performed by raters both trained and without training. Also, the interrater reliability within groups of raters from different areas of expertise was evaluated. A total of 36 participants were enrolled for this study, divided in two groups according to their level of experience with age estimation methods. Each rater was asked to allocate stages and standards for age assessment, evaluating ten orthopantomograms and ten hand-wrist roentgenograms. The interrater reliability expressed through the Fleiss Kappa coefficient and the agreement with the reference standard were calculated. The results showed that none of the categories analyzed could reach a good interrater reliability (ĸ > 0.8) for both methods. The study results highlighted variation and disagreement in the interpretation of the sample among raters and in the subsequent stages and standards allocation. In conclusion, the results of this study highlight that expertise does influence the reliability of the most utilized methods of age estimation of living individuals and stress the importance of proper training and practice, which could greatly increase the accuracy of age assessments.
| S-EPMC8813704 | biostudies-literature