The Child and Adolescent Services Assessment: Interrater Reliability and Predictors of Rater Disagreement.
Ontology highlight
ABSTRACT:
SUBMITTER: Schwartz KTG
PROVIDER: S-EPMC6193845 | biostudies-literature | 2018 Nov
REPOSITORIES: biostudies-literature
Publications
The Child and Adolescent Services Assessment: Interrater Reliability and Predictors of Rater Disagreement.
Schwartz Karen T G KTG Bowling Amanda A AA Dickerson John F JF Lynch Frances L FL Brent David A DA Porta Giovanna G Iyengar Satish S Weersing V Robin VR
Administration and policy in mental health 20181101 6
The current study evaluated the interrater reliability of the Child and Adolescent Services Assessment (CASA), a widely used structured interview measuring pediatric mental health service use. Interviews (N = 72) were randomly selected from a pediatric effectiveness trial, and audio was coded by an independent rater. Regressions were employed to identify predictors of rater disagreement. Interrater reliability was high for items (> 94%) and summary metrics (ICC > .79) across service sectors. Pre ...[more]
Similar Datasets
Project description:BackgroundPatients with upper limb action tremor frequently exhibit additional neurological signs of uncertain significance. Clinicians vary in their interpretation, and interrater agreement on the final diagnosis is poor.ObjectivesA new clinical tool for assessing the presence or absence of clinical signs that are important in axis-1 classification of tremor patients is introduced: the Standardized Tremor Elements Assessment (STEA). Interrater agreement is determined, and signs leading to disagreement in the final diagnosis are identified.MethodsThree tremor-focussed and one dystonia-focussed movement disorder specialists rated 59 videos of patients with upper limb action tremor syndromes using STEA. Interrater agreements for final diagnosis and STEA items were calculated.ResultsInterrater agreement regarding the final diagnosis was higher within the group of tremor specialists and poor between dystonia and tremor specialists. Greater agreement was found for items characterizing tremor than for signs of dystonia.ConclusionsClinical signs leading to diagnostic disagreement were identified with STEA, and STEA should therefore be useful in future studies of diagnostic disagreement. The thresholds for considering neurological signs as soft versus significant for ataxia, parkinsonism, dystonia, etc. are critically important in tremor classification and must be studied across movement disorder subspecialties, not simply within a pool of tremor specialists.
| S-EPMC8015892 | biostudies-literature
Project description:IntroductionClinical clerkships that rely on community preceptors as faculty and assessors are challenged by the variability in their assessments of student performance. This workshop was designed to improve the interrater reliability of preceptors' assessment of student performance.MethodsThis workshop uses a series of videos showing standardized student interactions with standardized patients and a preceptor. The videos feature a borderline satisfactory student-patient encounter and a superior student encounter. For each case, there is a video of the student providing an oral presentation of the encounter to a preceptor. The concept of a plus/delta card is also presented. Participants are asked to note positive elements of the interactions and elements needing improvement on the plus/delta card while watching the videos. Next, participants share their assessment of the students with the group using an audience response system. Group discussion allows for consensus and leads to decreased variation in preceptor assessments of each student's performance. This workshop has been delivered to preceptors in our institution as a 1-hour workshop. Additionally, a version has been presented twice at national medical teaching conferences as a faculty development workshop, instructing faculty how to facilitate the workshop.ResultsParticipant evaluations from the workshop show an increase in preceptors' awareness of specific student behaviors to observe as well as increased confidence with assessing more consistently across various student performances.DiscussionUsing videos with standardized patients, students, and preceptors demonstrating various qualities of performance is effective in improving precision in preceptors' assessment of students in clinical settings.
| S-EPMC6338197 | biostudies-literature
Project description:BACKGROUND:The potential for simulation-based learning in neurosurgical training has led the Congress of Neurosurgical Surgeons to develop a series of simulation modules. The Northwestern Objective Microanastomosis Assessment Tool (NOMAT) was created as the corresponding assessment tool for the Congress of Neurosurgical Surgeons Microanastomosis Module. The face and construct validity of the NOMAT have been previously established. OBJECTIVE:To further validate the NOMAT by determining its interrater reliability (IRR) between raters of varying levels of microsurgical expertise. METHODS:The NOMAT was used to assess residents' performance in a microanastomosis simulation module in 2 settings: Northwestern University and the Society of Neurological Surgeons 2014 Boot Camp at the University of Indiana. At Northwestern University, participants were scored by 2 experienced microsurgeons. At the University of Indiana, participants were scored by 2 postdoctoral fellows and an experienced microsurgeon. The IRR of NOMAT was estimated by computing the intraclass correlation coefficient using SPSS v22.0 (IBM, Armonk, New York). RESULTS:A total of 75 residents were assessed. At Northwestern University, 21 residents each performed microanastomosis on 2 model vessels of different sizes, one 3 mm and one 1 mm. At the University of Indiana, 54 residents performed a single microanastomosis procedure on 3-mm vessels. The intraclass correlation coefficient of the total NOMAT scores was 0.88 at Northwestern University and 0.78 at the University of Indiana. CONCLUSION:This study indicates high IRR for the NOMAT. These results suggest that the use of raters with varying levels of expertise does not compromise the precision or validity of the scale. This allows for a wider adoption of the scale and, hence, a greater potential educational impact.
| S-EPMC6312083 | biostudies-other
Project description:The Elements of Morphology Standard Terminology working group published standardized definitions for external ear morphology. The primary objective of our study was to use these descriptions to evaluate the interrater reliability for specific features associated with microtia. We invited six raters from three different subspecialities to rate 100 ear photographs on 32 features. We calculated overall and within specialty and professional experience intraclass correlation coefficients (ICC) and 95% confidence intervals. A total of 600 possible observations were recorded for each feature. The overall interrater reliability ranged from 0.04 (95% CI: 0.00-0.14) for the width of the antihelix inferior crus to 0.93 (95% CI: 0.91-0.95) for the presence of the inferior crux of the antihelix. The reliability for quantitative characteristics such as length or width of an ear structure was generally lower than the reliability for qualitative characteristics (e.g., presence or absence of an ear structure). Categories with very poor interrater reliability included anti-helix inferior crux width (0.04, 95% CI: 0.00-0.14), crux helix extension (0.17, 95% CI 0.00-0.37), and shape of the incisura (0.14, 95% CI: 0.01-0.27). There were no significant differences in reliability estimates by specialty or professional experience for most variables. Our study showed that it is feasible to systematically characterize many of structures of the ear that are affected in microtia. We incorporated these descriptions into a standardized phenotypic assessment tool (PAT-Microtia) that might be used in multicenter research studies to identify sub-phenotypes for future studies of microtia.
| S-EPMC3664118 | biostudies-literature
Project description:BackgroundFor the assessment of criminal responsibility in the context of sexual delinquency, the evaluation of the severity of a paraphilic disorder and the ability to be controlled are of particular importance. Established assessment criteria are insufficiently operationalized.Aim and methodsThe existing criteria were tested by experts of various professional groups with respect to the reliability. For this purpose, 14 experts assessed the criteria of 2 different sets of criteria based on 2 case vignettes (see electronic supplementary material).Results and conclusionThe interrater reliability (IRR) with reference to the criteria of Briken and Müller was higher than the previously established criteria of Boetticher et al. The evaluation of the subjective importance of the criteria of both assessment scales showed that all criteria were rated as at least of average importance. It could thus be useful to initially include both sets of criteria for a greater transparency in the assessment process and to investigate this procedure in further studies.
| S-EPMC7808990 | biostudies-literature
Project description:BackgroundLearning strategies are thinking tools that students can use to actively acquire information. Examples of learning strategies include mnemonics, charts, and maps. One strategy that may help students master the tsunami of information presented in medical school is the mind map learning strategy. Currently, there is no valid and reliable rubric to grade mind maps and this may contribute to their underutilization in medicine. Because concept maps and mind maps engage learners similarly at a metacognitive level, a valid and reliable concept map assessment scoring system was adapted to form the mind map assessment rubric (MMAR). The MMAR can assess mind map depth based upon concept-links, cross-links, hierarchies, examples, pictures, and colors. The purpose of this study was to examine interrater reliability of the MMAR.MethodsThis exploratory study was conducted at a US medical school as part of a larger investigation on learning strategies. Sixty-six (N = 66) first-year medical students were given a 394-word text passage followed by a 30-minute presentation on mind mapping. After the presentation, subjects were again given the text passage and instructed to create mind maps based upon the passage. The mind maps were collected and independently scored using the MMAR by 3 examiners. Interrater reliability was measured using the intraclass correlation coefficient (ICC) statistic. Statistics were calculated using SPSS version 12.0 (Chicago, IL).ResultsAnalysis of the mind maps revealed the following: concept-links ICC = .05 (95% CI, -.42 to .38), cross-links ICC = .58 (95% CI, .37 to .73), hierarchies ICC = .23 (95% CI, -.15 to .50), examples ICC = .53 (95% CI, .29 to .69), pictures ICC = .86 (95% CI, .79 to .91), colors ICC = .73 (95% CI, .59 to .82), and total score ICC = .86 (95% CI, .79 to .91).ConclusionThe high ICC value for total mind map score indicates strong MMAR interrater reliability. Pictures and colors demonstrated moderate to strong interrater reliability. We conclude that the MMAR may be a valid and reliable tool to assess mind maps in medicine. However, further research on the validity and reliability of the MMAR is necessary.
| S-EPMC2683832 | biostudies-literature
Project description:ObjectiveParanoia may be particularly prevalent during adolescence, building on the heightened social vulnerabilities at this age. Excessive mistrust may be corrosive for adolescent social relationships, especially in the context of mental health disorders. We set out to examine the prevalence, symptom associations, and persistence of paranoia in a cohort of young people attending child and adolescent mental health services.MethodA total of 301 patients (11-17 years old) completed measures of paranoia, affect, peer difficulties and behavioural problems. Clinicians also rated each participant's psychiatric symptoms. Patterns of association were examined using linear regressions and network analyses. In total, 105 patients repeated the measures several months later.ResultsMost of the adolescents had affective disorders (n = 195), self-harm/suicidality (n = 82), or neurodevelopmental conditions (n = 125). Few had suspected psychosis (n = 7). Rates of paranoia were approximately double compared with previous reports from the general population. In this patient sample, 35% had at least elevated paranoia, 15% had at least moderate paranoia, and 6% had high paranoia. Paranoia had moderate associations with clinician-rated peer difficulties, self-harm, and trauma, and small associations with clinician-rated social anxiety, depression, generalised anxiety, and educational problems. Network analyses showed paranoia had the strongest unique relationship with peer difficulties. Paths from peer difficulties to anxiety, self-harm, post-traumatic stress disorder symptoms, and behavioural problems were all via paranoia. Both self-harm and post-traumatic stress disorder were solely associated with paranoia in the network. Paranoia remained persistent for three-quarters and was associated with greater psychological problems over time.ConclusionParanoia is relatively common and persistent across a range of clinical presentations in youth. When paranoia occurs alongside emotional problems, important peer interactions may be adversely affected. Wider consideration of paranoia in adolescent patients is needed.
| S-EPMC8649424 | biostudies-literature
Project description:AimTo develop and test a new adverse drug reaction (ADR) causality assessment tool (CAT).MethodsA comparison between seven assessors of a new CAT, formulated by an expert focus group, compared with the Naranjo CAT in 80 cases from a prospective observational study and 37 published ADR case reports (819 causality assessments in total).Main outcome measuresUtilisation of causality categories, measure of disagreements, inter-rater reliability (IRR).ResultsThe Liverpool ADR CAT, using 40 cases from an observational study, showed causality categories of 1 unlikely, 62 possible, 92 probable and 125 definite (1, 62, 92, 125) and 'moderate' IRR (kappa 0.48), compared to Naranjo (0, 100, 172, 8) with 'moderate' IRR (kappa 0.45). In a further 40 cases, the Liverpool tool (0, 66, 81, 133) showed 'good' IRR (kappa 0.6) while Naranjo (1, 90, 185, 4) remained 'moderate'.ConclusionThe Liverpool tool assigns the full range of causality categories and shows good IRR. Further assessment by different investigators in different settings is needed to fully assess the utility of this tool.
| S-EPMC3237416 | biostudies-literature
Project description:BackgroundThere is a scarcity of evaluated tools to assess whether non-specialist providers achieve minimum levels of competency to effectively and safely deliver psychological interventions in low- and middle-income countries. The objective of this study was to evaluate the reliability and utility of the newly developed Working with children - Assessment of Competencies Tool (WeACT) to assess service providers' competencies in Gaza, Palestine.MethodsThe study evaluated; (1) psychometric properties of the WeACT based on observed role-plays by trainers/supervisors (N = 8); (2) sensitivity to change among service provider competencies (N = 25) using pre-and-post training WeACT scores on standardized role-plays; (3) in-service competencies among experienced service providers (N = 64) using standardized role-plays.ResultsWe demonstrated moderate interrater reliability [intraclass correlation coefficient, single measures, ICC = 0.68 (95% CI 0.48-0.86)] after practice, with high internal consistency (α = 0.94). WeACT assessments provided clinically relevant information on achieved levels of competencies (55% of the competencies were scored as adequate pre-training; 71% post-training; 62% in-service). Pre-post training assessment saw significant improvement in competencies (W = -3.64; p < 0.001).ConclusionThis study demonstrated positive results on the reliability and utility of the WeACT, with sufficient inter-rater agreement, excellent internal consistency, sensitivity to assess change, and providing insight needs for remedial training. The WeACT holds promise as a tool for monitoring quality of care when implementing evidence-based care at scale.
| S-EPMC8127642 | biostudies-literature
Project description:Schroth exercises for scoliosis are prescribed based on curve types. This study aimed to determine the reliability of an algorithm for classifying Schroth curve types. Forty-four consecutive volunteers with adolescent idiopathic scoliosis, 10 to 18 years old, with curves 10° to 50°, were recruited from a scoliosis clinic. Their standing posture and Adam's bending test were videotaped. Ten consecutive Schroth therapist volunteers from an international registry independently classified the curve types using the proposed classification algorithm. Videos were rated twice at least seven days apart. Reliability was calculated using the Gwet's AC1 agreement coefficient for all the raters and for subgroups reporting full understanding (well-trained) and with prior algorithm experience. The intra-rater and weighted agreement coefficients for all the raters were 0.64 (95% CI: 0.53-0.73) and 0.75 (0.63-0.84), respectively. For the well-trained raters, they were 0.70 (0.60-0.78) and 0.82 (0.73-0.88), respectively; for the experienced raters, they were 0.81 (0.77-0.85) and 0.89 (0.80-0.94), respectively. The inter-rater versus weighted agreement coefficients for all the raters were 0.43 (0.28-0.58) versus 0.48 (0.29-0.67). For the well-trained raters, they were 0.50 (0.38-0.61) versus 0.61 (0.49-0.72), and for the experienced raters, they were 0.67 (0.50-0.85) versus 0.79 (0.64-0.94). Full understanding and experience led to higher reliability. Use of the algorithm can help standardize Schroth exercise treatment.
| S-EPMC10047876 | biostudies-literature