The Child and Adolescent Services Assessment: Interrater Reliability and Predictors of Rater Disagreement.
Ontology highlight
ABSTRACT:
SUBMITTER: Schwartz KTG
PROVIDER: S-EPMC6193845 | biostudies-literature | 2018 Nov
REPOSITORIES: biostudies-literature
Publications
The Child and Adolescent Services Assessment: Interrater Reliability and Predictors of Rater Disagreement.
Schwartz Karen T G KTG Bowling Amanda A AA Dickerson John F JF Lynch Frances L FL Brent David A DA Porta Giovanna G Iyengar Satish S Weersing V Robin VR
Administration and policy in mental health 20181101 6
The current study evaluated the interrater reliability of the Child and Adolescent Services Assessment (CASA), a widely used structured interview measuring pediatric mental health service use. Interviews (N = 72) were randomly selected from a pediatric effectiveness trial, and audio was coded by an independent rater. Regressions were employed to identify predictors of rater disagreement. Interrater reliability was high for items (> 94%) and summary metrics (ICC > .79) across service sectors. Pre ...[more]
Similar Datasets
Project description:BackgroundPatients with upper limb action tremor frequently exhibit additional neurological signs of uncertain significance. Clinicians vary in their interpretation, and interrater agreement on the final diagnosis is poor.ObjectivesA new clinical tool for assessing the presence or absence of clinical signs that are important in axis-1 classification of tremor patients is introduced: the Standardized Tremor Elements Assessment (STEA). Interrater agreement is determined, and signs leading to disagreement in the final diagnosis are identified.MethodsThree tremor-focussed and one dystonia-focussed movement disorder specialists rated 59 videos of patients with upper limb action tremor syndromes using STEA. Interrater agreements for final diagnosis and STEA items were calculated.ResultsInterrater agreement regarding the final diagnosis was higher within the group of tremor specialists and poor between dystonia and tremor specialists. Greater agreement was found for items characterizing tremor than for signs of dystonia.ConclusionsClinical signs leading to diagnostic disagreement were identified with STEA, and STEA should therefore be useful in future studies of diagnostic disagreement. The thresholds for considering neurological signs as soft versus significant for ataxia, parkinsonism, dystonia, etc. are critically important in tremor classification and must be studied across movement disorder subspecialties, not simply within a pool of tremor specialists.
| S-EPMC8015892 | biostudies-literature
Project description:IntroductionClinical clerkships that rely on community preceptors as faculty and assessors are challenged by the variability in their assessments of student performance. This workshop was designed to improve the interrater reliability of preceptors' assessment of student performance.MethodsThis workshop uses a series of videos showing standardized student interactions with standardized patients and a preceptor. The videos feature a borderline satisfactory student-patient encounter and a superior student encounter. For each case, there is a video of the student providing an oral presentation of the encounter to a preceptor. The concept of a plus/delta card is also presented. Participants are asked to note positive elements of the interactions and elements needing improvement on the plus/delta card while watching the videos. Next, participants share their assessment of the students with the group using an audience response system. Group discussion allows for consensus and leads to decreased variation in preceptor assessments of each student's performance. This workshop has been delivered to preceptors in our institution as a 1-hour workshop. Additionally, a version has been presented twice at national medical teaching conferences as a faculty development workshop, instructing faculty how to facilitate the workshop.ResultsParticipant evaluations from the workshop show an increase in preceptors' awareness of specific student behaviors to observe as well as increased confidence with assessing more consistently across various student performances.DiscussionUsing videos with standardized patients, students, and preceptors demonstrating various qualities of performance is effective in improving precision in preceptors' assessment of students in clinical settings.
| S-EPMC6338197 | biostudies-literature
Project description:BackgroundInterrater reliability, aka intercoder reliability, is defined as true agreement between raters, aka coders, without chance agreement. It is used across many disciplines including medical and health research to measure the quality of ratings, coding, diagnoses, or other observations and judgements. While numerous indices of interrater reliability are available, experts disagree on which ones are legitimate or more appropriate. Almost all agree that percent agreement (ao), the oldest and the simplest index, is also the most flawed because it fails to estimate and remove chance agreement, which is produced by raters' random rating. The experts, however, disagree on which chance estimators are legitimate or better. The experts also disagree on which of the three factors, rating category, distribution skew, or task difficulty, an index should rely on to estimate chance agreement, or which factors the known indices in fact rely on. The most popular chance-adjusted indices, according to a functionalist view of mathematical statistics, assume that all raters conduct intentional and maximum random rating while typical raters conduct involuntary and reluctant random rating. The mismatches between the assumed and the actual rater behaviors cause the indices to rely on mistaken factors to estimate chance agreement, leading to the numerous paradoxes, abnormalities, and other misbehaviors of the indices identified by prior studies.MethodsWe conducted a 4 × 8 × 3 between-subject controlled experiment with 4 subjects per cell. Each subject was a rating session with 100 pairs of rating by two raters, totaling 384 rating sessions as the experimental subjects. The experiment tested seven best-known indices of interrater reliability against the observed reliabilities and chance agreements. Impacts of the three factors, i.e., rating category, distribution skew, and task difficulty, on the indices were tested.ResultsThe most criticized index, percent agreement (ao), showed as the most accurate predictor of reliability, reporting directional r2 = .84. It was also the third best approximator, overestimating observed reliability by 13 percentage points on average. The three most acclaimed and most popular indices, Scott's π, Cohen's κ and Krippendorff's α, underperformed all other indices, reporting directional r2 = .312 and underestimated reliability by 31.4 ~ 31.8 points. The newest index, Gwet's AC1, emerged as the second-best predictor and the most accurate approximator. Bennett et al's S ranked behind AC1, and Perreault and Leigh's Ir ranked the fourth both for prediction and approximation. The reliance on category and skew and failure to rely on difficulty explain why the six chance-adjusted indices often underperformed ao, which they were created to outperform. The evidence corroborated the notion that the chance-adjusted indices assume intentional and maximum random rating while the raters instead exhibited involuntary and reluctant random rating.ConclusionThe authors call for more empirical studies and especially more controlled experiments to falsify or qualify this study. If the main findings are replicated and the underlying theories supported, new thinking and new indices may be needed. Index designers may need to refrain from assuming intentional and maximum random rating, and instead assume involuntary and reluctant random rating. Accordingly, the new indices may need to rely on task difficulty, rather than distribution skew or rating category, to estimate chance agreement.
| S-EPMC9426226 | biostudies-literature
Project description:BACKGROUND:The potential for simulation-based learning in neurosurgical training has led the Congress of Neurosurgical Surgeons to develop a series of simulation modules. The Northwestern Objective Microanastomosis Assessment Tool (NOMAT) was created as the corresponding assessment tool for the Congress of Neurosurgical Surgeons Microanastomosis Module. The face and construct validity of the NOMAT have been previously established. OBJECTIVE:To further validate the NOMAT by determining its interrater reliability (IRR) between raters of varying levels of microsurgical expertise. METHODS:The NOMAT was used to assess residents' performance in a microanastomosis simulation module in 2 settings: Northwestern University and the Society of Neurological Surgeons 2014 Boot Camp at the University of Indiana. At Northwestern University, participants were scored by 2 experienced microsurgeons. At the University of Indiana, participants were scored by 2 postdoctoral fellows and an experienced microsurgeon. The IRR of NOMAT was estimated by computing the intraclass correlation coefficient using SPSS v22.0 (IBM, Armonk, New York). RESULTS:A total of 75 residents were assessed. At Northwestern University, 21 residents each performed microanastomosis on 2 model vessels of different sizes, one 3 mm and one 1 mm. At the University of Indiana, 54 residents performed a single microanastomosis procedure on 3-mm vessels. The intraclass correlation coefficient of the total NOMAT scores was 0.88 at Northwestern University and 0.78 at the University of Indiana. CONCLUSION:This study indicates high IRR for the NOMAT. These results suggest that the use of raters with varying levels of expertise does not compromise the precision or validity of the scale. This allows for a wider adoption of the scale and, hence, a greater potential educational impact.
| S-EPMC6312083 | biostudies-other
Project description:The Elements of Morphology Standard Terminology working group published standardized definitions for external ear morphology. The primary objective of our study was to use these descriptions to evaluate the interrater reliability for specific features associated with microtia. We invited six raters from three different subspecialities to rate 100 ear photographs on 32 features. We calculated overall and within specialty and professional experience intraclass correlation coefficients (ICC) and 95% confidence intervals. A total of 600 possible observations were recorded for each feature. The overall interrater reliability ranged from 0.04 (95% CI: 0.00-0.14) for the width of the antihelix inferior crus to 0.93 (95% CI: 0.91-0.95) for the presence of the inferior crux of the antihelix. The reliability for quantitative characteristics such as length or width of an ear structure was generally lower than the reliability for qualitative characteristics (e.g., presence or absence of an ear structure). Categories with very poor interrater reliability included anti-helix inferior crux width (0.04, 95% CI: 0.00-0.14), crux helix extension (0.17, 95% CI 0.00-0.37), and shape of the incisura (0.14, 95% CI: 0.01-0.27). There were no significant differences in reliability estimates by specialty or professional experience for most variables. Our study showed that it is feasible to systematically characterize many of structures of the ear that are affected in microtia. We incorporated these descriptions into a standardized phenotypic assessment tool (PAT-Microtia) that might be used in multicenter research studies to identify sub-phenotypes for future studies of microtia.
| S-EPMC3664118 | biostudies-literature
Project description:BackgroundFor the assessment of criminal responsibility in the context of sexual delinquency, the evaluation of the severity of a paraphilic disorder and the ability to be controlled are of particular importance. Established assessment criteria are insufficiently operationalized.Aim and methodsThe existing criteria were tested by experts of various professional groups with respect to the reliability. For this purpose, 14 experts assessed the criteria of 2 different sets of criteria based on 2 case vignettes (see electronic supplementary material).Results and conclusionThe interrater reliability (IRR) with reference to the criteria of Briken and Müller was higher than the previously established criteria of Boetticher et al. The evaluation of the subjective importance of the criteria of both assessment scales showed that all criteria were rated as at least of average importance. It could thus be useful to initially include both sets of criteria for a greater transparency in the assessment process and to investigate this procedure in further studies.
| S-EPMC7808990 | biostudies-literature
Project description:BackgroundLearning strategies are thinking tools that students can use to actively acquire information. Examples of learning strategies include mnemonics, charts, and maps. One strategy that may help students master the tsunami of information presented in medical school is the mind map learning strategy. Currently, there is no valid and reliable rubric to grade mind maps and this may contribute to their underutilization in medicine. Because concept maps and mind maps engage learners similarly at a metacognitive level, a valid and reliable concept map assessment scoring system was adapted to form the mind map assessment rubric (MMAR). The MMAR can assess mind map depth based upon concept-links, cross-links, hierarchies, examples, pictures, and colors. The purpose of this study was to examine interrater reliability of the MMAR.MethodsThis exploratory study was conducted at a US medical school as part of a larger investigation on learning strategies. Sixty-six (N = 66) first-year medical students were given a 394-word text passage followed by a 30-minute presentation on mind mapping. After the presentation, subjects were again given the text passage and instructed to create mind maps based upon the passage. The mind maps were collected and independently scored using the MMAR by 3 examiners. Interrater reliability was measured using the intraclass correlation coefficient (ICC) statistic. Statistics were calculated using SPSS version 12.0 (Chicago, IL).ResultsAnalysis of the mind maps revealed the following: concept-links ICC = .05 (95% CI, -.42 to .38), cross-links ICC = .58 (95% CI, .37 to .73), hierarchies ICC = .23 (95% CI, -.15 to .50), examples ICC = .53 (95% CI, .29 to .69), pictures ICC = .86 (95% CI, .79 to .91), colors ICC = .73 (95% CI, .59 to .82), and total score ICC = .86 (95% CI, .79 to .91).ConclusionThe high ICC value for total mind map score indicates strong MMAR interrater reliability. Pictures and colors demonstrated moderate to strong interrater reliability. We conclude that the MMAR may be a valid and reliable tool to assess mind maps in medicine. However, further research on the validity and reliability of the MMAR is necessary.
| S-EPMC2683832 | biostudies-literature
Project description:BackgroundAdolescence is a developmental period during which youth experience vulnerability to psychopathology. To build the foundation for a parsimonious psychopathology risk calculator while capturing the complexity and dynamic nature of the environment, the current study aimed to identify distinct risk and resilience profiles with a wide range of environmental factors guided by Bronfenbrenner's biopsychosocial ecological system theory. The association between the early-mid childhood risk profiles and psychopathology in adolescence were examined. Moreover, the predictive utility of early childhood irritability was evaluated in addition to the risk profiles.MethodsThe data from Future of Families and Child Wellbeing Study a nation-wide longitudinal study, were used in the latent profile analyses to identify the risk profiles with family, school, and neighborhood characteristics from 3 to 9 years old. To capture the socio-environmental and cultural nuances, we extracted three subsamples, including Black/African American (n = 2587), Hispanic/Latinx (n = 1577), and White (n = 776) for separate analyses. Risk profile memberships were used to predict adolescence psychopathology, including depression, anxiety, attention deficits, oppositional defiant disorder, and conduct disorder symptoms. The predictive utility of early childhood irritability above and beyond environmental risk profiles was evaluated using stepwise regression.ResultsThree risk profiles were identified in the Hispanic/Latinx and Black/African American subsamples, while four profiles were identified in White subsample. Almost all risk profile membership predicted both internalizing and externalizing psychopathology, while some profiles are predictive of externalizing symptoms only. Higher level of irritability predicted higher symptomatology in all five mental health outcomes above and beyond the environmental profiles.ConclusionsDistinct risk and resilience profiles primarily driven by parent and family characteristics were identified for all three major race/ethnicity groups. Our findings lay the foundation for a more efficient multi-tiered information gathering process in mental health clinical settings to aid the decision making for intervention and prevention.
| S-EPMC10694535 | biostudies-literature
Project description:AimTo study the intra-rater reliability and feasibility of the HexCom complexity assessment model by analyzing internal consistency, intra-rater reliability and response time.DesignTest-retest study with a selection of 11 clinical situations that cover the full scope of situations assessed by the HexCom model and which are responded to individually.LocationHome care, primary care, hospital and sociosanitary care. Two specialized palliative home care teams (PADES).ParticipantsA total of 20 professionals comprising 10 experts in palliative home care (PADES) and 10 professionals from general palliative care (primary care, hospital and sociosanitary care). These professionals came from the fields of family medicine (5), internal medicine (2), geriatrics (2), nursing (9), psychology (1) and social work (1).Main measurementsCronbach's alpha, weighted kappa, response time.ResultsCronbach's alpha of 0.91 for HexCom-Red and 0.87 for HexCom-Clin. Intra-rater reliability ranging from good to very good for HexCom-Red (kappa: 0.78-1) and from moderate to very good for HexCom-Clin (kappa: 0.58-0.91). Average response time of 0:57 for HexCom-Red and 3:80min for HexComClin.ConclusionsHexCom-Red and HexCom-Clin are reliable tools and feasible for use by all professionals involved in both general and specialized palliative care at different levels.
| S-EPMC9108442 | biostudies-literature
Project description:Introduction: To ensure the quality of clinical trial safety data, universal data standards are required. In 2019 the International Neonatal Consortium (INC) published a neonatal adverse event severity scale (NAESS) to standardize the reporting of adverse event (AE) severity. In this study the reliability of AE severity grading with INC NAESS was prospectively assessed in a real-world setting. Methods: Severity of AEs was assessed by two independent observers at each of four centers across the world. In each center two series of 30 neonatal adverse events were assessed by both observers: in a first phase with a generic (Common Terminology Criteria for Adverse Events, CTCAE) severity scale not specific to neonates, and in a second phase with INC NAESS (after a structured training). Intraclass correlation coefficients (ICC) were calculated to express inter-rater agreement in both phases, and bootstrap sampling was used to compare them. Results: 120 AEs were included in each of both phases. The ICC with the use of INC NAESS in phase 2 was 0.69. This represents a significant but modest improvement in comparison to the initial ICC of 0.66 in phase 1 (confidence interval of ratio of ICC in phase 2 to phase 1 = 1.005-1.146; excludes 1). The ICC was higher for those AEs for which a diagnosis specific AE severity table was available in INC NAESS (ICC 0.80). Discussion: Good inter-rater reliability of the INC NAESS was demonstrated in four neonatal intensive care units (NICUs) across the globe. The ICC is comparable to what is reported for scales with similar purposes in different populations. There is a modest, but significant, improvement in inter-rater agreement in comparison to the naïve phase without INC NAESS. The better performance when reviewers use AE-specific NAESS tables highlights the need to expand the number of AEs that are covered by specific criteria in the current version of INC NAESS.
| S-EPMC10512550 | biostudies-literature