Electroencephalography Technologist Inter-rater Agreement and Interpretation of Pediatric Critical Care Electroencephalography.
Ontology highlight
ABSTRACT: Objectives
Electroencephalography (EEG) technologists commonly screen continuous EEG. Until now, the inter-rater agreement or sensitivity for important EEG findings has been unknown in this group.Methods
Twenty-nine EEG technologists and three clinical neurophysiologists interpreted 90 five-minute samples of pediatric critical care EEG. Inter-rater agreement was examined with Cohen's kappa and Fleiss' kappa for EEG findings. A gold-standard consensus agreement was developed for examining sensitivity and specificity for seizures or discontinuity. Kruskal-Wallis tests with Benjamani-Hochberg corrections for multiple comparisons were utilized to examine associations between correct scoring and certification status and years of experience.Results
Aggregate agreement was moderate for seizures and fair for EEG background continuity among EEG technologists. Individual agreement for seizures and continuity varied from slight to substantial. For individual EEG technologists, sensitivity for seizures ranged from 44 to 93% and sensitivity for continuity ranged from 81 to 100%. Raters with Certified Long Term Monitoring credentials were more likely to identify seizures correctly.Significance
This is the first study to evaluate inter-rater agreement and interpretation correctness among EEG technologists interpreting pediatric critical care EEG. EEG technologists demonstrated better aggregate agreement for seizure detection than other EEG findings, yet individual results and internal consistency varied widely. These data provide important insight into the common practice of utilizing EEG technologists for screening critical care EEG.
SUBMITTER: Ahrens S
PROVIDER: S-EPMC7856064 | biostudies-literature |
REPOSITORIES: biostudies-literature
Similar Datasets
Project description:Researchers have long been interested in how social evaluations are made based upon first impressions of faces. It is also important to consider the level of agreement we see in such evaluations across raters and what this may tell us. Typically, high levels of inter-rater agreement for facial judgements are reported, but the measures used may be misleading. At present, studies commonly report Cronbach's α as a way to quantify agreement, although problematically, there are various issues with the use of this measure. Most importantly, because researchers treat raters as items, Cronbach's α is inflated by larger sample sizes even when agreement between raters is fixed. Here, we considered several alternative measures and investigated whether these better discriminate between traits that were predicted to show low (parental resemblance), intermediate (attractiveness, dominance, trustworthiness), and high (age, gender) levels of agreement. Importantly, the level of inter-rater agreement has not previously been studied for many of these traits. In addition, we investigated whether familiar faces resulted in differing levels of agreement in comparison with unfamiliar faces. Our results suggest that alternative measures may prove more informative than Cronbach's α when determining how well raters agree in their judgements. Further, we found no apparent influence of familiarity on levels of agreement. Finally, we show that, like attractiveness, both trustworthiness and dominance show significant levels of private taste (personal or idiosyncratic rater perceptions), although shared taste (perceptions shared with other raters) explains similar levels of variance in people's perceptions. In conclusion, we recommend that researchers investigating social judgements of faces consider alternatives to Cronbach's α but should also be prepared to examine both the potential value and origin of private taste as these might prove informative.
| S-EPMC6097668 | biostudies-literature
Project description:The aim of this study was to assess on videographs the intra- and inter-rater reproducibility of the Smile Esthetic Index (SEI) that has been previously validated on photographs. Smile videographs were obtained using a smartphone associated with the Smile Lite MDP mounted on a tripod. They were then randomized and evaluated twice consecutively at a 1-week interval by three periodontists according to the SEI based on 10 variables. Cohen's Kappa and Fleiss' Kappa tests were performed to measure intra- and inter-rater agreement. Sixty-five smile videographs of 24 men and 41 women (mean age 33 ± 11.3 years) were scored. A mean intra-rater agreement of 0.68 (0.64-0.73) was obtained, representing substantial agreement. The inter-rater agreement calculated for each variable ranged from 0.31 for the variable "absence of visible excessive gingiva" to 0.90 for the variable "absence of diastema and/or missing inter-dental papilla." Within the limits of this study, we have demonstrated that it was possible to use videographs to reproducibly evaluate an aesthetic score (SEI) previously validated on photographs.
| S-EPMC9139191 | biostudies-literature
Project description:BackgroundIn inter-rater agreement studies, the assessment behaviour of raters can be influenced by their experience, training levels, the degree of willingness to take risks, and the availability of clear guidelines for the assessment. When the assessment behaviour of raters differentiates for some levels of an ordinal classification, a grey zone occurs between the corresponding adjacent cells to these levels around the main diagonal of the table. A grey zone introduces a negative bias to the estimate of the agreement level between the raters. In that sense, it is crucial to detect the existence of a grey zone in an agreement table.MethodsIn this study, a framework composed of a metric and the corresponding threshold is developed to identify grey zones in an agreement table. The symmetry model and Cohen's kappa are used to define the metric, and the threshold is based on a nonlinear regression model. A numerical study is conducted to assess the accuracy of the developed framework. Real data examples are provided to illustrate the use of the metric and the impact of identifying a grey zone.ResultsThe sensitivity and specificity of the proposed framework are shown to be very high under moderate, substantial, and near-perfect agreement levels for [Formula: see text] and [Formula: see text] tables and sample sizes greater than or equal to 100 and 50, respectively. Real data examples demonstrate that when a grey zone is detected in the table, it is possible to report a notably higher level of agreement in the studies.ConclusionsThe accuracy of the proposed framework is sufficiently high; hence, it provides practitioners with a precise way to detect the grey zones in agreement tables.
| S-EPMC9814438 | biostudies-literature
Project description:Background and aimsVarious studies have been previously conducted on the diagnosis of lymphadenopathy as benign or malignant, but the results vary. These studies did not describe the inter-rater agreement on the EUS features of lymphadenopathy. In this study, we evaluate the inter-rater agreement on EUS features and propose EUS diagnostic norms for lymphadenopathy based on inter-rater agreement.MethodA total of 68 lymph nodes subjected to EUS-fine needle aspiration (FNA) were reviewed by five endoscopic experts. The EUS features evaluated lymph node size, shape, border, margin, echogenicity, homogeneity, and the hilum of the lymph node. Inter-rater agreement (multi-rater kappa statics) was performed. We established new criteria using results with a high degree of inter-rater agreement from EUS features and compared them with the former criteria.ResultThere was a moderate agreement on shape, kappa (K) = 0.44 (95% confidence interval [CI]: 0.34-0.54), and fair agreement on echogenicity, homogeneity, border, and hilum of the lymph node, K (95% CI) = 0.33 (0.17-0.38), 0.34 (0.26-0.35), 0.22 (0.21-0.31), and 0.22 (0.11-0.26), respectively. This resulted in the establishment of new EUS diagnostic criteria using shape, long axis > 20 mm and short axis > 10 mm. New criteria were superior to old criteria (area under the curve 0.82 vs 0.52, P < 0.001).ConclusionEUS diagnostic criteria for lymphadenopathy based on inter-rater agreement were more accurate than old criteria. This result will be useful for the diagnosis of lymphadenopathy.
| S-EPMC6386303 | biostudies-literature
Project description:Objectives To explore agreement among healthcare professionals assessing eligibility for work disability benefits.Design Systematic review and narrative synthesis of reproducibility studies.Data sources Medline, Embase, and PsycINFO searched up to 16 March 2016, without language restrictions, and review of bibliographies of included studies.Eligibility criteria Observational studies investigating reproducibility among healthcare professionals performing disability evaluations using a global rating of working capacity and reporting inter-rater reliability by a statistical measure or descriptively. Studies could be conducted in insurance settings, where decisions on ability to work include normative judgments based on legal considerations, or in research settings, where decisions on ability to work disregard normative considerations. : Teams of paired reviewers identified eligible studies, appraised their methodological quality and generalisability, and abstracted results with pretested forms. As heterogeneity of research designs and findings impeded a quantitative analysis, a descriptive synthesis stratified by setting (insurance or research) was performed.Results From 4562 references, 101 full text articles were reviewed. Of these, 16 studies conducted in an insurance setting and seven in a research setting, performed in 12 countries, met the inclusion criteria. Studies in the insurance setting were conducted with medical experts assessing claimants who were actual disability claimants or played by actors, hypothetical cases, or short written scenarios. Conditions were mental (n=6, 38%), musculoskeletal (n=4, 25%), or mixed (n=6, 38%). Applicability of findings from studies conducted in an insurance setting to real life evaluations ranged from generalisable (n=7, 44%) and probably generalisable (n=3, 19%) to probably not generalisable (n=6, 37%). Median inter-rater reliability among experts was 0.45 (range intraclass correlation coefficient 0.86 to κ-0.10). Inter-rater reliability was poor in six studies (37%) and excellent in only two (13%). This contrasts with studies conducted in the research setting, where the median inter-rater reliability was 0.76 (range 0.91-0.53), and 71% (5/7) studies achieved excellent inter-rater reliability. Reliability between assessing professionals was higher when the evaluation was guided by a standardised instrument (23 studies, P=0.006). No such association was detected for subjective or chronic health conditions or the studies' generalisability to real world evaluation of disability (P=0.46, 0.45, and 0.65, respectively).Conclusions Despite their common use and far reaching consequences for workers claiming disabling injury or illness, research on the reliability of medical evaluations of disability for work is limited and indicates high variation in judgments among assessing professionals. Standardising the evaluation process could improve reliability. Development and testing of instruments and structured approaches to improve reliability in evaluation of disability are urgently needed.
| S-EPMC5283380 | biostudies-literature
Project description:ObjectiveTo determine the inter-rater agreement (IRA) of a standardized nomenclature for EEG spectrogram patterns, and to estimate the probability distribution of ictal-interictal continuum (IIC) patterns vs. other EEG patterns within each category in this nomenclature.MethodsWe defined seven spectrogram categories: "Solid Flames", "Irregular Flames", "Broadband-monotonous", "Narrowband-monotonous", "Stripes", "Low power", and "Artifact". Ten electroencephalographers scored 115 spectrograms and the corresponding raw EEG samples. Gwet's agreement coefficient was used to calculate IRA.ResultsSolid Flames represented seizures or IIC patterns 69.4% of the time. Irregular Flames represented seizures or IIC patterns 38.7% of the time. Broadband-monotonous primarily corresponded with seizures or IIC (54.3%) and Narrowband-monotonous with focal or generalized slowing (43.8%). Stripes were associated with burst-suppression (37.2%) and generalized suppression (34.4%). Low Power category was associated with generalized suppression (94%). There was "near perfect" agreement for Solid Flames (κ = 94.36), Low power (κ = 92.61), and Artifact (κ = 93.72). There was "substantial agreement" for all other categories (κ = 74.65-79.49).ConclusionsThis EEG spectrogram nomenclature has high IRA among electroencephalographers.SignificanceThe nomenclature can be a useful tool for EEG screening. Future studies are needed to determine if using this nomenclature shortens time to IIC identification, and how best to use it in practice to reduce time to intervention.
| S-EPMC7461156 | biostudies-literature
Project description:Objectives: It has been recommended that clinical trials of Traditional Chinese Medicine (TCM) would be more ecologically valid if its characteristic mode of diagnostic reasoning were integrated into their design. In that context, however, it is also widely held that demonstrating a high level of agreement on initial TCM diagnoses is necessary for the replicability that the biomedical paradigm requires for the conclusions from such trials. Our aim was to review, summarize, and critique quantitative experimental studies of inter-rater agreement in TCM, and some of their underlying assumptions. Design: Systematic electronic searches were conducted for articles that reported a quantitative measure of inter-rater agreement across a number of rating choices based on examinations of human subjects in person by TCM practitioners, and published in English language peer-reviewed journals. Publications in languages other than English were not included, nor those appearing in other than peer-reviewed journals. Predefined categories of information were extracted from full texts by two investigators working independently. Each article was scored for methodological quality. Outcome measures: Design features across all studies and levels of inter-rater agreement across studies that reported the same type of outcome statistic were compared. Results: Twenty-one articles met inclusion criteria. Fourteen assessed inter-rater agreement on TCM diagnoses, two on diagnostic signs found upon traditional TCM examination, and five on novel rating schemes derived from TCM theory and practice. Raters were students of TCM colleges or graduates of TCM training programs with 3 or more years experience and licensure. Type of outcome statistic varied. Mean rates of pairwise agreement averaged 57% (median 65, range 19-96) across the 9 studies reporting them. Mean Cohen's kappa averaged 0.34 (median 0.34, range 0.07-0.59) across the seven studies reporting them. Meta-analysis was not possible due to variations in study design and outcome statistics. High risks of bias and confounding, and deficits in statistical reporting were common. Conclusions: With a few exceptions, the levels of agreement were low to moderate. Most studies had significant deficits of both methodology and reporting. Results overall suggest a few design features that might contribute to higher levels of agreement. These should be studied further with better experimental controls and more thorough reporting of outcomes. In addition, methods of complex systems analysis should be explored to more adequately model the relationship between clinical outcomes, and the series of diagnoses and treatments that are the norm in actual TCM practice.
| S-EPMC6864748 | biostudies-literature
Project description:The Revised Hammersmith Scale (RHS) for Spinal Muscular Atrophy (SMA) was designed as a psychometrically robust clinical outcome assessment to assess physical abilities of patients with type 2 and 3 SMA. The reliability properties of the RHS have not yet been reported. A prospective RHS reliability study was undertaken in a UK cohort of experienced neuromuscular paediatric Physiotherapists. Reliability testing was conducted via a virtual survey platform two weeks apart. Through the virtual platform participants scored videos of two RHS assessments, one of a child with SMA 2 and one of a child with SMA 3. Inter and intra-rater reliability was analysed using a type 3 Intraclass Correlation Coefficient (ICC). Intra-rater agreement was further analysed using Bland Altman (BA) Limits of Agreement (LOA) and plots. The acceptable inter and intra-rater variability was set as a change of ± 2 by the international team of expert physiotherapists who developed the RHS. Inter-rater agreement, n = 22 raters, type 3 ICC was 0.989 (95% CI 0.944 to 1.00), 97.7% of scores were within the acceptable limits of ± 2 points. Intra-rater agreement, n = 21 raters, type 3 ICC ranged from 0.922 to 1.0, with 97.6% of scores within the acceptable limits of ± 2 points. The mean SMA 2 intra-rater difference was -0.10 (-0.6 to 0.4), with lower LOA -2.24 and upper LOA +2.04. Intra-rater difference between tests for SMA 3 intra-rater difference was -0.05 (-0.6 to 0.5), with lower LOA -2.48 and upper LOA +2.38. Intra-rater scoring precision fell within BA agreement limits of ±2 points. The results demonstrate that the RHS is highly reliable when used by experienced UK physiotherapists, and variability of test scores regarding inter and intra-rater reliability was confirmed to lie within ±2 points.
| S-EPMC9770369 | biostudies-literature
Project description:BackgroundEar, Nose and Throat (ENT) primary contact models of care use audiologists as the first triage point for children referred to ENT for middle ear and hearing concerns; and have shown reduced waiting time, improved ENT surgical conversion rates and increased service capacity. This study aimed to investigate 'safety and quality' of the model by looking at agreement between audiologists' and an ENT's clinical decisions. METHODS: We performed an inter-rater agreement study on diagnosis and management decisions made by audiologists and an ENT for 50 children seen in an Australian hospital's ENT primary contact service, and examined the nature and patterns of disagreements.ResultsProfessionals agreed on at least one site-of-lesion diagnosis for all children (100%) and on the primary management for 74% (Gwet's AC1 = 0.67). Management disagreements clustered around i) providing 'watchful waiting' versus sooner medical opinion (18%), and ii) providing monitoring versus discharge for children with no current symptoms (8%). There were no cases where the audiologist recommended discharge when the ENT recommended further medical opinion.ConclusionsOur novel research provides further evidence that Audiologist-led primary contact models for children with middle ear and hearing concerns are safe as well as efficient.
| S-EPMC9641870 | biostudies-literature
Project description:The point of departure for this study was the need for an instrument capturing social activities updated to the living circumstances of the 21st century. Starting out from the Frenchay Activities Index (FAI), the objective was to investigate the inter-rater agreement of the scoring of a modified and extended Swedish version, capturing activity performance and participation. Thirty-one older people, living in the community post stroke, were interviewed in their homes by two raters using the Swedish FAI version, extended with items on use of mode of transport and use of telephone, and modified with additional scales. Besides the original frequency scale, the new scales captured changes in frequency, reasons for change, and performance satisfaction; inter-rater agreement was analyzed with kappa statistics. Overall, the inter-rater agreement was high or very high, with weighted [Formula: see text] = 0.924 for the frequency scale and [Formula: see text]= 0.784-0.940 for the new scales. As a conclusion, while further validity and reliability testing is necessary, when out-of-home activities are of interest the scoring of the modified and extended Swedish FAI version can be administered with high inter-rater agreement.
| S-EPMC5549121 | biostudies-other