Electroencephalography Technologist Inter-rater Agreement and Interpretation of Pediatric Critical Care Electroencephalography.
Ontology highlight
ABSTRACT: Objectives
Electroencephalography (EEG) technologists commonly screen continuous EEG. Until now, the inter-rater agreement or sensitivity for important EEG findings has been unknown in this group.Methods
Twenty-nine EEG technologists and three clinical neurophysiologists interpreted 90 five-minute samples of pediatric critical care EEG. Inter-rater agreement was examined with Cohen's kappa and Fleiss' kappa for EEG findings. A gold-standard consensus agreement was developed for examining sensitivity and specificity for seizures or discontinuity. Kruskal-Wallis tests with Benjamani-Hochberg corrections for multiple comparisons were utilized to examine associations between correct scoring and certification status and years of experience.Results
Aggregate agreement was moderate for seizures and fair for EEG background continuity among EEG technologists. Individual agreement for seizures and continuity varied from slight to substantial. For individual EEG technologists, sensitivity for seizures ranged from 44 to 93% and sensitivity for continuity ranged from 81 to 100%. Raters with Certified Long Term Monitoring credentials were more likely to identify seizures correctly.Significance
This is the first study to evaluate inter-rater agreement and interpretation correctness among EEG technologists interpreting pediatric critical care EEG. EEG technologists demonstrated better aggregate agreement for seizure detection than other EEG findings, yet individual results and internal consistency varied widely. These data provide important insight into the common practice of utilizing EEG technologists for screening critical care EEG.
SUBMITTER: Ahrens S
PROVIDER: S-EPMC7856064 | biostudies-literature |
REPOSITORIES: biostudies-literature
Similar Datasets
Project description:Researchers have long been interested in how social evaluations are made based upon first impressions of faces. It is also important to consider the level of agreement we see in such evaluations across raters and what this may tell us. Typically, high levels of inter-rater agreement for facial judgements are reported, but the measures used may be misleading. At present, studies commonly report Cronbach's ? as a way to quantify agreement, although problematically, there are various issues with the use of this measure. Most importantly, because researchers treat raters as items, Cronbach's ? is inflated by larger sample sizes even when agreement between raters is fixed. Here, we considered several alternative measures and investigated whether these better discriminate between traits that were predicted to show low (parental resemblance), intermediate (attractiveness, dominance, trustworthiness), and high (age, gender) levels of agreement. Importantly, the level of inter-rater agreement has not previously been studied for many of these traits. In addition, we investigated whether familiar faces resulted in differing levels of agreement in comparison with unfamiliar faces. Our results suggest that alternative measures may prove more informative than Cronbach's ? when determining how well raters agree in their judgements. Further, we found no apparent influence of familiarity on levels of agreement. Finally, we show that, like attractiveness, both trustworthiness and dominance show significant levels of private taste (personal or idiosyncratic rater perceptions), although shared taste (perceptions shared with other raters) explains similar levels of variance in people's perceptions. In conclusion, we recommend that researchers investigating social judgements of faces consider alternatives to Cronbach's ? but should also be prepared to examine both the potential value and origin of private taste as these might prove informative.
| S-EPMC6097668 | biostudies-literature
Project description:Background and aimsVarious studies have been previously conducted on the diagnosis of lymphadenopathy as benign or malignant, but the results vary. These studies did not describe the inter-rater agreement on the EUS features of lymphadenopathy. In this study, we evaluate the inter-rater agreement on EUS features and propose EUS diagnostic norms for lymphadenopathy based on inter-rater agreement.MethodA total of 68 lymph nodes subjected to EUS-fine needle aspiration (FNA) were reviewed by five endoscopic experts. The EUS features evaluated lymph node size, shape, border, margin, echogenicity, homogeneity, and the hilum of the lymph node. Inter-rater agreement (multi-rater kappa statics) was performed. We established new criteria using results with a high degree of inter-rater agreement from EUS features and compared them with the former criteria.ResultThere was a moderate agreement on shape, kappa (K) = 0.44 (95% confidence interval [CI]: 0.34-0.54), and fair agreement on echogenicity, homogeneity, border, and hilum of the lymph node, K (95% CI) = 0.33 (0.17-0.38), 0.34 (0.26-0.35), 0.22 (0.21-0.31), and 0.22 (0.11-0.26), respectively. This resulted in the establishment of new EUS diagnostic criteria using shape, long axis > 20 mm and short axis > 10 mm. New criteria were superior to old criteria (area under the curve 0.82 vs 0.52, P < 0.001).ConclusionEUS diagnostic criteria for lymphadenopathy based on inter-rater agreement were more accurate than old criteria. This result will be useful for the diagnosis of lymphadenopathy.
| S-EPMC6386303 | biostudies-literature
Project description:Objectives To explore agreement among healthcare professionals assessing eligibility for work disability benefits.Design Systematic review and narrative synthesis of reproducibility studies.Data sources Medline, Embase, and PsycINFO searched up to 16 March 2016, without language restrictions, and review of bibliographies of included studies.Eligibility criteria Observational studies investigating reproducibility among healthcare professionals performing disability evaluations using a global rating of working capacity and reporting inter-rater reliability by a statistical measure or descriptively. Studies could be conducted in insurance settings, where decisions on ability to work include normative judgments based on legal considerations, or in research settings, where decisions on ability to work disregard normative considerations. : Teams of paired reviewers identified eligible studies, appraised their methodological quality and generalisability, and abstracted results with pretested forms. As heterogeneity of research designs and findings impeded a quantitative analysis, a descriptive synthesis stratified by setting (insurance or research) was performed.Results From 4562 references, 101 full text articles were reviewed. Of these, 16 studies conducted in an insurance setting and seven in a research setting, performed in 12 countries, met the inclusion criteria. Studies in the insurance setting were conducted with medical experts assessing claimants who were actual disability claimants or played by actors, hypothetical cases, or short written scenarios. Conditions were mental (n=6, 38%), musculoskeletal (n=4, 25%), or mixed (n=6, 38%). Applicability of findings from studies conducted in an insurance setting to real life evaluations ranged from generalisable (n=7, 44%) and probably generalisable (n=3, 19%) to probably not generalisable (n=6, 37%). Median inter-rater reliability among experts was 0.45 (range intraclass correlation coefficient 0.86 to κ-0.10). Inter-rater reliability was poor in six studies (37%) and excellent in only two (13%). This contrasts with studies conducted in the research setting, where the median inter-rater reliability was 0.76 (range 0.91-0.53), and 71% (5/7) studies achieved excellent inter-rater reliability. Reliability between assessing professionals was higher when the evaluation was guided by a standardised instrument (23 studies, P=0.006). No such association was detected for subjective or chronic health conditions or the studies' generalisability to real world evaluation of disability (P=0.46, 0.45, and 0.65, respectively).Conclusions Despite their common use and far reaching consequences for workers claiming disabling injury or illness, research on the reliability of medical evaluations of disability for work is limited and indicates high variation in judgments among assessing professionals. Standardising the evaluation process could improve reliability. Development and testing of instruments and structured approaches to improve reliability in evaluation of disability are urgently needed.
| S-EPMC5283380 | biostudies-literature
Project description:ObjectiveTo determine the inter-rater agreement (IRA) of a standardized nomenclature for EEG spectrogram patterns, and to estimate the probability distribution of ictal-interictal continuum (IIC) patterns vs. other EEG patterns within each category in this nomenclature.MethodsWe defined seven spectrogram categories: "Solid Flames", "Irregular Flames", "Broadband-monotonous", "Narrowband-monotonous", "Stripes", "Low power", and "Artifact". Ten electroencephalographers scored 115 spectrograms and the corresponding raw EEG samples. Gwet's agreement coefficient was used to calculate IRA.ResultsSolid Flames represented seizures or IIC patterns 69.4% of the time. Irregular Flames represented seizures or IIC patterns 38.7% of the time. Broadband-monotonous primarily corresponded with seizures or IIC (54.3%) and Narrowband-monotonous with focal or generalized slowing (43.8%). Stripes were associated with burst-suppression (37.2%) and generalized suppression (34.4%). Low Power category was associated with generalized suppression (94%). There was "near perfect" agreement for Solid Flames (κ = 94.36), Low power (κ = 92.61), and Artifact (κ = 93.72). There was "substantial agreement" for all other categories (κ = 74.65-79.49).ConclusionsThis EEG spectrogram nomenclature has high IRA among electroencephalographers.SignificanceThe nomenclature can be a useful tool for EEG screening. Future studies are needed to determine if using this nomenclature shortens time to IIC identification, and how best to use it in practice to reduce time to intervention.
| S-EPMC7461156 | biostudies-literature
Project description:Objectives: It has been recommended that clinical trials of Traditional Chinese Medicine (TCM) would be more ecologically valid if its characteristic mode of diagnostic reasoning were integrated into their design. In that context, however, it is also widely held that demonstrating a high level of agreement on initial TCM diagnoses is necessary for the replicability that the biomedical paradigm requires for the conclusions from such trials. Our aim was to review, summarize, and critique quantitative experimental studies of inter-rater agreement in TCM, and some of their underlying assumptions. Design: Systematic electronic searches were conducted for articles that reported a quantitative measure of inter-rater agreement across a number of rating choices based on examinations of human subjects in person by TCM practitioners, and published in English language peer-reviewed journals. Publications in languages other than English were not included, nor those appearing in other than peer-reviewed journals. Predefined categories of information were extracted from full texts by two investigators working independently. Each article was scored for methodological quality. Outcome measures: Design features across all studies and levels of inter-rater agreement across studies that reported the same type of outcome statistic were compared. Results: Twenty-one articles met inclusion criteria. Fourteen assessed inter-rater agreement on TCM diagnoses, two on diagnostic signs found upon traditional TCM examination, and five on novel rating schemes derived from TCM theory and practice. Raters were students of TCM colleges or graduates of TCM training programs with 3 or more years experience and licensure. Type of outcome statistic varied. Mean rates of pairwise agreement averaged 57% (median 65, range 19-96) across the 9 studies reporting them. Mean Cohen's kappa averaged 0.34 (median 0.34, range 0.07-0.59) across the seven studies reporting them. Meta-analysis was not possible due to variations in study design and outcome statistics. High risks of bias and confounding, and deficits in statistical reporting were common. Conclusions: With a few exceptions, the levels of agreement were low to moderate. Most studies had significant deficits of both methodology and reporting. Results overall suggest a few design features that might contribute to higher levels of agreement. These should be studied further with better experimental controls and more thorough reporting of outcomes. In addition, methods of complex systems analysis should be explored to more adequately model the relationship between clinical outcomes, and the series of diagnoses and treatments that are the norm in actual TCM practice.
| S-EPMC6864748 | biostudies-literature
Project description:OBJECTIVE:Surgical site infection (SSI) is the most common nosocomial infection, in vascular surgery patients, who experience a high rate of readmission. Facilitating transition from hospital to outpatient care with digital image-based wound monitoring has the potential to detect and to enable treatment of SSI at an early stage. In this study, we evaluated whether smartphone digital images can supplant in-person evaluation of postoperative vascular surgery wounds. METHODS:We developed a wound assessment checklist using previously validated criteria. We recruited adults who underwent a vascular surgical procedure between 2014 and 2015, involving an incision of at least 3 cm, from a high-volume academic vascular surgery service. Vascular surgery care providers evaluated wounds in person using the assessment checklist; a different group of providers evaluated wounds by a smartphone digital image. Inter-rater agreement coefficients for wound characteristics and treatment plan were calculated within and between the in-person group and the digital image group; the sensitivity and specificity of digital images relative to in-person evaluation were determined. RESULTS:We assessed a total of 80 wounds. Regardless of modality, inter-rater agreement was poor when wounds were evaluated for the presence of ecchymosis and redness; moderate for cellulitis; and high for the presence of a drain, necrosis, or dehiscence. As expected, the presence of drainage was more readily observed in person. Inter-rater agreement was high for both in-person and image-based assessment with respect to course of treatment, with near-perfect agreement for treatments ranging from antibiotics to surgical débridement to hospital readmission. No difference in agreement emerged when raters evaluated poor-quality compared with high-quality images. For most parameters, specificity was higher than sensitivity for image-based compared with "gold standard" in-person assessment. CONCLUSIONS:Using smartphone digital images is a valid method for evaluating postoperative vascular surgery wounds and is comparable to in-person evaluation with regard to most wound characteristics. The inter-rater reliability for determining treatment recommendations was universally high. Remote wound monitoring and assessment may play an integral role in future transitional care models to decrease readmission for SSI in vascular or other surgical patients. These findings will inform smartphone implementation in the clinical care setting as wound images transition from informal clinical communication to becoming part of the care standard.
| S-EPMC4913032 | biostudies-literature
Project description:PurposeTo create a standardized method for evaluating the video recordings of patients self-instilling eye drops and to determine the level of agreement of eye drop instillation efficacy, safety and efficiency ratings by three masked graders.DesignProspective cross-sectional study.Participants78 patients with open-angle glaucoma or ocular hypertension who had at least 6 months of experience with the use of eye drop medications.MethodsParticipants were video recorded while self-instilling artificial tears sequentially to both eyes. Three masked observers graded these video recordings on three criteria: efficacy (the determination of whether an eye drop was instilled on the ocular surface), safety (assessment of whether the tip of the medication bottle made contact with the ocular surface or eyelids), and efficiency (the number of eye drops expressed from the bottle).Main outcome measuresAfter grading the video recordings based on efficacy, safety, and efficiency, kappa statistics were used to estimate inter-rater agreement.ResultsThe mean kappa level of agreement for efficacy, safety, and efficiency was 0.64 (95% confidence interval (CI), 0.42-0.87), 0.73 (95% CI, 0.58-0.88), and 0.62 (95% CI, 0.42-0.81), respectively.ConclusionsWe demonstrated good inter-rater reproducibility of the masked analysis of video recordings of patients self-instilling eye drops based on three criteria: efficiency, safety, and efficacy.
| S-EPMC4711577 | biostudies-literature
Project description:The point of departure for this study was the need for an instrument capturing social activities updated to the living circumstances of the 21st century. Starting out from the Frenchay Activities Index (FAI), the objective was to investigate the inter-rater agreement of the scoring of a modified and extended Swedish version, capturing activity performance and participation. Thirty-one older people, living in the community post stroke, were interviewed in their homes by two raters using the Swedish FAI version, extended with items on use of mode of transport and use of telephone, and modified with additional scales. Besides the original frequency scale, the new scales captured changes in frequency, reasons for change, and performance satisfaction; inter-rater agreement was analyzed with kappa statistics. Overall, the inter-rater agreement was high or very high, with weighted [Formula: see text] = 0.924 for the frequency scale and [Formula: see text]= 0.784-0.940 for the new scales. As a conclusion, while further validity and reliability testing is necessary, when out-of-home activities are of interest the scoring of the modified and extended Swedish FAI version can be administered with high inter-rater agreement.
| S-EPMC5549121 | biostudies-other
Project description:ObjectiveDifferent definitions have been used for screening for rheumatic heart disease (RHD). This led to the development of the 2012 evidence-based World Heart Federation (WHF) echocardiographic criteria. The objective of this study is to determine the intra-rater and inter-rater reliability and agreement in differentiating no RHD from mild RHD using the WHF echocardiographic criteria.MethodsA standard set of 200 echocardiograms was collated from prior population-based surveys and uploaded for blinded web-based reporting. Fifteen international cardiologists reported on and categorised each echocardiogram as no RHD, borderline or definite RHD. Intra-rater and inter-rater reliability was calculated using Cohen's and Fleiss' free-marginal multirater kappa (κ) statistics, respectively. Agreement assessment was expressed as percentages. Subanalyses assessed reproducibility and agreement parameters in detecting individual components of WHF criteria.ResultsSample size from a statistical standpoint was 3000, based on repeated reporting of the 200 studies. The inter-rater and intra-rater reliability of diagnosing definite RHD was substantial with a kappa of 0.65 and 0.69, respectively. The diagnosis of pathological mitral and aortic regurgitation was reliable and almost perfect, kappa of 0.79 and 0.86, respectively. Agreement for morphological changes of RHD was variable ranging from 0.54 to 0.93 κ.ConclusionsThe WHF echocardiographic criteria enable reproducible categorisation of echocardiograms as definite RHD versus no or borderline RHD and hence it would be a suitable tool for screening and monitoring disease progression. The study highlights the strengths and limitations of the WHF echo criteria and provides a platform for future revisions.
| S-EPMC6591009 | biostudies-literature
Project description:The interpretation of critical care electroencephalography (EEG) studies is challenging because of the presence of many periodic and rhythmic patterns of uncertain clinical significance. Defining the clinical significance of these patterns requires standardized terminology with high interrater agreement (IRA). We sought to evaluate IRA for the final, published American Clinical Neurophysiology Society (ACNS)-approved version of the critical care EEG terminology (2012 version). Our evaluation included terms not assessed previously and incorporated raters with a broad range of EEG reading experience.After reviewing a set of training slides, 49 readers independently completed a Web-based test consisting of 11 identical questions for each of 37 EEG samples (407 questions). Questions assessed whether a pattern was an electrographic seizure; pattern location (main term 1), pattern type (main term 2); and presence and classification of eight other key features ("plus" modifiers, sharpness, absolute and relative amplitude, frequency, number of phases, fluctuation/evolution, and the presence of "triphasic" morphology).IRA statistics (? values) were almost perfect (90-100%) for seizures, main terms 1 and 2, the +S modifier (superimposed spikes/sharp waves or sharply contoured rhythmic delta activity), sharpness, absolute amplitude, frequency, and number of phases. Agreement was substantial for the +F (superimposed fast activity) and +R (superimposed rhythmic delta activity) modifiers (66% and 67%, respectively), moderate for triphasic morphology (58%), and fair for evolution (21%).IRA for most terms in the ACNS critical care EEG terminology is high. These terms are suitable for multicenter research on the clinical significance of critical care EEG patterns. A PowerPoint slide summarizing this article is available for download in the Supporting Information section http://dx.doi.org/10.1111/epi.12653/supinfo.
| S-EPMC4879939 | biostudies-literature