Prospective evaluation of interrater agreement between EEG technologists and neurophysiologists.
Ontology highlight
ABSTRACT:
SUBMITTER: Beuchat I
PROVIDER: S-EPMC8238944 | biostudies-literature |
REPOSITORIES: biostudies-literature
Similar Datasets
Project description:The interpretation of critical care electroencephalography (EEG) studies is challenging because of the presence of many periodic and rhythmic patterns of uncertain clinical significance. Defining the clinical significance of these patterns requires standardized terminology with high interrater agreement (IRA). We sought to evaluate IRA for the final, published American Clinical Neurophysiology Society (ACNS)-approved version of the critical care EEG terminology (2012 version). Our evaluation included terms not assessed previously and incorporated raters with a broad range of EEG reading experience.After reviewing a set of training slides, 49 readers independently completed a Web-based test consisting of 11 identical questions for each of 37 EEG samples (407 questions). Questions assessed whether a pattern was an electrographic seizure; pattern location (main term 1), pattern type (main term 2); and presence and classification of eight other key features ("plus" modifiers, sharpness, absolute and relative amplitude, frequency, number of phases, fluctuation/evolution, and the presence of "triphasic" morphology).IRA statistics (? values) were almost perfect (90-100%) for seizures, main terms 1 and 2, the +S modifier (superimposed spikes/sharp waves or sharply contoured rhythmic delta activity), sharpness, absolute amplitude, frequency, and number of phases. Agreement was substantial for the +F (superimposed fast activity) and +R (superimposed rhythmic delta activity) modifiers (66% and 67%, respectively), moderate for triphasic morphology (58%), and fair for evolution (21%).IRA for most terms in the ACNS critical care EEG terminology is high. These terms are suitable for multicenter research on the clinical significance of critical care EEG patterns. A PowerPoint slide summarizing this article is available for download in the Supporting Information section http://dx.doi.org/10.1111/epi.12653/supinfo.
| S-EPMC4879939 | biostudies-literature
Project description:PurposeTo determine the agreement between five different methods of ictal EEG source imaging, and to assess their accuracy in presurgical evaluation of patients with focal epilepsy. It was hypothesized that high agreement between methods was associated with higher localization-accuracy.MethodsEEGs were recorded with a 64-electrode array. Thirty-eight seizures from 22 patients were analyzed using five different methods phase mapping, dipole fitting, CLARA, cortical-CLARA and minimum norm. Localization accuracy was determined at sub-lobar level. Reference standard was the final decision of the multidisciplinary epilepsy surgery team, and, for the operated patients, outcome one year after surgery.ResultsAgreement between all methods was obtained in 13 patients (59%) and between all but one methods in additional six patients (27%). There was a trend for minimum norm being less accurate than phase mapping, but none of the comparisons reached significance. Source imaging in cases with agreement between all methods was not more accurate than in the other cases. Ictal source imaging achieved an accuracy of 73% (for operated patients: 86%).ConclusionThere was good agreement between different methods of ictal source imaging. However, good inter-method agreement did not necessarily imply accurate source localization, since all methods faced the limitations of the inverse solution.
| S-EPMC5176190 | biostudies-literature
Project description:OBJECTIVE:The presence of interictal epileptiform discharges (IED) in the electroencephalogram (EEG) is a key finding in the medical workup of a patient with suspected epilepsy. However, inter-rater agreement (IRA) regarding the presence of IED is imperfect, leading to incorrect and delayed diagnoses. An improved understanding of which IED attributes mediate expert IRA might help in developing automatic methods for IED detection able to emulate the abilities of experts. Therefore, using a set of IED scored by a large number of experts, we set out to determine which attributes of IED predict expert agreement regarding the presence of IED. METHODS:IED were annotated on a 5-point scale by 18 clinical neurophysiologists within 200 30-s EEG segments from recordings of 200 patients. 5538 signal analysis features were extracted from the waveforms, including wavelet coefficients, morphological features, signal energy, nonlinear energy operator response, electrode location, and spectrogram features. Feature selection was performed by applying elastic net regression and support vector regression (SVR) was applied to predict expert opinion, with and without the feature selection procedure and with and without several types of signal normalization. RESULTS:Multiple types of features were useful for predicting expert annotations, but particular types of wavelet features performed best. Local EEG normalization also enhanced best model performance. As the size of the group of EEGers used to train the models was increased, the performance of the models leveled off at a group size of around 11. CONCLUSIONS:The features that best predict inter-rater agreement among experts regarding the presence of IED are wavelet features, using locally standardized EEG. Our models for predicting expert opinion based on EEGer's scores perform best with a large group of EEGers (more than 10). SIGNIFICANCE:By examining a large group of EEG signal analysis features we found that wavelet features with certain wavelet basis functions performed best to identify IEDs. Local normalization also improves predictability, suggesting the importance of IED morphology over amplitude-based features. Although most IED detection studies in the past have used opinion from three or fewer experts, our study suggests a "wisdom of the crowd" effect, such that pooling over a larger number of expert opinions produces a better correlation between expert opinion and objectively quantifiable features of the EEG.
| S-EPMC5842710 | biostudies-literature
Project description:PURPOSE:Many neonates undergo electroencephalogram (EEG) monitoring to identify and manage acute symptomatic seizures. Information about brain function contained in the EEG background data may also help predict neurobehavioral outcomes. For EEG background features to be useful as prognostic indicators, the interpretation of these features must be standardized across electroencephalographers. We aimed at determining the interrater and intrarater agreement among electroencephalographers interpreting neonatal EEG background patterns. METHODS:Five neonatal electroencephalographers reviewed 5-to-7.5-minute epochs of EEG from full-term neonates who underwent continuous conventional EEG monitoring. The EEG assessment tool used to classify background patterns was based on the American Clinical Neurophysiology Society's guideline for neonatal EEG terminology. Interrater and intrarater agreement were measured using Kappa coefficients. RESULTS:Interrater agreement was consistently highest for voltage (binary: substantial, kappa = 0.783; categorical: moderate, kappa = 0.562), seizure presence (fair-substantial; kappa = 0.375-0.697), continuity (moderate; kappa = 0.481), burst voltage (moderate; kappa = 0.574), suppressed background presence (moderate-substantial; kappa = 0.493-0.643), delta activity presence (fair-moderate; kappa = 0.369-0.432), theta activity presence (fair-moderate; kappa = 0.347-0.600), presence of graphoelements (fair; kappa = 0.381), and overall impression (binary: moderate, kappa = 0.495; categorical: fair-moderate, kappa = 0.347, 0.465). Agreement was poor or inconsistent for all other patterns. Intrarater agreement was variable, with highest average agreement for voltage (binary: substantial, kappa = 0.75; categorical: substantial, kappa = 0.714) and highest consistent agreement for continuity (moderate-substantial; kappa = 0.43-0.67) and overall impression (moderate-substantial; kappa = 0.42-0.68). CONCLUSIONS:This study demonstrates substantial variability in neonatal EEG background interpretation across electroencephalographers, indicating a need for educational and technological strategies aimed at improving performance.
| S-EPMC6322680 | biostudies-literature
Project description:BackgroundAnterior cruciate ligament (ACL) rupture is the most common ligament injury treated surgically by orthopaedic surgeons. The gold standard for the treatment of the majority of primary ACL tears is ACL reconstruction. However, novel methods of repair, such as bridge-enhanced ACL repair (BEAR), are currently being investigated as alternatives to reconstruction. To assess patients for midsubstance repair suitability, clarify the prognostic implications of injury location and damage, and evaluate the results of a repair technique, it is important to have a baseline classification system or grading scale that is reproducible across surgeons, particularly for multicenter collaboration. Currently, no such system or scale exists.PurposeTo develop an arthroscopic ACL tear classification system and to evaluate its interobserver reliability.Study designCohort study (diagnosis); Level of evidence, 3.MethodsEleven fellowship-trained orthopaedic surgeon investigators reviewed 75 video clips containing arthroscopic evaluation of a torn ACL and then completed the 6-question ACL Pathology Evaluation Form. Agreement statistics including exact agreement, Fleiss ?, Gwet agreement coefficient 1 (AC1), and Gwet AC2 were then calculated to assess interobserver reliability.ResultsIn aggregate, the multiple assessments of observer reproducibility revealed that surgeon participants in this study, when evaluating the same injury, agreed roughly 80% of the time on whether (1) at least 50% of the tibial footprint remained, (2) the remaining tibial stump was ?10 mm, and (3) the injury was therefore reparable using the BEAR procedure. Participants also agreed roughly 60% of the time on exactly how many suturable bundles were available. These characteristics are believed to be most important, among those studied, in determining whether a torn ACL is amenable to midsubstance repair.ConclusionThis study is the first of its kind to demonstrate the interobserver reliability of arthroscopic classification of ACL tears. We have demonstrated that this classification system, though not ideally reproducible, is reliable enough across surgeons at multiple institutions for use in multicenter studies.RegistrationNCT03776162 (ClinicalTrials.gov identifier).
| S-EPMC7720329 | biostudies-literature
Project description:ObjectiveTo systematically review the literature and assess agreement on the Alberta Stroke Program Early CT Score (ASPECTS) among clinicians involved in the management of thrombectomy candidates.MethodsStudies assessing agreement using ASPECTS published from 2000 to 2015 were reviewed. Fifteen raters reviewed and scored the anonymized CT scans of 30 patients recruited in a local thrombectomy trial during 2 independent sessions, in order to study intrarater and interrater agreement. Agreement was measured using intraclass correlation coefficients (ICCs) and Fleiss kappa statistics for ASPECTS and dichotomized ASPECTS at various cutoff values.ResultsThe review yielded 30 articles reporting 40 measures of agreement. Populations, methods, analyses, and results were heterogeneous (slight to excellent agreement), precluding a meta-analysis. When analyzed as a categorical variable, intrarater agreement was slight to moderate (κ = 0.042-0.469); it reached a substantial level (κ > 0.6) in 11/15 raters when the score was dichotomized (0-5 vs 6-10). The interrater ICCs varied between 0.672 and 0.811, but agreement was slight to moderate (κ = 0.129-0.315). Even in the best of cases, when ASPECTS was dichotomized as 0-5 vs 6-10, interrater agreement did not reach a substantial level (κ = 0.561), which translates into at least 5 of 15 raters not giving the same dichotomized verdict in 15% of patients.ConclusionsIn patients considered for thrombectomy, there may be insufficient agreement between clinicians for ASPECTS to be reliably used as a criterion for treatment decisions.
| S-EPMC4955274 | biostudies-literature
Project description:BackgroundThe present study takes place within a geriatric program, aiming at improving the diagnosis and management of geriatric syndromes in primary care. Within this program it was of prime importance to be able to rely on a robust and reproducible geriatric consultation to use as a gold standard for evaluating a primary care brief assessment tool. The specific objective of the present study was thus assessing the agreement and reliability of a comprehensive geriatric consultation.MethodThe study was conducted at the outpatient clinic of the Service of Geriatric Medicine, University of Lausanne, Switzerland. All community-dwelling older persons aged 70 years and above were eligible. Patients were excluded if they hadn't a primary care physician, they were unable to speak French, or they were already assessed by a geriatrician within the last 12 months. A set of 9 geriatricians evaluated 20 patients. Each patient was assessed twice within a 2-month delay. Geriatric consultations were based on a structured evaluation process, leading to rating the following geriatric conditions: functional, cognitive, visual, and hearing impairment, mood disorders, risk of fall, osteoporosis, malnutrition, and urinary incontinence. Reliability and agreement estimates on each of these items were obtained using a three-way Intraclass Correlation and a three-way Observed Disagreement index. The latter allowed a decomposition of overall disagreement into disagreements due to each source of error variability (visit, rater and random).ResultsAgreement ranged between 0.62 and 0.85. For most domains, geriatrician-related error variability explained an important proportion of disagreement. Reliability ranged between 0 and 0.8. It was poor/moderate for visual impairment, malnutrition and risk of fall, and good/excellent for functional/cognitive/hearing impairment, osteoporosis, incontinence and mood disorders.ConclusionsSix out of nine items of the geriatric consultation described in this study (functional/cognitive/hearing impairment, osteoporosis, incontinence and mood disorders) present a good to excellent reliability and can safely be used as a reference (gold standard) to evaluate the diagnostic performance of a primary care brief assessment tool. More objective/significant measures are needed to improve reliability of malnutrition, visual impairment, and risk of fall assessment before they can serve as a safe gold standard of a primary care tool.
| S-EPMC5517926 | biostudies-literature
Project description:PurposeElectroencephalography (EEG) remains the gold standard for identifying rhythmic and periodic patterns in critically ill patients. Residents have frequent exposures to EEG and critically ill patients during their training. Our study aimed to assess resident competency in the use of the American Clinical Neurophysiology Society (ACNS) critical care EEG terminology.MethodsAfter self-guided reading and a 2-hour session reviewing the ACNS critical care EEG Terminology training slides, 16 adult neurology residents (PGY 2-4) completed the ACNS certification test. Performance scores were reported as average percent agreement (PA%) with a previously established 5-member expert panel. Interrater agreement was calculated to gauge consensus among peers within the resident cohort. Self-reported comfort levels using the terminology were also obtained.ResultsThe overall pass rate for our cohort was 50% and the median score was 74%. The terms with the highest PA% were: seizures (86.4%), main term 1 (78%), main term 2 (74%). Interrater agreement scores (kappa values) were almost perfect for seizure, and substantial for main terms 1 and 2.ConclusionsOur data suggests that with minimal investment, adult neurology residents at various stages of training can effectively learn the ACNS critical care EEG Terminology.
| S-EPMC6778405 | biostudies-literature
Project description:BackgroundNasal endoscopy is an important part of the clinical evaluation of patients with chronic rhinosinusitis. However, the objectivity and interrater agreement of the procedure related findings have not been well studied, especially in patients who have previously had sinus surgery.MethodsPatients with a history of endoscopic sinus surgery for chronic rhinosinusitis were prospectively enrolled from a tertiary rhinology practice. Fourteen endoscopic nasal examinations were recorded using digital video capture software. Each patient also underwent computed tomography (CT) and completed the Sinonasal Outcome Test (SNOT-22). Blinded review of inflammatory and anatomic findings for each video was independently performed by 5 academic rhinologists at separate institutions. Comparisons were performed using the unweighted Fleiss' kappa statistic (K(f) ) and the prevalence- and bias-adjusted kappa (PABAK).ResultsThere were no significant correlations between age, Lund-Mackay score, or SNOT-22 score. Interrater agreement was variable across the characteristics studied. Mean PABAK was excellent for the assessment of polyps (K(f) = 0.886); moderate for the assessments of middle turbinate (MT) integrity (K(f) = 0.543), MT position (K(f) = 0.443), maxillary sinus patency (K(f) = 0.593), and ethmoid sinus patency (K(f) = 0.429); fair for discharge (K(f) = 0.314), synechiae (K(f) = 0.257), and middle meatus patency (K(f) = 0.229); and poor for MT mucosal changes (K(f) = 0.148) and uncinate process (K(f) = 0.126).ConclusionThis study was notable for variability in the interrater agreement among the inflammatory and anatomic attributes that were examined. Further standardization of nasal endoscopy with regard to interpretation may improve the reliability of this procedure in clinical practice.
| S-EPMC3448826 | biostudies-literature
Project description:To determine the interobserver agreement (IOA) of neonatal seizure detection using the gold standard of conventional, multichannel EEG.A cohort of full-term neonates at risk of acute encephalopathy was included in this prospective study. The EEG recordings of these neonates were independently reviewed for seizures by three international experts. The IOA was estimated using statistical measures including Fleiss' kappa and percentage agreement assessed over seizure events (event basis) and seizure duration (temporal basis).A total of 4066 h of EEG recordings from 70 neonates were reviewed with an average of 2555 seizures detected. The IOA was high with temporal assessment resulting in a kappa of 0.827 (95% CI: 0.769-0.865; n = 70). The median agreement was 83.0% (interquartile range [IQR]: 76.6-89.5%; n = 33) for seizure and 99.7% (IQR: 98.9-99.8%; n = 70) for nonseizure EEG. Analysis of events showed a median agreement of 83.0% (IQR: 72.9-86.6%; n = 33) for seizures with 0.018 disagreements per hour (IQR: 0.000-0.090 per hour; n = 70). Observers were more likely to disagree when a seizure was less than 30 sec. Overall, 33 neonates were diagnosed with seizures and 28 neonates were not, by all three observers. Of the remaining nine neonates with contradictory EEG detections, seven presented with low total seizure burden.The IOA is high among experts for the detection of neonatal seizures using conventional, multichannel EEG. Agreement is reduced when seizures are rare or have short duration. These findings support EEG-based decision making in the neonatal intensive care unit, inform EEG interpretation guidelines, and provide benchmarks for seizure detection algorithms.
| S-EPMC4693620 | biostudies-other