Dataset Information

Interrater Reliability of Experts in Identifying Interictal Epileptiform Discharges in Electroencephalograms.

ABSTRACT: Importance:The validity of using electroencephalograms (EEGs) to diagnose epilepsy requires reliable detection of interictal epileptiform discharges (IEDs). Prior interrater reliability (IRR) studies are limited by small samples and selection bias. Objective:To assess the reliability of experts in detecting IEDs in routine EEGs. Design, Setting, and Participants:This prospective analysis conducted in 2 phases included as participants physicians with at least 1 year of subspecialty training in clinical neurophysiology. In phase 1, 9 experts independently identified candidate IEDs in 991 EEGs (1 expert per EEG) reported in the medical record to contain at least 1 IED, yielding 87?636 candidate IEDs. In phase 2, the candidate IEDs were clustered into groups with distinct morphological features, yielding 12?602 clusters, and a representative candidate IED was selected from each cluster. We added 660 waveforms (11 random samples each from 60 randomly selected EEGs reported as being free of IEDs) as negative controls. Eight experts independently scored all 13?262 candidates as IEDs or non-IEDs. The 1051 EEGs in the study were recorded at the Massachusetts General Hospital between 2012 and 2016. Main Outcomes and Measures:Primary outcome measures were percentage of agreement (PA) and beyond-chance agreement (Gwet ?) for individual IEDs (IED-wise IRR) and for whether an EEG contained any IEDs (EEG-wise IRR). Secondary outcomes were the correlations between numbers of IEDs marked by experts across cases, calibration of expert scoring to group consensus, and receiver operating characteristic analysis of how well multivariate logistic regression models may account for differences in the IED scoring behavior between experts. Results:Among the 1051 EEGs assessed in the study, 540 (51.4%) were those of females and 511 (48.6%) were those of males. In phase 1, 9 experts each marked potential IEDs in a median of 65 (interquartile range [IQR], 28-332) EEGs. The total number of IED candidates marked was 87?636. Expert IRR for the 13?262 individually annotated IED candidates was fair, with the mean PA being 72.4% (95% CI, 67.0%-77.8%) and mean ? being 48.7% (95% CI, 37.3%-60.1%). The EEG-wise IRR was substantial, with the mean PA being 80.9% (95% CI, 76.2%-85.7%) and mean ? being 69.4% (95% CI, 60.3%-78.5%). A statistical model based on waveform morphological features, when provided with individualized thresholds, explained the median binary scores of all experts with a high degree of accuracy of 80% (range, 73%-88%). Conclusions and Relevance:This study's findings suggest that experts can identify whether EEGs contain IEDs with substantial reliability. Lower reliability regarding individual IEDs may be largely explained by various experts applying different thresholds to a common underlying statistical model.

SUBMITTER: Jing J

PROVIDER: S-EPMC6806666 | biostudies-literature | 2020 Jan

REPOSITORIES: biostudies-literature

ACCESS DATA

Publications

Interrater Reliability of Experts in Identifying Interictal Epileptiform Discharges in Electroencephalograms.

Jing Jin J Herlopian Aline A Karakis Ioannis I Ng Marcus M Halford Jonathan J JJ Lam Alice A Maus Douglas D Chan Fonda F Dolatshahi Marjan M Muniz Carlos F CF Chu Catherine C Sacca Valeria V Pathmanathan Jay J Ge WenDong W Sun Haoqi H Dauwels Justin J Cole Andrew J AJ Hoch Daniel B DB Cash Sydney S SS Westover M Brandon MB

JAMA neurology 20200101 1

<h4>Importance</h4>The validity of using electroencephalograms (EEGs) to diagnose epilepsy requires reliable detection of interictal epileptiform discharges (IEDs). Prior interrater reliability (IRR) studies are limited by small samples and selection bias.<h4>Objective</h4>To assess the reliability of experts in detecting IEDs in routine EEGs.<h4>Design, setting, and participants</h4>This prospective analysis conducted in 2 phases included as participants physicians with at least 1 year of subsp ...[more]

PMID: 31633742

Similar Datasets

Project description:OBJECTIVE:The presence of interictal epileptiform discharges (IED) in the electroencephalogram (EEG) is a key finding in the medical workup of a patient with suspected epilepsy. However, inter-rater agreement (IRA) regarding the presence of IED is imperfect, leading to incorrect and delayed diagnoses. An improved understanding of which IED attributes mediate expert IRA might help in developing automatic methods for IED detection able to emulate the abilities of experts. Therefore, using a set of IED scored by a large number of experts, we set out to determine which attributes of IED predict expert agreement regarding the presence of IED. METHODS:IED were annotated on a 5-point scale by 18 clinical neurophysiologists within 200 30-s EEG segments from recordings of 200 patients. 5538 signal analysis features were extracted from the waveforms, including wavelet coefficients, morphological features, signal energy, nonlinear energy operator response, electrode location, and spectrogram features. Feature selection was performed by applying elastic net regression and support vector regression (SVR) was applied to predict expert opinion, with and without the feature selection procedure and with and without several types of signal normalization. RESULTS:Multiple types of features were useful for predicting expert annotations, but particular types of wavelet features performed best. Local EEG normalization also enhanced best model performance. As the size of the group of EEGers used to train the models was increased, the performance of the models leveled off at a group size of around 11. CONCLUSIONS:The features that best predict inter-rater agreement among experts regarding the presence of IED are wavelet features, using locally standardized EEG. Our models for predicting expert opinion based on EEGer's scores perform best with a large group of EEGers (more than 10). SIGNIFICANCE:By examining a large group of EEG signal analysis features we found that wavelet features with certain wavelet basis functions performed best to identify IEDs. Local normalization also improves predictability, suggesting the importance of IED morphology over amplitude-based features. Although most IED detection studies in the past have used opinion from three or fewer experts, our study suggests a "wisdom of the crowd" effect, such that pooling over a larger number of expert opinions produces a better correlation between expert opinion and objectively quantifiable features of the EEG.

Project description:ObjectiveTo evaluate the diagnostic performance of artificial intelligence (AI)-based algorithms for identifying the presence of interictal epileptiform discharges (IEDs) in routine (20-min) electroencephalography (EEG) recordings.MethodsWe evaluated two approaches: a fully automated one and a hybrid approach, where three human raters applied an operational IED definition to assess the automated detections grouped into clusters by the algorithms. We used three previously developed AI algorithms: Encevis, SpikeNet, and Persyst. The diagnostic gold standard (epilepsy or not) was derived from video-EEG recordings of patients' habitual clinical episodes. We compared the algorithms with the gold standard at the recording level (epileptic or not). The independent validation data set (not used for training) consisted of 20-min EEG recordings containing sharp transients (epileptiform or not) from 60 patients: 30 with epilepsy (with a total of 340 IEDs) and 30 with nonepileptic paroxysmal events. We compared sensitivity, specificity, overall accuracy, and the review time-burden of the fully automated and hybrid approaches, with the conventional visual assessment of the whole recordings, based solely on unrestricted expert opinion.ResultsFor all three AI algorithms, the specificity of the fully automated approach was too low for clinical implementation (16.67%; 63.33%; 3.33%), despite the high sensitivity (96.67%; 66.67%; 100.00%). Using the hybrid approach significantly increased the specificity (93.33%; 96.67%; 96.67%) with good sensitivity (93.33%; 56.67%; 76.67%). The overall accuracy of the hybrid methods (93.33%; 76.67%; 86.67%) was similar to the conventional visual assessment of the whole recordings (83.33%; 95% confidence interval [CI]: 71.48-91.70%; p > .5), yet the time-burden of review was significantly lower (p < .001).SignificanceThe hybrid approach, where human raters apply the operational IED criteria to automated detections of AI-based algorithms, has high specificity, good sensitivity, and overall accuracy similar to conventional EEG reading, with a significantly lower time-burden. The hybrid approach is accurate and suitable for clinical implementation.

Project description:ObjectiveThis study was undertaken to evaluate the influence that subject-specific factors have on intracranial interictal epileptiform discharge (IED) rates in persons with refractory epilepsy.MethodsOne hundred fifty subjects with intracranial electrodes performed multiple sessions of a free recall memory task; this standardized task controlled for subject attention levels. We utilized a dominance analysis to rank the importance of subject-specific factors based on their relative influence on IED rates. Linear mixed-effects models were employed to comprehensively examine factors with highly ranked importance.ResultsAntiseizure medication (ASM) status, time of testing, and seizure onset zone (SOZ) location were the highest-ranking factors in terms of their impact on IED rates. The average IED rate of electrodes in SOZs was 34% higher than the average IED rate of electrodes outside of SOZs (non-SOZ; p < .001). However, non-SOZ electrodes had similar IED rates regardless of the subject's SOZ location (p = .99). Subjects on older generation (p < .001) and combined generation (p < .001) ASM regimens had significantly lower IED rates relative to the group taking no ASMs; newer generation ASM regimens demonstrated a nonsignificant association with IED rates (p = .13). Of the ASMs included in this study, the following ASMs were associated with significant reductions in IED rates: levetiracetam (p < .001), carbamazepine (p < .001), lacosamide (p = .03), zonisamide (p = .01), lamotrigine (p = .03), phenytoin (p = .03), and topiramate (p = .01). We observed a nonsignificant association between time of testing and IED rates (morning-afternoon p = .15, morning-evening p = .85, afternoon-evening p = .26).SignificanceThe current study ranks the relative influence that subject-specific factors have on IED rates and highlights the importance of considering certain factors, such as SOZ location and ASM status, when analyzing IEDs for clinical or research purposes.

Dataset Information

Interrater Reliability of Experts in Identifying Interictal Epileptiform Discharges in Electroencephalograms.

Publications

Interrater Reliability of Experts in Identifying Interictal Epileptiform Discharges in Electroencephalograms.

Similar Datasets

OmicsDI is part of the ELIXIR infrastructure

Tweets