Early CT changes in patients admitted for thrombectomy: Intrarater and interrater agreement.
Ontology highlight
ABSTRACT: Objective
To systematically review the literature and assess agreement on the Alberta Stroke Program Early CT Score (ASPECTS) among clinicians involved in the management of thrombectomy candidates.Methods
Studies assessing agreement using ASPECTS published from 2000 to 2015 were reviewed. Fifteen raters reviewed and scored the anonymized CT scans of 30 patients recruited in a local thrombectomy trial during 2 independent sessions, in order to study intrarater and interrater agreement. Agreement was measured using intraclass correlation coefficients (ICCs) and Fleiss kappa statistics for ASPECTS and dichotomized ASPECTS at various cutoff values.Results
The review yielded 30 articles reporting 40 measures of agreement. Populations, methods, analyses, and results were heterogeneous (slight to excellent agreement), precluding a meta-analysis. When analyzed as a categorical variable, intrarater agreement was slight to moderate (κ = 0.042-0.469); it reached a substantial level (κ > 0.6) in 11/15 raters when the score was dichotomized (0-5 vs 6-10). The interrater ICCs varied between 0.672 and 0.811, but agreement was slight to moderate (κ = 0.129-0.315). Even in the best of cases, when ASPECTS was dichotomized as 0-5 vs 6-10, interrater agreement did not reach a substantial level (κ = 0.561), which translates into at least 5 of 15 raters not giving the same dichotomized verdict in 15% of patients.Conclusions
In patients considered for thrombectomy, there may be insufficient agreement between clinicians for ASPECTS to be reliably used as a criterion for treatment decisions.
SUBMITTER: Farzin B
PROVIDER: S-EPMC4955274 | biostudies-literature |
REPOSITORIES: biostudies-literature
Similar Datasets
Project description:PURPOSE:Many neonates undergo electroencephalogram (EEG) monitoring to identify and manage acute symptomatic seizures. Information about brain function contained in the EEG background data may also help predict neurobehavioral outcomes. For EEG background features to be useful as prognostic indicators, the interpretation of these features must be standardized across electroencephalographers. We aimed at determining the interrater and intrarater agreement among electroencephalographers interpreting neonatal EEG background patterns. METHODS:Five neonatal electroencephalographers reviewed 5-to-7.5-minute epochs of EEG from full-term neonates who underwent continuous conventional EEG monitoring. The EEG assessment tool used to classify background patterns was based on the American Clinical Neurophysiology Society's guideline for neonatal EEG terminology. Interrater and intrarater agreement were measured using Kappa coefficients. RESULTS:Interrater agreement was consistently highest for voltage (binary: substantial, kappa = 0.783; categorical: moderate, kappa = 0.562), seizure presence (fair-substantial; kappa = 0.375-0.697), continuity (moderate; kappa = 0.481), burst voltage (moderate; kappa = 0.574), suppressed background presence (moderate-substantial; kappa = 0.493-0.643), delta activity presence (fair-moderate; kappa = 0.369-0.432), theta activity presence (fair-moderate; kappa = 0.347-0.600), presence of graphoelements (fair; kappa = 0.381), and overall impression (binary: moderate, kappa = 0.495; categorical: fair-moderate, kappa = 0.347, 0.465). Agreement was poor or inconsistent for all other patterns. Intrarater agreement was variable, with highest average agreement for voltage (binary: substantial, kappa = 0.75; categorical: substantial, kappa = 0.714) and highest consistent agreement for continuity (moderate-substantial; kappa = 0.43-0.67) and overall impression (moderate-substantial; kappa = 0.42-0.68). CONCLUSIONS:This study demonstrates substantial variability in neonatal EEG background interpretation across electroencephalographers, indicating a need for educational and technological strategies aimed at improving performance.
| S-EPMC6322680 | biostudies-literature
Project description:BACKGROUND:The aim of this study was to assess the inter- and intrarater reliability of noncontrast CT (NCCT) markers [Black Hole Sign (BH), Blend Sign (BS), Island Sign (IS), and Hypodensities (HD)] and Spot Sign (SS) on CTA in patients with spontaneous intracerebral hemorrhage (ICH). METHODS:Patients with spontaneous ICH at three German tertiary stroke centers were retrospectively included. Each CT scan was rated for four NCCT markers and SS on CTA by two radiology residents. Raters were blind to all demographic and outcome data. Inter- and intrarater agreement was determined by Cohen's kappa (?) coefficient and percentage of agreement. RESULTS:Interrater agreement was excellent in 473 included patients, ranging from 96% to 99%. Interrater ? ranged from 0.85 (95% CI [0.78-0.91]) to 0.97 (95% CI [0.94-0.99]) for NCCT markers and 0.93 (95% CI [0.88-0.98]) for SS, all p-values < 0.001. Intrarrater agreement ranged from 96% to 100%, with ? ranging from 0.85 (95% CI [0.78-0.91]) to 1.00 (95% CI [0.10-0.85]) for NCCT markers and 0.96 (95% CI [0.92-1.00]) for SS, all p-values < 0.001. CONCLUSIONS:NCCT imaging findings and SS on CTA have good-to-excellent inter- and intrarater reliabilities, with the highest agreement for BH and SS.
| S-EPMC7231301 | biostudies-literature
Project description:Anatomists and radiologists use the Zaidi-Dayal and Richards-Jabbour scales to study the shape of the foramen magnum. Our aim is to measure the interrater and intrarater agreement and reliability of ratings made using the two scales. We invited 16 radiology residents to attend two sessions, four weeks apart. During each session, we asked the residents to classify the shape of the foramen magnum in 35 images using both scales. We used Fleiss' κ to measure interrater reliability and Cohen's κ to measure intrarater reliability. The interrater reliability of ratings made using the Zaidi-Dayal scale was 0.34 (0.26-0.46) for session one and 0.30 (0.24-0.39) for session two, and the intrarater reliability was 0.39 (0.34-0.44). The interrater reliability of ratings made using the Richards-Jabbour scale was 0.14 (0.10-0.19) for session one and 0.12 (0.09-0.17) for session two, and the intrarater reliability was 0.11 (0.07-0.15). In conclusion, the interrater and intrarater agreement and reliability of ratings made using the Zaidi-Dayal and Richards-Jabbour scales are inadequate. We recommend an objective method by Zdilla et al. to researchers interested in studying the shape of the foramen magnum.
| S-EPMC6895155 | biostudies-literature
Project description:BackgroundThe vestibular evoked myogenic potential triggered by galvanic vestibular stimulation (galvanic-VEMP) has been used to assess the function of the vestibulospinal motor tract and is a candidate biomarker to predict and monitor the human T-cell lymphotropic virus type 1 (HTLV-1) associated myelopathy (HAM). This study determined the agreement and reliability of this exam.MethodsGalvanic-VEMP was performed in 96 participants, of which 24 patients presented HAM, 27 HTLV-1-asymptomatic carriers, and 45 HTLV-1-negative asymptomatic controls. Galvanic vestibular stimulation was achieved by passing a binaural and bipolar current at a 2 milliamperes (mA) intensity for 400 milliseconds (ms) between the mastoid processes. Galvanic-VEMP electromyographic wave responses of short latency (SL) and medium latency (ML) were recorded from the gastrocnemius muscle. Intrarater (test-retest) and interrater (two independent examiners) agreement and reliability were assessed by standard error of measurement (SEM), coefficient of repeatability (CR), intraclass correlation coefficient (ICC), and Kappa coefficient.ResultsIn the total sample (n = 96), SL and ML medians were 56 ms (IQR 52-66) and 120 ms (IQR 107-130), respectively. The intrarater repeatability measures for SL and ML were, respectively: SEM of 6 and 8 ms; CR of 16 and 22 ms; ICC of 0.80 (p<0.001) and 0.91 (p<0.001); and a Kappa coefficient of 0.53 (p<0.001) and 0.82 (p<0.001). The interrater reproducibility measures for SL and ML were, respectively: SEM of 3 and 10 ms; CR of 8 and 27 ms; ICC of 0.95 (p<0.001) and 0.86 (p<0.001); and a Kappa coefficient of 0.77 (p<0.001) and 0.88 (p<0.001).ConclusionGalvanic-VEMP is a reliable and reproducible method to define the integrity of the vestibulospinal tract. Longitudinal studies will clarify its validity in the clinical context, aimed at achieving an early diagnosis and the monitoring of HAM.
| S-EPMC6160040 | biostudies-literature
Project description:BackgroundGuidelines recommend the use of perfusion computed tomography (CT) to identify emergent large vessel ischemic stroke (ELVIS) patients who are likely to benefit from endovascular thrombectomy (EVT) if they present within 6-24 hour (late window) of stroke onset. We aim to determine if the interrater and intrarater reliability among physicians when recommending EVT is significantly different when perfusion CT or non-perfusion CT is reviewed.MethodsA total of 30 non-consecutive patients will be selected from our institutional database comprising 3144 cranial CT scans performed for acute stroke symptoms January 2018 to August 2022. The clinical and radiologic data of the 30 patients will be presented in random order to a group of 29 physicians in two separate sessions at least three weeks apart. In each session, the physicians will evaluate each patient once with automated perfusion images and once without. We will use non-overlapping 95% confidence intervals and difference in agreement classification as criteria to suggest a difference between the Gwet AC1 statistics (κG).DiscussionThe results obtained from this study, combined with the clinical outcomes data of patients categorized through the two imaging techniques and a cost-effectiveness analysis, will offer a comprehensive evaluation of the clinical utility of perfusion CT neuroimaging. Should there be no significant disparity in the reliability of decisions made by clinicians using the two neuroimaging protocols, it may be necessary to revise existing recommendations regarding neuroimaging in the later time window to align with these findings.
| S-EPMC10826946 | biostudies-literature
Project description:Clinical evaluation of scapular dyskinesis (SD) aims to identify abnormal scapulothoracic movement, underlying causal factors, and the potential relationship with shoulder symptoms. The literature proposes different methods of dynamic clinical evaluation of SD, but improved reliability and agreement values are needed. The present study aimed to evaluate the intrarater and interrater agreement and reliability of three SD classifications: 1) 4-type classification, 2) Yes/No classification, and 3) scapular dyskinesis test (SDT). Seventy-five young athletes, including 45 men and 30 women, were evaluated. Raters evaluated the SD based on the three methods during one series of 8-10 cycles (at least eight and maximum of ten) of forward flexion and abduction with an external load under the observation of two raters trained to diagnose SD. The evaluation protocol was repeated after 3 h for intrarater analysis. The agreement percentage was calculated by dividing the observed agreement by the total number of observations. Reliability was calculated using Cohen Kappa coefficient, with a 95% confidence interval (CI), defined by Kappa coefficient ±1.96 multiplied by the measurement standard error. The interrater analyses showed an agreement percentage between 80% and 95.9% and an almost perfect reliability (k>0.81) for the three classification methods in all the test conditions, except the 4-type and SDT classification methods, which had substantial reliability (k<0.80) in shoulder abduction. Intrarater analyses showed agreement percentages between 80.7% and 89.3% and substantial reliability (0.67 to 0.81) for both raters in the three classifications. CIs ranged from moderate to almost perfect categories. This indicates that the three SD classification methods investigated in this study showed high reliability values for both intrarater and interrater evaluation throughout a protocol that provided SD evaluation training of raters and included several repetitions of arm movements with external load during a live assessment.
| S-EPMC5531566 | biostudies-literature
Project description:Background and purposeDiffusion tensor imaging (DTI) has become a valuable tool in both the research and clinical evaluation of subjects. We sought to quantify interobserver and intraobserver variability of diffusivity and diffusion anisotropy measurements with regard to specific regions of interest (ROIs).Materials and methodsThe subject group consisted of 5 healthy control subjects and 7 study subjects (all males; 16-19 years old; mean age = 17.5 years), as part of a protocol for closed head injury. Two whole-brain DTI scans were acquired on a 3T scanner for each subject. Analysis was performed using a ROI approach. Two independent observers analyzed the apparent diffusion coefficient (ADC) and fractional anisotropy (FA) indices in the corpus callosum, cortical spinal tract, internal capsules (ICs), basal ganglia, and centrum semiovale (CSO). Intraobserver and interobserver variability were calculated for the mean ADC, FA, and ordered eigenvalues of the diffusion tensor (lambda(1), lambda(2), and lambda(3)).ResultsThe overall kappa statistic for intraobserver variability for both observers showed slight-to-substantial agreement (kappa = 0.02-0.69), however FA values in the CSO showed only slight agreement. Interobserver agreement was also slight to substantial for these DTI measurements with high variability in FA values in the IC and CSO.ConclusionsWhen one is comparing 2 DTI measurements, it is important to assess intraobserver and interobserver variability. We recommend caution in the analysis of DTI contrasts in the IC and CSO, because we have found the widest range of variability in measurements within these structures.
| S-EPMC8118842 | biostudies-literature
Project description:The interpretation of critical care electroencephalography (EEG) studies is challenging because of the presence of many periodic and rhythmic patterns of uncertain clinical significance. Defining the clinical significance of these patterns requires standardized terminology with high interrater agreement (IRA). We sought to evaluate IRA for the final, published American Clinical Neurophysiology Society (ACNS)-approved version of the critical care EEG terminology (2012 version). Our evaluation included terms not assessed previously and incorporated raters with a broad range of EEG reading experience.After reviewing a set of training slides, 49 readers independently completed a Web-based test consisting of 11 identical questions for each of 37 EEG samples (407 questions). Questions assessed whether a pattern was an electrographic seizure; pattern location (main term 1), pattern type (main term 2); and presence and classification of eight other key features ("plus" modifiers, sharpness, absolute and relative amplitude, frequency, number of phases, fluctuation/evolution, and the presence of "triphasic" morphology).IRA statistics (? values) were almost perfect (90-100%) for seizures, main terms 1 and 2, the +S modifier (superimposed spikes/sharp waves or sharply contoured rhythmic delta activity), sharpness, absolute amplitude, frequency, and number of phases. Agreement was substantial for the +F (superimposed fast activity) and +R (superimposed rhythmic delta activity) modifiers (66% and 67%, respectively), moderate for triphasic morphology (58%), and fair for evolution (21%).IRA for most terms in the ACNS critical care EEG terminology is high. These terms are suitable for multicenter research on the clinical significance of critical care EEG patterns. A PowerPoint slide summarizing this article is available for download in the Supporting Information section http://dx.doi.org/10.1111/epi.12653/supinfo.
| S-EPMC4879939 | biostudies-literature
Project description:OBJECTIVE:The presence of interictal epileptiform discharges (IED) in the electroencephalogram (EEG) is a key finding in the medical workup of a patient with suspected epilepsy. However, inter-rater agreement (IRA) regarding the presence of IED is imperfect, leading to incorrect and delayed diagnoses. An improved understanding of which IED attributes mediate expert IRA might help in developing automatic methods for IED detection able to emulate the abilities of experts. Therefore, using a set of IED scored by a large number of experts, we set out to determine which attributes of IED predict expert agreement regarding the presence of IED. METHODS:IED were annotated on a 5-point scale by 18 clinical neurophysiologists within 200 30-s EEG segments from recordings of 200 patients. 5538 signal analysis features were extracted from the waveforms, including wavelet coefficients, morphological features, signal energy, nonlinear energy operator response, electrode location, and spectrogram features. Feature selection was performed by applying elastic net regression and support vector regression (SVR) was applied to predict expert opinion, with and without the feature selection procedure and with and without several types of signal normalization. RESULTS:Multiple types of features were useful for predicting expert annotations, but particular types of wavelet features performed best. Local EEG normalization also enhanced best model performance. As the size of the group of EEGers used to train the models was increased, the performance of the models leveled off at a group size of around 11. CONCLUSIONS:The features that best predict inter-rater agreement among experts regarding the presence of IED are wavelet features, using locally standardized EEG. Our models for predicting expert opinion based on EEGer's scores perform best with a large group of EEGers (more than 10). SIGNIFICANCE:By examining a large group of EEG signal analysis features we found that wavelet features with certain wavelet basis functions performed best to identify IEDs. Local normalization also improves predictability, suggesting the importance of IED morphology over amplitude-based features. Although most IED detection studies in the past have used opinion from three or fewer experts, our study suggests a "wisdom of the crowd" effect, such that pooling over a larger number of expert opinions produces a better correlation between expert opinion and objectively quantifiable features of the EEG.
| S-EPMC5842710 | biostudies-literature
Project description:BackgroundNasal endoscopy is an important part of the clinical evaluation of patients with chronic rhinosinusitis. However, the objectivity and interrater agreement of the procedure related findings have not been well studied, especially in patients who have previously had sinus surgery.MethodsPatients with a history of endoscopic sinus surgery for chronic rhinosinusitis were prospectively enrolled from a tertiary rhinology practice. Fourteen endoscopic nasal examinations were recorded using digital video capture software. Each patient also underwent computed tomography (CT) and completed the Sinonasal Outcome Test (SNOT-22). Blinded review of inflammatory and anatomic findings for each video was independently performed by 5 academic rhinologists at separate institutions. Comparisons were performed using the unweighted Fleiss' kappa statistic (K(f) ) and the prevalence- and bias-adjusted kappa (PABAK).ResultsThere were no significant correlations between age, Lund-Mackay score, or SNOT-22 score. Interrater agreement was variable across the characteristics studied. Mean PABAK was excellent for the assessment of polyps (K(f) = 0.886); moderate for the assessments of middle turbinate (MT) integrity (K(f) = 0.543), MT position (K(f) = 0.443), maxillary sinus patency (K(f) = 0.593), and ethmoid sinus patency (K(f) = 0.429); fair for discharge (K(f) = 0.314), synechiae (K(f) = 0.257), and middle meatus patency (K(f) = 0.229); and poor for MT mucosal changes (K(f) = 0.148) and uncinate process (K(f) = 0.126).ConclusionThis study was notable for variability in the interrater agreement among the inflammatory and anatomic attributes that were examined. Further standardization of nasal endoscopy with regard to interpretation may improve the reliability of this procedure in clinical practice.
| S-EPMC3448826 | biostudies-literature