Interobserver Agreement Among Uveitis Experts on Uveitic Diagnoses: The Standardization of Uveitis Nomenclature Experience.
Ontology highlight
ABSTRACT: Purpose
To evaluate the interobserver agreement among uveitis experts on the diagnosis of the specific uveitic disease.Design
Interobserver agreement analysis.Methods
Five committees, each comprised of 9 individuals and working in parallel, reviewed cases from a preliminary database of 25 uveitic diseases, collected by disease, and voted independently online whether the case was the disease in question or not. The agreement statistic, κ, was calculated for the 36 pairwise comparisons for each disease, and a mean κ was calculated for each disease. After the independent online voting, committee consensus conference calls, using nominal group techniques, reviewed all cases not achieving supermajority agreement (>75%) on the diagnosis in the online voting to attempt to arrive at a supermajority agreement.Results
A total of 5766 cases for the 25 diseases were evaluated. The overall mean κ for the entire project was 0.39, with disease-specific variation ranging from 0.23 to 0.79. After the formalized consensus conference calls to address cases that did not achieve supermajority agreement in the online voting, supermajority agreement overall was reached on approximately 99% of cases, with disease-specific variation ranging from 96% to 100%.Conclusions
Agreement among uveitis experts on diagnosis is moderate at best but can be improved by discussion among them. These data suggest the need for validated and widely used classification criteria in the field of uveitis.
SUBMITTER: Jabs DA
PROVIDER: S-EPMC8489730 | biostudies-literature |
REPOSITORIES: biostudies-literature
Similar Datasets
Project description:BackgroundGiven the recent increased focus on evidence-based medicine, it is critical that diseases and syndromes have accurate and complete descriptions, including standardized and widely accepted terminologies. Standardizing these descriptions and terminologies is necessary to develop tools such as computerized data entry forms and classification criteria. This need is especially true for diseases that are relatively uncommon, such as uveitis.ObjectivesTo develop a standardized and internationally accepted terminology for the field of uveitis.MethodsThe Standardization of Uveitis Nomenclature (SUN) Working Group (WG) is an international group of 79 uveitis experts from 18 countries and 62 clinical centers. Initial terminology was developed utilizing a "modified" green field approach, which was enhanced through web-based surveys and teleconferences via a "modified" Delphi technique. Terms were mapped provisionally into ontologic dimensions for each syndrome. The Working Group then met and utilized nominal group techniques as a formalized method of finalizing the mappings.ResultsMapping of terms into dimensions to describe 28 major uveitic diseases was confirmed using nominal group techniques (achieving super-majority consensus) for each of the diseases at a meeting of the entire WG.ConclusionsThe SUN WG utilized an informatics-based approach to develop a standardized and internationally accepted terminology for the uveitides.
| S-EPMC8728398 | biostudies-literature
Project description:PurposeTo evaluate the interobserver reproducibility of two-dimensional shear wave elastography (2D-SWE) in measuring liver stiffness (LS) and to investigate factors related to liver 2D-SWE.Materials and methodsA prospective study was conducted between August 2011 and August 2012 in rheumatoid arthritis patients who had been treated with methotrexate. Interobserver reproducibility of 2D-SWE was evaluated, and the relationship between interobserver difference in LS and related factors was analyzed using linear regression analyses. We considered age, sex, alanine transaminase, cholesterol, body mass index (BMI), and waist circumference as clinical factors, and the mean value of standard deviation (SDM), its difference between two examiners, mean diameter of the regions of interest (ROIM), and its difference in the elasticity map as investigation factors. The cut-off value for significant factors to predict interobserver discrepancies in LS-based fibrosis stage was also inspected.ResultsIn total, 176 patients were enrolled. The intraclass correlation coefficient between the two examiners was 0.784. In the univariate analysis, SDM and ROIM were independently associated with interobserver differences in LS as well as BMI, waist circumference, and the difference of ROI, but SDM and ROIM were the only ones significantly related in multivariate analysis (p<0.001 and p = 0.021, respectively). The best cut-off value for SDM in predicting interobserver discrepancy in LS-based fibrosis stage was 1.4.ConclusionsInterobserver reproducibility of 2D-SWE for measuring LS was good and SDM was the most significantly associated factor with interobserver differences in LS and interobserver discordance in LS-based fibrosis stage.
| S-EPMC5393601 | biostudies-literature
Project description:BackgroundLung ultrasound (LUS) has received considerable interest in the clinical evaluation of patients with COVID-19. Previously described LUS manifestations for COVID-19 include B-lines, consolidations, and pleural thickening. The interrater reliability (IRR) of these findings for COVID-19 is unknown.MethodsThis study was conducted between March and June 2020. Nine physicians (hospitalists: n = 4; emergency medicine: n = 5) from 3 medical centers independently evaluated n = 20 LUS scans (n = 180 independent observations) collected from patients with COVID-19, diagnosed via RT-PCR. These studies were randomly selected from an image database consisting of COVID-19 patients evaluated in the emergency department with portable ultrasound devices. Physicians were blinded to any patient information or previous LUS interpretation. Kappa values (κ) were used to calculate IRR.ResultsThere was substantial IRR on the following items: normal LUS scan (κ = 0.79 [95% CI: 0.72-0.87]), presence of B-lines (κ = 0.79 [95% CI: 0.72-0.87]), ≥3 B-lines observed (κ = 0.72 [95% CI: 0.64-0.79]). Moderate IRR was observed for the presence of any consolidation (κ = 0.57 [95% CI: 0.50-0.64]), subpleural consolidation (κ = 0.49 [95% CI: 0.42-0.56]), and presence of effusion (κ = 0.49 [95% CI: 0.41-0.56]). Fair IRR was observed for pleural thickening (κ = 0.23 [95% CI: 0.15-0.30]).DiscussionMany LUS manifestations for COVID-19 appear to have moderate to substantial IRR across providers from multiple specialties utilizing differing portable devices. The most reliable LUS findings with COVID-19 may include the presence/count of B-lines or determining if a scan is normal. Clinical protocols for LUS with COVID-19 may require additional observers for the confirmation of less reliable findings such as consolidations.
| S-EPMC8013417 | biostudies-literature
Project description:Observers differ in their judgment while assessing physical signs in a patient. We had undertaken a goitre prevalence survey amongst school students in a Rural Health Training Centre, Pune district (Maharashtra) during October 1992. Four teams of trained observers were used for detection of goitre. This study was undertaken to estimate the extent and acceptability of interobserver agreement amongst the four teams. Observer variation/agreement was measured by two methods viz. kappa coefficient and proportion of agreement. The proportion of agreement appears to be a better measure of observer agreement as it could make a distinction between normality (absence of goitre) and abnormality (presence of goitre). In the present study, the proportion of agreement for abnormality ranged between 0.62 - 0.83. This measure was considered as indicating a good interobserver agreement in detecting goitre in the survey that was undertaken.
| S-EPMC5529923 | biostudies-other
Project description:To determine the interobserver agreement (IOA) of neonatal seizure detection using the gold standard of conventional, multichannel EEG.A cohort of full-term neonates at risk of acute encephalopathy was included in this prospective study. The EEG recordings of these neonates were independently reviewed for seizures by three international experts. The IOA was estimated using statistical measures including Fleiss' kappa and percentage agreement assessed over seizure events (event basis) and seizure duration (temporal basis).A total of 4066 h of EEG recordings from 70 neonates were reviewed with an average of 2555 seizures detected. The IOA was high with temporal assessment resulting in a kappa of 0.827 (95% CI: 0.769-0.865; n = 70). The median agreement was 83.0% (interquartile range [IQR]: 76.6-89.5%; n = 33) for seizure and 99.7% (IQR: 98.9-99.8%; n = 70) for nonseizure EEG. Analysis of events showed a median agreement of 83.0% (IQR: 72.9-86.6%; n = 33) for seizures with 0.018 disagreements per hour (IQR: 0.000-0.090 per hour; n = 70). Observers were more likely to disagree when a seizure was less than 30 sec. Overall, 33 neonates were diagnosed with seizures and 28 neonates were not, by all three observers. Of the remaining nine neonates with contradictory EEG detections, seven presented with low total seizure burden.The IOA is high among experts for the detection of neonatal seizures using conventional, multichannel EEG. Agreement is reduced when seizures are rare or have short duration. These findings support EEG-based decision making in the neonatal intensive care unit, inform EEG interpretation guidelines, and provide benchmarks for seizure detection algorithms.
| S-EPMC4693620 | biostudies-other
Project description:Ultrasonography is the best available tool for the initial work-up of thyroid nodules. Substantial interobserver variability has been documented in the recognition and reporting of some of the lesion characteristics. A number of classification systems have been developed to estimate the likelihood of malignancy: several of them have been endorsed by scientific societies, but their reproducibility is yet to be assessed. We evaluated the interobserver variability of the AACE/ACE/AME, ACR, ATA, EU-TIRADS and K-TIRADS classification systems and the interobserver concordance in the indication to FNA biopsy. Two raters independently evaluated 1055 ultrasound images of thyroid nodules identified in 265 patients at multiple time points, in two separate sets (501 and 554 images). After the first set of nodules, a joint reading was performed to reach a consensus in the feature definitions. The interobserver agreement (Krippendorff alpha) in the first set of nodules was 0.47, 0.49, 0.49, 0.61 and 0.53, for AACE/ACE/AME, ACR, ATA, EU-TIRADS and K-TIRADS systems, respectively. The agreement for the indication to biopsy was substantial to near-perfect, being 0.73, 0.61, 0.75, 0.68 and 0.82, respectively (Cohen's kappa). For all systems, agreement on the nodules of the second set increased. Despite the wide variability in the description of single ultrasonographic features, the classification systems may improve the interobserver agreement that further ameliorates after a specific training. When selecting nodules to be submitted to FNA biopsy, that is main purpose of these classifications, the interobserver agreement is substantial to almost perfect.
| S-EPMC5744624 | biostudies-literature
Project description:AimTo assess the role of imaging in the early management of encephalitis and the agreement on findings in a well-defined cohort of suspected encephalitis cases enrolled in the Prospective Aetiological Study of Encephalitis conducted by the Health Protection Agency (now incorporated into Public Health England).Materials and methodsEighty-five CT examinations from 68 patients and 101 MRI examinations from 80 patients with suspected encephalitis were independently rated by three neuroradiologists blinded to patient and clinical details. The level of agreement on the interpretation of images was measured using the kappa statistic. The sensitivity, specificity, and negative and positive predictive values of CT and MRI for herpes simplex virus (HSV) encephalitis and acute disseminated encephalomyelitis (ADEM) were estimated.ResultsThe kappa value for interobserver agreement on rating the scans as normal or abnormal was good (0.65) for CT and moderate (0.59) for MRI. Agreement for HSV encephalitis was very good for CT (0.87) and MRI (0.82), but only fair for ADEM (0.32 CT; 0.31 MRI). Similarly, the overall sensitivity of imaging for HSV encephalitis was ∼80% for both CT and MRI, whereas for ADEM it was 0% for CT and 20% for MRI. MRI specificity for HSV encephalitis between 3-10 days after symptom onset was 100%.ConclusionThere is a subjective component to scan interpretation that can have important implications for the clinical management of encephalitis cases. Neuroradiologists were good at diagnosing HSV encephalitis; however, agreement was worse for ADEM and other alternative aetiologies. Findings highlight the importance of a comprehensive and multidisciplinary approach to diagnosing the cause of encephalitis that takes into account individual clinical, microbiological, and radiological features of each patient.
| S-EPMC5021199 | biostudies-literature
Project description:Neisseria meningitidis isolates are conventionally classified by serosubtyping, which characterizes the reactivities of the PorA outer membrane protein variable-region (VR) epitopes with monoclonal antibodies (MAbs). A newer method (PorA VR typing) uses predicted amino acid sequences derived from DNA sequence analysis. The resulting classification schemes are not standardized, offering conflicting and sometimes irreconcilable data from the two methods. In this paper, we propose a standardization of the PorA VR typing nomenclature that incorporates serologic information from traditional PorA serosubtyping with molecular data from predicted VR sequences. We performed a comprehensive literature and database search, generating a collection of strains and DNA sequences that reflects the diversity within PorA that exists to date. We have arranged this information in a comprehensive logical model that includes both serosubtype and PorA VR type assignments. Our data demonstrate that the current panel of serosubtype-defining MAbs underestimates PorA VR variability by at least 50%. Our proposal for VR typing is informative because amino acid sequence and serologic information, when serosubtype-defining MAbs are available, can be deduced simultaneously from the PorA VR designation. This scheme will be useful in future classification and applied epidemiologic studies of N. meningitidis, being a systematic way of selecting PorA vaccine candidates and analyzing vaccine coverage and failure.
| S-EPMC96214 | biostudies-literature
Project description:The aim of this study is to evaluate intra- and interobserver agreement for measurement of intracranial, cerebellar, and thalamic volume with the Virtual Organ Computer-aided AnaLysis (VOCAL) technique in three-dimensional ultrasound images, in comparison to two-dimensional measurements of these brain structures. Three-dimensional ultrasound images of the brains of 80 fetuses at 20-24 weeks' gestational age were obtained from YOUth, a Dutch prospective cohort study. Two observers performed offline measurement of the occipitofrontal diameter, intracranial volume, transcerebellar diameter, cerebellar volume, and thalamic width, area, and volume, independently. VOCAL was used for calculation of the volumes. The two-way random, single measures intraclass correlation coefficient (ICC) was used for analysis of agreement and Bland-Altman plots were configured. Intra- and interobserver agreement was almost perfect for occipitofrontal diameter (intra ICC 0.88, 95% CI 0.82-0.92; inter ICC 0.91, 95% CI 0.85-0.94), intracranial volume (intra ICC 0.96, 95% CI 0.91-0.98; inter ICC 0.97, 95% CI 0.96-0.98) and transcerebellar diameter (intra ICC 0.91, 95% CI 0.86-0.94; inter ICC 0.86, 95% CI 0.78-0.910). For cerebellar volume, the intraobserver agreement was almost perfect (0.85, 95% CI 0.76-0.90), whereas the interobserver agreement was substantial (0.75, 95% CI 0.44-0.88). Agreement was only moderate for thalamic measurements. Bland-Altman plots for the volume measurements are normally distributed with acceptable mean differences and 95% limits of agreement. The intra- and interobserver agreement of the measurement of intracranial and cerebellar volume with VOCAL was almost perfect. These measurements are therefore reliable, and can be used to investigate fetal brain development. Thalamic measurements are not reliable enough.
| S-EPMC6866380 | biostudies-literature
Project description:ObjectiveThe objective was to assess the interobserver agreement rate, progression rates and malignancy rates in the assessment of complex renal cysts (≥ Bosniak IIF) using a population-based database.MethodsA regional database identified 452 complex renal cysts in 415 patients between 2009 and 2019. Each patient was tracked and followed up using a unique identifier and deterministic linkage methodology. The interobserver agreement rate between radiologists was calculated using a weighted kappa statistic. Progression and malignancy rates of cysts (Bosniak ≥IIF) over the 11-year period were calculated.ResultsThe linear-weighted kappa value was 0.69 for all complex cysts. The rate of progression and regression of Bosniak IIF cysts was 4.6% (7/151) and 3.3% (5/151), respectively. All malignant IIF cysts progressed within 16 months of diagnosis. The malignancy rate of surgically resected Bosniak III and IV cysts was 79.3% (23/29) and 84.5% (39/46), respectively. Of all malignant tumours, 73.8% and 93.7% were of low ISUP grade and low stage, respectively.ConclusionsThis study further confirms that there is a good degree of agreement between radiologists in classifying complex renal masses using the Bosniak classification. The progression rate of Bosniak IIF cysts is low, but the malignancy rates of surgically resected Bosniak IIF, III and IV cysts are high. Benign cysts are frequently resected, and a very high proportion of histopathologically confirmed cancers in complex renal cysts are of low grade and stage.Key points• There is a good degree of agreement between radiologists in classifying complex renal masses using the Bosniak classification. • The rate of progression of Bosniak IIF cysts is low, and malignant cysts progress early during surveillance. Although the malignancy rates of resected Bosniak IIF, III and IV cysts are high, the rate of benign cyst resection is significant.
| S-EPMC7813744 | biostudies-literature