Inter-Coder Agreement in One-to-Many Classification: Fuzzy Kappa.
Ontology highlight
ABSTRACT:
SUBMITTER: Kirilenko AP
PROVIDER: S-EPMC4775035 | biostudies-other | 2016
REPOSITORIES: biostudies-other
Publications
Inter-Coder Agreement in One-to-Many Classification: Fuzzy Kappa.
Kirilenko Andrei P AP Stepchenkova Svetlana S
PloS one 20160302 3
Content analysis involves classification of textual, visual, or audio data. The inter-coder agreement is estimated by making two or more coders to classify the same data units, with subsequent comparison of their results. The existing methods of agreement estimation, e.g., Cohen's kappa, require that coders place each unit of content into one and only one category (one-to-one coding) from the pre-established set of categories. However, in certain data domains (e.g., maps, photographs, databases ...[more]
Similar Datasets
Project description:We assessed several agreement coefficients applied in 2x2 contingency tables, which are commonly applied in research due to dichotomization. Here, we not only studied some specific estimators but also developed a general method for the study of any estimator candidate to be an agreement measurement. This method was developed in open-source R codes and it is available to the researchers. We tested this method by verifying the performance of several traditional estimators over all possible configurations with sizes ranging from 1 to 68 (total of 1,028,789 tables). Cohen's kappa showed handicapped behavior similar to Pearson's r, Yule's Q, and Yule's Y. Scott's pi, and Shankar and Bangdiwala's B seem to better assess situations of disagreement than agreement between raters. Krippendorff's alpha emulates, without any advantage, Scott's pi in cases with nominal variables and two raters. Dice's F1 and McNemar's chi-squared incompletely assess the information of the contingency table, showing the poorest performance among all. We concluded that Cohen's kappa is a measurement of association and McNemar's chi-squared assess neither association nor agreement; the only two authentic agreement estimators are Holley and Guilford's G and Gwet's AC1. The latter two estimators also showed the best performance over the range of table sizes and should be considered as the first choices for agreement measurement in contingency 2x2 tables. All procedures and data were implemented in R and are available to download from Harvard Dataverse https://doi.org/10.7910/DVN/HMYTCK.
| S-EPMC10615996 | biostudies-literature
Project description:Sanitary risk inspection, an observation protocol for identifying contamination hazards around water sources, is promoted for managing rural water supply safety. However, it is unclear how far different observers consistently identify contamination hazards and consistently classify water source types using standard typologies. This study aimed to quantify inter-observer agreement in hazard identification and classification of rural water sources. Six observers separately visited 146 domestic water sources in Siaya County, Kenya, in wet and dry seasons. Each observer independently classified the source type and conducted a sanitary risk inspection using a standard protocol. Water source types assigned by an experienced observer were cross-tabulated against those of his colleagues, as were contamination hazards identified, and inter-observer agreement measures calculated. Agreement between hazards observed by the most experienced observer versus his colleagues was significant but low (intra-class correlation = 0.49), with inexperienced observers detecting fewer hazards. Inter-observer agreement in classifying water sources was strong (Cohen's kappa = 0.84). However, some source types were frequently misclassified, such as sources adapted to cope with water insecurity (e.g. tanks drawing on both piped and rainwater). Observers with limited training and experience thus struggle to consistently identify hazards using existing protocols, suggesting observation protocols require revision and their implementation should be supported by comprehensive training. Findings also indicate that field survey teams struggle to differentiate some water source types based on a standard water source classification, particularly sources adapted to cope with water insecurity. These findings demonstrate uncertainties underpinning international monitoring and analyses of safe water access via household surveys.
| S-EPMC7661424 | biostudies-literature
Project description:The aim of this study is to determine inter-laboratory variability on embryo assessment using time-lapse platform and conventional morphological assessment. This study compares the data obtained from a pilot study of external quality control (EQC) of time lapse, performed in 2014, with the classical EQC of the Spanish Society for the Study of Reproductive Biology (ASEBIR) performed in 2013 and 2014. In total, 24 laboratories (8 using EmbryoScope™, 15 using Primo Vision™ and one with both platforms) took part in the pilot study. The clinics that used EmbryoScope™ analysed 31 embryos and those using Primo Vision™ analysed 35. The classical EQC was implemented by 39 clinics, based on an analysis of 25 embryos per year. Both groups were required to evaluate various qualitative morphological variables (cell fragmentation, the presence of vacuoles, blastomere asymmetry and multinucleation), to classify the embryos in accordance with ASEBIR criteria and to stipulate the clinical decision taken. In the EQC time-lapse pilot study, the groups were asked to determine, as well as the above characteristics, the embryo development times, the number, opposition and size of pronuclei, the direct division of 1 into 3 cells and/or of 3 into 5 cells and false divisions. The degree of agreement was determined by calculating the intra-class correlation coefficients and the coefficient of variation for the quantitative variables and the Gwet index for the qualitative variables. For both EmbryoScope™ and Primo Vision™, two periods of greater inter-laboratory variability were observed in the times of embryo development events. One peak of variability was recorded among the laboratories addressing the first embryo events (extrusion of the second polar body and the appearance of pronuclei); the second peak took place between the times corresponding to the 8-cell and morula stages. In most of the qualitative variables analysed regarding embryo development, there was almost-perfect inter-laboratory agreement among conventional morphological assessment (CMA), EmbryoScope™ and Primo Vision™, except for false divisions, vacuoles and asymmetry (users of all methods) and multinucleation (users of Primo Vision™), where the degree of agreement was lower. The inter-laboratory agreement on embryo classification according to the ASEBIR criteria was moderate-substantial (Gwet 0.41-0.80) for the laboratories using CMA and EmbryoScope™, and fair-moderate (Gwet 0.21-0.60) for those using Primo Vision™. The inter-laboratory agreement for clinical decision was moderate (Gwet 0.41-0.60) on day 5 for CMA users and almost perfect (Gwet 0.81-1) for time-lapse users. In conclusion, time-lapse technology does not improve inter-laboratory agreement on embryo classification or the analysis of each morphological variable. Moreover, depending on the time-lapse platform used, inter-laboratory agreement may be lower than that obtained by CMA. However, inter-laboratory agreement on clinical decisions is improved with the use of time lapse, regardless of the platform used.
| S-EPMC5571938 | biostudies-literature
Project description:ObjectivesThe purpose of this investigation was to evaluate the inter- and intraobserver variability of the updated #Enzian classification of endometriosis on MRI and to evaluate the influence of reader experience on interobserver concordance.MethodsThis was a prospective single-center study. All patients were included who received an MRI of the pelvis for evaluation of endometriosis between March and July 2023 and who have provided written informed consent. Images were reviewed independently for endometriosis by three radiologists, utilizing the MRI-applicable categories of the #Enzian classification. Two radiologists had experience in pelvic MRI and endometriosis imaging. One radiologist had no specific experience in pelvic MRI and received a one-hour briefing beforehand.ResultsFifty consecutive patients (mean age, 34.9 years ±8.6 [standard deviation]) were prospectively evaluated. Interobserver agreement was excellent for diagnosis of deep infiltrating endometriosis (Fleiss' kappa: 0.89; 95% CI 0.73-1.00; p < 0.001) and endometriomas (Fleiss' kappa: 0.93; 95% CI 0.77-1.00; p < 0.001). For the experienced readers, interobserver agreement in the assessment of compartments A, B and C was excellent (κw ranging from 0.84; 95% CI 0.71-0.97; p < 0.001 to 0.89; 95% CI 0.82-0.97; p < 0.001). For the pairings of the experienced readers to the reader without specific experience in pelvic MRI, agreement was substantial to excellent (κw ranging from 0.64; 95% CI 0.44-0.85; p < 0.001 to 0.91; 95% CI 0.84-0.98; p < 0.001). Intraobserver variability was excellent for compartments A, B and C (κw ranging from 0.85; 95% CI 0.73-0.96; p < 0.001 to 0.95; 95% CI 0.89-1.00; p < 0.001).ConclusionWith sufficient experience, the #Enzian classification enables the achievement of excellent inter- and intraobserver agreement in MRI-based diagnosis of deep infiltrating endometriosis and endometriomas.
| S-EPMC10690940 | biostudies-literature
Project description:The Doha agreement classification is used to classify groin pain in athletes. We evaluated the inter-examiner reliability of this classification system. We prospectively recruited 48 male athletes (66 symptomatic sides) with groin pain between 10-2017 and 03-2020 at a sports medicine hospital in Qatar. Two examiners (23 and 10 years of clinical experience) performed history taking, and a standardized clinical examination blinded to each other's findings. Examiners classified groin pain using the Doha agreement terminology (adductor-, inguinal-, iliopsoas-, pubic-, hip-related groin pain, or other causes of groin pain). Multiple entities were ranked in order of perceived clinical importance. Each side was classified separately for bilateral groin pain. Inter-examiner reliability was calculated using Cohen's Kappa statistic (κ). Inter-examiner reliability was slight to moderate for adductor- (κ = 0.40), inguinal- (κ = 0.44), iliopsoas- (κ = 0.57), and pubic-related groin pain (κ = 0.12), substantial for hip-related groin pain (κ = 0.62), and slight for "other causes of groin pain" (κ = 0.13). Ranking entities in order of perceived clinical importance improved inter-examiner reliability for adductor-, inguinal-, and iliopsoas-related groin pain (κ = 0.52-0.65), but not for pubic (κ = 0.12), hip (κ = 0.51), and "other causes of groin pain" (κ = 0.03). For participants with unilateral groin pain classified with a single entity (n = 7), there was 100% agreement between the two examiners. Inter-examiner reliability of the Doha agreement meeting classification system varied from slight to substantial, depending on the clinical entity. Agreement between examiners was perfect when athletes were classified with a single clinical entity of groin pain, but lower when athletes were classified with multiple clinical entities.
| S-EPMC10092143 | biostudies-literature
Project description:The degree of inter-rater agreement is usually assessed through κ -type coefficients and the extent of agreement is then characterized by comparing the value of the adopted coefficient against a benchmark scale. Through two motivating examples, it is displayed the different behavior of some κ -type coefficients due to asymmetric distribution of marginal frequencies over categories. In order to investigate the robustness of four κ -type coefficients for nominal and ordinal classifications and of an inferential benchmarking procedure that, differently from straightforward benchmarking, does not neglect the influence of the experimental conditions, an extensive Monte Carlo simulation study has been conducted. The robustness has been investigated for several scenarios, differing for sample size, rating scale dimension, number of raters, frequency distribution of rater classifications, pattern of agreement across raters. Simulation results reveal an higher paradoxical behavior of Fleiss kappa and Conger kappa with ordinal rather than nominal classifications; the coefficients robustness improves with increasing sample size and number of raters for both nominal and ordinal classifications whereas robustness improves with rating scale dimension only for nominal classifications. By identifying the scenarios (ie, minimum sample size, number of raters, rating scale dimension) with acceptable robustness, this study provides guidelines about the design of robust agreement studies.
| S-EPMC9303799 | biostudies-literature
Project description:Screening and diagnostic procedures often require a physician's subjective interpretation of a patient's test result using an ordered categorical scale to define the patient's disease severity. Because of wide variability observed between physicians' ratings, many large-scale studies have been conducted to quantify agreement between multiple experts' ordinal classifications in common diagnostic procedures such as mammography. However, very few statistical approaches are available to assess agreement in these large-scale settings. Many existing summary measures of agreement rely on extensions of Cohen's kappa. These are prone to prevalence and marginal distribution issues, become increasingly complex for more than three experts, or are not easily implemented. Here we propose a model-based approach to assess agreement in large-scale studies based upon a framework of ordinal generalized linear mixed models. A summary measure of agreement is proposed for multiple experts assessing the same sample of patients' test results according to an ordered categorical scale. This measure avoids some of the key flaws associated with Cohen's kappa and its extensions. Simulation studies are conducted to demonstrate the validity of the approach with comparison with commonly used agreement measures. The proposed methods are easily implemented using the software package R and are applied to two large-scale cancer agreement studies.
| S-EPMC4560692 | biostudies-literature
Project description:In clinical studies, it is often of interest to see the diagnostic agreement among clinicians on certain symptoms. Previous work has focused on the agreement between two clinicians under two different conditions or the agreement among multiple clinicians under one condition. Few have discussed the agreement study with a design where multiple clinicians examine the same group of patients under two different conditions. In this paper, we use the intraclass kappa statistic for assessing nominal scale agreement with such a design. We derive an explicit variance formula for the difference of correlated kappa statistics and conduct hypothesis testing for the equality of kappa statistics. Simulation studies show that the method performs well with realistic sample sizes and may be superior to a method that did not take into account the measurement dependence structure. The practical utility of the method is illustrated on data from an eosinophilic esophagitis (EoE) study.
| S-EPMC9631095 | biostudies-literature
Project description:Researchers have long been interested in how social evaluations are made based upon first impressions of faces. It is also important to consider the level of agreement we see in such evaluations across raters and what this may tell us. Typically, high levels of inter-rater agreement for facial judgements are reported, but the measures used may be misleading. At present, studies commonly report Cronbach's α as a way to quantify agreement, although problematically, there are various issues with the use of this measure. Most importantly, because researchers treat raters as items, Cronbach's α is inflated by larger sample sizes even when agreement between raters is fixed. Here, we considered several alternative measures and investigated whether these better discriminate between traits that were predicted to show low (parental resemblance), intermediate (attractiveness, dominance, trustworthiness), and high (age, gender) levels of agreement. Importantly, the level of inter-rater agreement has not previously been studied for many of these traits. In addition, we investigated whether familiar faces resulted in differing levels of agreement in comparison with unfamiliar faces. Our results suggest that alternative measures may prove more informative than Cronbach's α when determining how well raters agree in their judgements. Further, we found no apparent influence of familiarity on levels of agreement. Finally, we show that, like attractiveness, both trustworthiness and dominance show significant levels of private taste (personal or idiosyncratic rater perceptions), although shared taste (perceptions shared with other raters) explains similar levels of variance in people's perceptions. In conclusion, we recommend that researchers investigating social judgements of faces consider alternatives to Cronbach's α but should also be prepared to examine both the potential value and origin of private taste as these might prove informative.
| S-EPMC6097668 | biostudies-literature
Project description:BackgroundRecent developments in diagnostic techniques for malaria, particularly DNA probes and sero-immunology, have raised questions as to how these techniques might be used to facilitate malaria diagnosis at the most peripheral levels of the primary health care system. At present, malaria diagnosis is based on the standard microscopic examination of blood films in most field epidemiologic studies and is likely to remain so in the immediate future in Africa. The objective of this study was to assess inter-observer agreement for the examination of Giemsa-stained slides for Plasmodium falciparum parasites.MethodsChildren aged 0 to 10 years were enrolled yearly in Bancoumana village (West Africa), mainly during the transmission season (June to October). The blood smears obtained from the persistently negative children in June 1996, August 1996, October 1996 and March 1997 were systematically re-examined. A stratified random sample (10%) proportional to the following parasite density classes 1-100, 101-5000, and 5001 and over was taken from the slides collected. The kappa statistics and the intra-class correlation were used as measures of agreement the first and the second slide examinations.ResultsThe weighted kappa statistic, widely used as a chance-corrected measure for nominal agreement, showed excellent inter-observer agreement (κ(w)=0.7926; 95% CI [0.7588, 0.8263]; p=0.01). The intra-class correlation co-efficient had the same value of 0.7926 confirming the appropriateness of the weighted kappa statistic. Inter-observer agreement for slides read as negative by one observer, or as containing more than 100 parasites per μl, was excellent: 97% (493/506) and 92% (145/158), respectively. In contrast, the inter-observer agreement for slides read by one observer as containing 1-100 parasites/μl was poor, 36% (96/268).ConclusionsIn field conditions in Mali, there was a high reproducibility for slides reported as negative or as having more than 100 parasites per μl. However, smears with readings of 1-100 parasites per μl were less reproducible and should be re-examined carefully.
| S-EPMC3849530 | biostudies-literature