Inter-Coder Agreement in One-to-Many Classification: Fuzzy Kappa.
Ontology highlight
ABSTRACT:
SUBMITTER: Kirilenko AP
PROVIDER: S-EPMC4775035 | biostudies-other | 2016
REPOSITORIES: biostudies-other
Publications
Inter-Coder Agreement in One-to-Many Classification: Fuzzy Kappa.
Kirilenko Andrei P AP Stepchenkova Svetlana S
PloS one 20160302 3
Content analysis involves classification of textual, visual, or audio data. The inter-coder agreement is estimated by making two or more coders to classify the same data units, with subsequent comparison of their results. The existing methods of agreement estimation, e.g., Cohen's kappa, require that coders place each unit of content into one and only one category (one-to-one coding) from the pre-established set of categories. However, in certain data domains (e.g., maps, photographs, databases ...[more]
Similar Datasets
Project description:Sanitary risk inspection, an observation protocol for identifying contamination hazards around water sources, is promoted for managing rural water supply safety. However, it is unclear how far different observers consistently identify contamination hazards and consistently classify water source types using standard typologies. This study aimed to quantify inter-observer agreement in hazard identification and classification of rural water sources. Six observers separately visited 146 domestic water sources in Siaya County, Kenya, in wet and dry seasons. Each observer independently classified the source type and conducted a sanitary risk inspection using a standard protocol. Water source types assigned by an experienced observer were cross-tabulated against those of his colleagues, as were contamination hazards identified, and inter-observer agreement measures calculated. Agreement between hazards observed by the most experienced observer versus his colleagues was significant but low (intra-class correlation?=?0.49), with inexperienced observers detecting fewer hazards. Inter-observer agreement in classifying water sources was strong (Cohen's kappa?=?0.84). However, some source types were frequently misclassified, such as sources adapted to cope with water insecurity (e.g. tanks drawing on both piped and rainwater). Observers with limited training and experience thus struggle to consistently identify hazards using existing protocols, suggesting observation protocols require revision and their implementation should be supported by comprehensive training. Findings also indicate that field survey teams struggle to differentiate some water source types based on a standard water source classification, particularly sources adapted to cope with water insecurity. These findings demonstrate uncertainties underpinning international monitoring and analyses of safe water access via household surveys.
| S-EPMC7661424 | biostudies-literature
Project description:The aim of this study is to determine inter-laboratory variability on embryo assessment using time-lapse platform and conventional morphological assessment. This study compares the data obtained from a pilot study of external quality control (EQC) of time lapse, performed in 2014, with the classical EQC of the Spanish Society for the Study of Reproductive Biology (ASEBIR) performed in 2013 and 2014. In total, 24 laboratories (8 using EmbryoScope™, 15 using Primo Vision™ and one with both platforms) took part in the pilot study. The clinics that used EmbryoScope™ analysed 31 embryos and those using Primo Vision™ analysed 35. The classical EQC was implemented by 39 clinics, based on an analysis of 25 embryos per year. Both groups were required to evaluate various qualitative morphological variables (cell fragmentation, the presence of vacuoles, blastomere asymmetry and multinucleation), to classify the embryos in accordance with ASEBIR criteria and to stipulate the clinical decision taken. In the EQC time-lapse pilot study, the groups were asked to determine, as well as the above characteristics, the embryo development times, the number, opposition and size of pronuclei, the direct division of 1 into 3 cells and/or of 3 into 5 cells and false divisions. The degree of agreement was determined by calculating the intra-class correlation coefficients and the coefficient of variation for the quantitative variables and the Gwet index for the qualitative variables. For both EmbryoScope™ and Primo Vision™, two periods of greater inter-laboratory variability were observed in the times of embryo development events. One peak of variability was recorded among the laboratories addressing the first embryo events (extrusion of the second polar body and the appearance of pronuclei); the second peak took place between the times corresponding to the 8-cell and morula stages. In most of the qualitative variables analysed regarding embryo development, there was almost-perfect inter-laboratory agreement among conventional morphological assessment (CMA), EmbryoScope™ and Primo Vision™, except for false divisions, vacuoles and asymmetry (users of all methods) and multinucleation (users of Primo Vision™), where the degree of agreement was lower. The inter-laboratory agreement on embryo classification according to the ASEBIR criteria was moderate-substantial (Gwet 0.41-0.80) for the laboratories using CMA and EmbryoScope™, and fair-moderate (Gwet 0.21-0.60) for those using Primo Vision™. The inter-laboratory agreement for clinical decision was moderate (Gwet 0.41-0.60) on day 5 for CMA users and almost perfect (Gwet 0.81-1) for time-lapse users. In conclusion, time-lapse technology does not improve inter-laboratory agreement on embryo classification or the analysis of each morphological variable. Moreover, depending on the time-lapse platform used, inter-laboratory agreement may be lower than that obtained by CMA. However, inter-laboratory agreement on clinical decisions is improved with the use of time lapse, regardless of the platform used.
| S-EPMC5571938 | biostudies-literature
Project description:ObjectivesThe purpose of this investigation was to evaluate the inter- and intraobserver variability of the updated #Enzian classification of endometriosis on MRI and to evaluate the influence of reader experience on interobserver concordance.MethodsThis was a prospective single-center study. All patients were included who received an MRI of the pelvis for evaluation of endometriosis between March and July 2023 and who have provided written informed consent. Images were reviewed independently for endometriosis by three radiologists, utilizing the MRI-applicable categories of the #Enzian classification. Two radiologists had experience in pelvic MRI and endometriosis imaging. One radiologist had no specific experience in pelvic MRI and received a one-hour briefing beforehand.ResultsFifty consecutive patients (mean age, 34.9 years ±8.6 [standard deviation]) were prospectively evaluated. Interobserver agreement was excellent for diagnosis of deep infiltrating endometriosis (Fleiss' kappa: 0.89; 95% CI 0.73-1.00; p < 0.001) and endometriomas (Fleiss' kappa: 0.93; 95% CI 0.77-1.00; p < 0.001). For the experienced readers, interobserver agreement in the assessment of compartments A, B and C was excellent (κw ranging from 0.84; 95% CI 0.71-0.97; p < 0.001 to 0.89; 95% CI 0.82-0.97; p < 0.001). For the pairings of the experienced readers to the reader without specific experience in pelvic MRI, agreement was substantial to excellent (κw ranging from 0.64; 95% CI 0.44-0.85; p < 0.001 to 0.91; 95% CI 0.84-0.98; p < 0.001). Intraobserver variability was excellent for compartments A, B and C (κw ranging from 0.85; 95% CI 0.73-0.96; p < 0.001 to 0.95; 95% CI 0.89-1.00; p < 0.001).ConclusionWith sufficient experience, the #Enzian classification enables the achievement of excellent inter- and intraobserver agreement in MRI-based diagnosis of deep infiltrating endometriosis and endometriomas.
| S-EPMC10690940 | biostudies-literature
Project description:The degree of inter-rater agreement is usually assessed through κ -type coefficients and the extent of agreement is then characterized by comparing the value of the adopted coefficient against a benchmark scale. Through two motivating examples, it is displayed the different behavior of some κ -type coefficients due to asymmetric distribution of marginal frequencies over categories. In order to investigate the robustness of four κ -type coefficients for nominal and ordinal classifications and of an inferential benchmarking procedure that, differently from straightforward benchmarking, does not neglect the influence of the experimental conditions, an extensive Monte Carlo simulation study has been conducted. The robustness has been investigated for several scenarios, differing for sample size, rating scale dimension, number of raters, frequency distribution of rater classifications, pattern of agreement across raters. Simulation results reveal an higher paradoxical behavior of Fleiss kappa and Conger kappa with ordinal rather than nominal classifications; the coefficients robustness improves with increasing sample size and number of raters for both nominal and ordinal classifications whereas robustness improves with rating scale dimension only for nominal classifications. By identifying the scenarios (ie, minimum sample size, number of raters, rating scale dimension) with acceptable robustness, this study provides guidelines about the design of robust agreement studies.
| S-EPMC9303799 | biostudies-literature
Project description:Screening and diagnostic procedures often require a physician's subjective interpretation of a patient's test result using an ordered categorical scale to define the patient's disease severity. Because of wide variability observed between physicians' ratings, many large-scale studies have been conducted to quantify agreement between multiple experts' ordinal classifications in common diagnostic procedures such as mammography. However, very few statistical approaches are available to assess agreement in these large-scale settings. Many existing summary measures of agreement rely on extensions of Cohen's kappa. These are prone to prevalence and marginal distribution issues, become increasingly complex for more than three experts, or are not easily implemented. Here we propose a model-based approach to assess agreement in large-scale studies based upon a framework of ordinal generalized linear mixed models. A summary measure of agreement is proposed for multiple experts assessing the same sample of patients' test results according to an ordered categorical scale. This measure avoids some of the key flaws associated with Cohen's kappa and its extensions. Simulation studies are conducted to demonstrate the validity of the approach with comparison with commonly used agreement measures. The proposed methods are easily implemented using the software package R and are applied to two large-scale cancer agreement studies.
| S-EPMC4560692 | biostudies-literature
Project description:BackgroundRecent developments in diagnostic techniques for malaria, particularly DNA probes and sero-immunology, have raised questions as to how these techniques might be used to facilitate malaria diagnosis at the most peripheral levels of the primary health care system. At present, malaria diagnosis is based on the standard microscopic examination of blood films in most field epidemiologic studies and is likely to remain so in the immediate future in Africa. The objective of this study was to assess inter-observer agreement for the examination of Giemsa-stained slides for Plasmodium falciparum parasites.MethodsChildren aged 0 to 10 years were enrolled yearly in Bancoumana village (West Africa), mainly during the transmission season (June to October). The blood smears obtained from the persistently negative children in June 1996, August 1996, October 1996 and March 1997 were systematically re-examined. A stratified random sample (10%) proportional to the following parasite density classes 1-100, 101-5000, and 5001 and over was taken from the slides collected. The kappa statistics and the intra-class correlation were used as measures of agreement the first and the second slide examinations.ResultsThe weighted kappa statistic, widely used as a chance-corrected measure for nominal agreement, showed excellent inter-observer agreement (?(w)=0.7926; 95% CI [0.7588, 0.8263]; p=0.01). The intra-class correlation co-efficient had the same value of 0.7926 confirming the appropriateness of the weighted kappa statistic. Inter-observer agreement for slides read as negative by one observer, or as containing more than 100 parasites per ?l, was excellent: 97% (493/506) and 92% (145/158), respectively. In contrast, the inter-observer agreement for slides read by one observer as containing 1-100 parasites/?l was poor, 36% (96/268).ConclusionsIn field conditions in Mali, there was a high reproducibility for slides reported as negative or as having more than 100 parasites per ?l. However, smears with readings of 1-100 parasites per ?l were less reproducible and should be re-examined carefully.
| S-EPMC3849530 | biostudies-literature
Project description:Researchers have long been interested in how social evaluations are made based upon first impressions of faces. It is also important to consider the level of agreement we see in such evaluations across raters and what this may tell us. Typically, high levels of inter-rater agreement for facial judgements are reported, but the measures used may be misleading. At present, studies commonly report Cronbach's ? as a way to quantify agreement, although problematically, there are various issues with the use of this measure. Most importantly, because researchers treat raters as items, Cronbach's ? is inflated by larger sample sizes even when agreement between raters is fixed. Here, we considered several alternative measures and investigated whether these better discriminate between traits that were predicted to show low (parental resemblance), intermediate (attractiveness, dominance, trustworthiness), and high (age, gender) levels of agreement. Importantly, the level of inter-rater agreement has not previously been studied for many of these traits. In addition, we investigated whether familiar faces resulted in differing levels of agreement in comparison with unfamiliar faces. Our results suggest that alternative measures may prove more informative than Cronbach's ? when determining how well raters agree in their judgements. Further, we found no apparent influence of familiarity on levels of agreement. Finally, we show that, like attractiveness, both trustworthiness and dominance show significant levels of private taste (personal or idiosyncratic rater perceptions), although shared taste (perceptions shared with other raters) explains similar levels of variance in people's perceptions. In conclusion, we recommend that researchers investigating social judgements of faces consider alternatives to Cronbach's ? but should also be prepared to examine both the potential value and origin of private taste as these might prove informative.
| S-EPMC6097668 | biostudies-literature
Project description:Deep learning has been successfully applied to many classification problems including underwater challenges. However, a long-standing issue with deep learning is the need for large and consistently labeled datasets. Although current approaches in semi-supervised learning can decrease the required amount of annotated data by a factor of 10 or even more, this line of research still uses distinct classes. For underwater classification, and uncurated real-world datasets in general, clean class boundaries can often not be given due to a limited information content in the images and transitional stages of the depicted objects. This leads to different experts having different opinions and thus producing fuzzy labels which could also be considered ambiguous or divergent. We propose a novel framework for handling semi-supervised classifications of such fuzzy labels. It is based on the idea of overclustering to detect substructures in these fuzzy labels. We propose a novel loss to improve the overclustering capability of our framework and show the benefit of overclustering for fuzzy labels. We show that our framework is superior to previous state-of-the-art semi-supervised methods when applied to real-world plankton data with fuzzy labels. Moreover, we acquire 5 to 10% more consistent predictions of substructures.
| S-EPMC8512301 | biostudies-literature
Project description:As gene expression profile data from DNA microarrays accumulate rapidly, there is a natural need to compare data across labs and platforms. Comparisons of microarray data can be quite challenging due to data complexity and variability. Different labs may adopt different technology platforms. One may ask about the degree of agreement we can expect from different labs and different platforms. To address this question, we conducted a study of inter-lab and inter-platform agreement of microarray data across three platforms and three labs. The statistical measures of consistency and agreement used in this paper are the Pearson correlation, intraclass correlation, kappa coefficients, and a measure of intra-transcript correlation. The three platforms used in the present paper were Affymetrix GeneChip, custom cDNA arrays, and custom oligo arrays. Using the within-platform variability as a benchmark, we found that these technology platforms exhibited an acceptable level of agreement, but the agreement between two technologies within the same lab was greater than that between two labs using the same technology. The consistency of replicates in each experiment varies from lab to lab. When there is high consistency among replicates, different technologies show good agreement within and across labs using the same RNA samples. On the other hand, the lab effect, especially when confounded with the RNA sample effect, plays a bigger role than the platform effect on data agreement.
| S-EPMC1142313 | biostudies-literature
Project description:The mammalian target of rapamycin (mTOR) signaling pathway is a master regulator of cell growth and metabolism. Deregulation of the mTOR pathway has been implicated in a number of human diseases such as cancer, diabetes, obesity, neurological diseases, and genetic disorders. Rapamycin, a specific inhibitor of mTOR, has been shown to be useful in the treatment of certain diseases. Here we discuss its mechanism of action and highlight recent findings regarding the effects and limitations of rapamycin monotherapy and the potential utility of combination therapy with rapamycin.
| S-EPMC3972801 | biostudies-literature