Browse
Submit Data
Databases
API
Help

Dataset Information

0 Views

0 Connections

0 Citations

0 Reanalyses

0 Downloads

Omics score: 0

Asymptotic variability of (multilevel) multirater kappa coefficients.

ABSTRACT: Agreement studies are of paramount importance in various scientific domains. When several observers classify objects on categorical scales, agreement can be quantified through multirater kappa coefficients. In most statistical packages, the standard error of these coefficients is only available under the null hypothesis that the coefficient is equal to zero, preventing the construction of confidence intervals in the general case. The aim of this paper is triple. First, simple analytic formulae for the standard error of multirater kappa coefficients will be given in the general case. Second, these formulae will be extended to the case of multilevel data structures. The formulae are based on simple matrix algebra and are implemented in the R package "multiagree". Third, guidelines on the choice between the different mulitrater kappa coefficients will be provided.

SUBMITTER: Vanbelle S

PROVIDER: S-EPMC6745615 | biostudies-literature | 2019 Oct-Nov

REPOSITORIES: biostudies-literature

ACCESS DATA

Json Xml

Publications

Asymptotic variability of (multilevel) multirater kappa coefficients.

Vanbelle Sophie S

Statistical methods in medical research 20180822 10-11

Agreement studies are of paramount importance in various scientific domains. When several observers classify objects on categorical scales, agreement can be quantified through multirater kappa coefficients. In most statistical packages, the standard error of these coefficients is only available under the null hypothesis that the coefficient is equal to zero, preventing the construction of confidence intervals in the general case. The aim of this paper is triple. First, simple analytic formulae f ...[more]

PMID: 30132375

Similar Datasets

Comparing dependent kappa coefficients obtained on multilevel data.

Project description:Reliability and agreement are two notions of paramount importance in medical and behavioral sciences. They provide information about the quality of the measurements. When the scale is categorical, reliability and agreement can be quantified through different kappa coefficients. The present paper provides two simple alternatives to more advanced modeling techniques, which are not always adequate in case of a very limited number of subjects, when comparing several dependent kappa coefficients obtained on multilevel data. This situation frequently arises in medical sciences, where multilevel data are common. Dependent kappa coefficients can result from the assessment of the same individuals at various occasions or when each member of a group is compared to an expert, for example. The method is based on simple matrix calculations and is available in the R package "multiagree". Moreover, the statistical properties of the proposed method are studied using simulations. Although this paper focuses on kappa coefficients, the method easily extends to other statistical measures.

| S-EPMC5600130 | biostudies-literature

Robustness of κ -type coefficients for clinical agreement.

Project description:The degree of inter-rater agreement is usually assessed through κ -type coefficients and the extent of agreement is then characterized by comparing the value of the adopted coefficient against a benchmark scale. Through two motivating examples, it is displayed the different behavior of some κ -type coefficients due to asymmetric distribution of marginal frequencies over categories. In order to investigate the robustness of four κ -type coefficients for nominal and ordinal classifications and of an inferential benchmarking procedure that, differently from straightforward benchmarking, does not neglect the influence of the experimental conditions, an extensive Monte Carlo simulation study has been conducted. The robustness has been investigated for several scenarios, differing for sample size, rating scale dimension, number of raters, frequency distribution of rater classifications, pattern of agreement across raters. Simulation results reveal an higher paradoxical behavior of Fleiss kappa and Conger kappa with ordinal rather than nominal classifications; the coefficients robustness improves with increasing sample size and number of raters for both nominal and ordinal classifications whereas robustness improves with rating scale dimension only for nominal classifications. By identifying the scenarios (ie, minimum sample size, number of raters, rating scale dimension) with acceptable robustness, this study provides guidelines about the design of robust agreement studies.

| S-EPMC9303799 | biostudies-literature

Asymptotic Standard Errors of Parameter Scale Transformation Coefficients in Test Equating Under the Nominal Response Model.

Project description:Researchers have developed a characteristic curve procedure to estimate the parameter scale transformation coefficients in test equating under the nominal response model. In the study, the delta method was applied to derive the standard error expressions for computing the standard errors for the estimates of the parameter scale transformation coefficients. This brief report presents the results of a simulation study that examined the accuracy of the derived formulas and compared the performance of this analytical method with that of the multiple imputation method. The results indicated that the standard errors produced by the delta method were very close to the criterion standard errors as well as those yielded by the multiple imputation method under all the simulation conditions.

| S-EPMC7876637 | biostudies-literature

Population variability under stressors is dependent on body mass growth and asymptotic body size.

Project description:The recruitment and biomass of a fish stock are influenced by their environmental conditions and anthropogenic pressures such as fishing. The variability in the environment often translates into fluctuations in recruitment, which then propagate throughout the stock biomass. In order to manage fish stocks sustainably, it is necessary to understand their dynamics. Here, we systematically explore the dynamics and sensitivity of fish stock recruitment and biomass to environmental noise. Using an age-structured and trait-based model, we explore random noise (white noise) and autocorrelated noise (red noise) in combination with low to high levels of harvesting. We determine the vital rates of stocks covering a wide range of possible body mass (size) growth rates and asymptotic size parameter combinations. Our study indicates that the variability of stock recruitment and biomass are probably correlated with the stock's asymptotic size and growth rate. We find that fast-growing and large-sized fish stocks are likely to be less vulnerable to disturbances than slow-growing and small-sized fish stocks. We show how the natural variability in fish stocks is amplified by fishing, not just for one stock but for a broad range of fish life histories.

| S-EPMC7062104 | biostudies-literature

Multilevel superposition for deciphering the conformational variability of protein ensembles.

Project description:The dynamics and variability of protein conformations are directly linked to their functions. Many comparative studies of X-ray protein structures have been conducted to elucidate the relevant conformational changes, dynamics and heterogeneity. The rapid increase in the number of experimentally determined structures has made comparison an effective tool for investigating protein structures. For example, it is now possible to compare structural ensembles formed by enzyme species, variants or the type of ligands bound to them. In this study, the author developed a multilevel model for estimating two covariance matrices that represent inter- and intra-ensemble variability in the Cartesian coordinate space. Principal component analysis using the two estimated covariance matrices identified the inter-/intra-enzyme variabilities, which seemed to be important for the enzyme functions, with the illustrative examples of cytochrome P450 family 2 enzymes and class A $\beta$-lactamases. In P450, in which each enzyme has its own active site of a distinct size, an active-site motion shared universally between the enzymes was captured as the first principal mode of the intra-enzyme covariance matrix. In this case, the method was useful for understanding the conformational variability after adjusting for the differences between enzyme sizes. The developed method is advantageous in small ensemble-size problems and hence promising for use in comparative studies on experimentally determined structures where ensemble sizes are smaller than those generated, for example, by molecular dynamics simulations.

| S-EPMC10983786 | biostudies-literature

The logarithmic contributions to the [Formula: see text] asymptotic massive Wilson coefficients and operator matrix elements in deeply inelastic scattering.

Project description:We calculate the logarithmic contributions to the massive Wilson coefficients for deep-inelastic scattering in the asymptotic region [Formula: see text] to 3-loop order in the fixed flavor number scheme and present the corresponding expressions for the massive operator matrix elements needed in the variable flavor number scheme. Explicit expressions are given in Mellin [Formula: see text]-space.

| S-EPMC4404500 | biostudies-other

Variability of renal apparent diffusion coefficients: limitations of the monoexponential model for diffusion quantification.

Project description:To investigate whether variability in reported renal apparent diffusion coefficient (ADC) values in literature can be explained by the use of different diffusion weightings (b values) and the use of a monoexponential model to calculate ADC.This prospective study was approved by institutional review board and was HIPAA-compliant, and all subjects gave written informed consent. Diffusion-weighted (DW) imaging of the kidneys was performed in three healthy volunteers to generate reference diffusion decay curves. In a literature meta-analysis, the authors resampled the reference curves at the various b values used in 19 published studies of normal kidneys (reported ADC = [2.0-4.1] x 10(-3) mm(2) / sec for cortex and [1.9-5.1] x 10(-3) mm(2) / sec for medulla) and then fitted the resampled signals by monoexponential model to produce "predicted" ADC. Correlation plots were used to compare the predicted ADC values with the published values obtained with the same b values.Significant correlation was found between the reported and predicted ADC values for whole renal parenchyma (R(2) = 0.50, P = .002), cortex (R(2) = 0.87, P = .0002), and medulla (R(2) = 0.61, P = .0129), indicating that most of the variability in reported ADC values arises from limitations of a monoexponential model and use of different b values.The use of a monoexponential function for DW imaging analysis and variably sampled diffusion weighting plays a substantial role in causing the variability in ADC of healthy kidneys. For maximum reliability in renal apparent diffusion coefficient quantification, data for monoexponential analysis should be acquired at a fixed set of b values or a biexponential model should be used. (c) RSNA, 2010.

| S-EPMC2851010 | biostudies-other

Variability in Palliative Care Use after Intracerebral Hemorrhage at US Hospitals: A Multilevel Analysis.

Project description:BACKGROUND:Palliative care (PC) is an essential component of comprehensive care of patients with intracerebral hemorrhage (ICH). In the present study, we sought to characterize the variability of PC use after ICH among US hospitals. METHODS:ICH admissions from hospitals with at least 12 annual ICH cases were identified in the Nationwide Inpatient Sample between 2008 and 2011. We used multilevel logistic regression modeling to estimate between-hospital variance in PC use. We calculated the intraclass correlation coefficient (ICC), proportional variance change, and median OR after accounting for individual-level and hospital-level covariates. RESULTS:Among 26,791 ICH admissions, 12.5% received PC (95% CI 11.5-13.5). Among the 629 included hospitals, the median rate of PC use was 9.1 (interquartile range 1.5-19.3) per 100 ICH admissions, and 150 (23.9%) hospitals had no recorded PC use. The ICC of the random intercept (null) model was 0.274, suggesting that 27.4% of the overall variability in PC use was due to between-hospital variability. Adding hospital-level covariates to the model accounted for 25.8% of the between-hospital variance observed in the null model, with 74.2% of between-hospital variance remaining unexplained. The median OR of the fully adjusted model was 2.62 (95% CI 2.41-2.89), indicating that a patient moving from 1 hospital to another with a higher intrinsic propensity of PC use had a 2.63-fold median increase in the odds of receiving PC, independent of patient and hospital factors. CONCLUSIONS:Substantial variation in PC use after ICH exists among US hospitals. A substantial proportion of this between-hospital variability remains unexplained even after accounting for patient and hospital characteristics.

| S-EPMC6698400 | biostudies-literature

Tips for learners of evidence-based medicine: 3. Measures of observer variability (kappa statistic).

Project description: Not available

| S-EPMC527344 | biostudies-literature

Stability and robustness of asymptotic autocatalytic systems.

Project description:Here, we address the consequences of the extension in the space of a simple model of a system that is closed to efficient causation: the (M,R)-system model. To do so, we use a diffusion term to describe the collective motion of the nutrients' concentration across the compartmentalized space that defines the organism. We show that the non-trivial stable steady state remains despite such generalization, as long as the system is small enough to deal with the transport of the precursors to feed the entire protocell and dispose of a sufficient concentration of it in its surroundings. Such consideration explains the emergence of a bifurcation with two parameters that we characterize. Finally, we show that the robustness of the system under catastrophic losses of catalysts also remains, preserving the original's model character.

| S-EPMC7511346 | biostudies-literature

OmicsDI is part of the ELIXIR infrastructure

OmicsDI is an Elixir interoperability service. Learn more ›

Tweets

OmicsDI Databases

PRIDE
PeptideAtlas
MassIVE
JPOST Repository
Physiome Model Repository

EGA
EVA
ENA
LINCS
PAXDB
Cell Collective

MetaboLights
Metabolomics Workbench
MetabolomeExpress
GNPS
BioModels
FAIRDOMHub

ArrayExpress
dbGaP
ExpressionAtlas
GEO
NODE

Information

Databases
Help
API
Contact us
Code on GitHub
Terms of use
Submit Data