Dataset Information

Comparison of confidence interval methods for an intra-class correlation coefficient (ICC).

ABSTRACT:

Background

The intraclass correlation coefficient (ICC) is widely used in biomedical research to assess the reproducibility of measurements between raters, labs, technicians, or devices. For example, in an inter-rater reliability study, a high ICC value means that noise variability (between-raters and within-raters) is small relative to variability from patient to patient. A confidence interval or Bayesian credible interval for the ICC is a commonly reported summary. Such intervals can be constructed employing either frequentist or Bayesian methodologies.

Methods

This study examines the performance of three different methods for constructing an interval in a two-way, crossed, random effects model without interaction: the Generalized Confidence Interval method (GCI), the Modified Large Sample method (MLS), and a Bayesian method based on a noninformative prior distribution (NIB). Guidance is provided on interval construction method selection based on study design, sample size, and normality of the data. We compare the coverage probabilities and widths of the different interval methods.

Results

We show that, for the two-way, crossed, random effects model without interaction, care is needed in interval method selection because the interval estimates do not always have properties that the user expects. While different methods generally perform well when there are a large number of levels of each factor, large differences between the methods emerge when the number of one or more factors is limited. In addition, all methods are shown to lack robustness to certain hard-to-detect violations of normality when the sample size is limited.

Conclusions

Decision rules and software programs for interval construction are provided for practical implementation in the two-way, crossed, random effects model without interaction. All interval methods perform similarly when the data are normal and there are sufficient numbers of levels of each factor. The MLS and GCI methods outperform the NIB when one of the factors has a limited number of levels and the data are normally distributed or nearly normally distributed. None of the methods work well if the number of levels of a factor are limited and data are markedly non-normal. The software programs are implemented in the popular R language.

SUBMITTER: Ionan AC

PROVIDER: S-EPMC4258044 | biostudies-literature | 2014 Nov

REPOSITORIES: biostudies-literature

ACCESS DATA

Publications

Comparison of confidence interval methods for an intra-class correlation coefficient (ICC).

Ionan Alexei C AC Polley Mei-Yin C MY McShane Lisa M LM Dobbin Kevin K KK

BMC medical research methodology 20141122

<h4>Background</h4>The intraclass correlation coefficient (ICC) is widely used in biomedical research to assess the reproducibility of measurements between raters, labs, technicians, or devices. For example, in an inter-rater reliability study, a high ICC value means that noise variability (between-raters and within-raters) is small relative to variability from patient to patient. A confidence interval or Bayesian credible interval for the ICC is a commonly reported summary. Such intervals can b ...[more]

PMID: 25417040

Similar Datasets

Project description:When data is derived under a single or multiple lower limits of quantification (LLOQ), estimation of distribution parameters as well as precision of these estimates appear to be challenging, as the way to account for unquantifiable observations due to LLOQs needs particular attention. The aim of this investigation is to characterize the precision of censored sample maximum likelihood estimates of the mean for normal, exponential and Poisson distribution affected by one or two LLOQs using confidence intervals (CI). In a simulation study, asymptotic and bias-corrected accelerated bootstrap CIs for the location parameter mean are compared with respect to coverage proportion and interval width. To enable this examination, we derived analytical expressions of the maximum likelihood location parameter estimate for the assumption of exponentially and Poisson distributed data, where the censored sample method and simple imputation method are used to account for LLOQs. Additionally, we vary the proportion of observations below the LLOQs. When based on the censored sample estimate, the bootstrap CI led to higher coverage proportions and narrower interval width than the asymptotic CI. The results differed by underlying distribution. Under the assumption of normality, the CI's coverage proportion and width suffered most from high proportions of unquantifiable observations. For exponentially and Poisson distributed data, both CI approaches delivered similar results. To derive the CIs, the point estimates from the censored sample method are preferable, because the point estimate of the simple imputation method leads to higher bias for all investigated distributions. This biased simple imputation estimate impairs the coverage proportion of the respective CI. The bootstrap CI surpassed the asymptotic CIs with respect to coverage proportion for the investigated choice of distributional assumptions. The variety of distributions for which the methods are suitable gives the applicant a widely usable tool to handle LLOQ affected data with appropriate approaches.

Dataset Information

Comparison of confidence interval methods for an intra-class correlation coefficient (ICC).

Background

Methods

Results

Conclusions

Publications

Comparison of confidence interval methods for an intra-class correlation coefficient (ICC).

Similar Datasets

OmicsDI is part of the ELIXIR infrastructure

Tweets