Agreement and reliability statistics for shapes.
Ontology highlight
ABSTRACT:
SUBMITTER: Smith TB
PROVIDER: S-EPMC6107162 | biostudies-literature | 2018
REPOSITORIES: biostudies-literature
Publications
We describe a methodology for assessing agreement and reliability among a set of shapes. Motivated by recent studies of the reliability of manually segmented medical images, we focus on shapes composed of rasterized, binary-valued data representing closed geometric regions of interest. The methodology naturally generalizes to N dimensions and other data types, though. We formulate the shape variance, shape correlation and shape intraclass correlation coefficient (ICC) in terms of a simple distan ...[more]
Similar Datasets
Project description:Human perception, cognition, and action are laced with seemingly arbitrary mappings. In particular, sound has a strong spatial connotation: Sounds are high and low, melodies rise and fall, and pitch systematically biases perceived sound elevation. The origins of such mappings are unknown. Are they the result of physiological constraints, do they reflect natural environmental statistics, or are they truly arbitrary? We recorded natural sounds from the environment, analyzed the elevation-dependent filtering of the outer ear, and measured frequency-dependent biases in human sound localization. We find that auditory scene statistics reveals a clear mapping between frequency and elevation. Perhaps more interestingly, this natural statistical mapping is tightly mirrored in both ear-filtering properties and in perceived sound location. This suggests that both sound localization behavior and ear anatomy are fine-tuned to the statistics of natural auditory scenes, likely providing the basis for the spatial connotation of human hearing.
| S-EPMC4000839 | biostudies-other
Project description:In clinical studies, it is often of interest to see the diagnostic agreement among clinicians on certain symptoms. Previous work has focused on the agreement between two clinicians under two different conditions or the agreement among multiple clinicians under one condition. Few have discussed the agreement study with a design where multiple clinicians examine the same group of patients under two different conditions. In this paper, we use the intraclass kappa statistic for assessing nominal scale agreement with such a design. We derive an explicit variance formula for the difference of correlated kappa statistics and conduct hypothesis testing for the equality of kappa statistics. Simulation studies show that the method performs well with realistic sample sizes and may be superior to a method that did not take into account the measurement dependence structure. The practical utility of the method is illustrated on data from an eosinophilic esophagitis (EoE) study.
| S-EPMC9631095 | biostudies-literature
Project description:BACKGROUND: Reliable suicide statistics are a prerequisite for suicide monitoring and prevention. The aim of this study was to assess the reliability of suicide statistics through a systematic review of the international literature. METHODS: We searched for relevant publications in EMBASE, Ovid Medline, PubMed, PsycINFO and the Cochrane Library up to October 2010. In addition, we screened related studies and reference lists of identified studies. We included studies published in English, German, French, Spanish, Norwegian, Swedish and Danish that assessed the reliability of suicide statistics. We excluded case reports, editorials, letters, comments, abstracts and statistical analyses. All three authors independently screened the abstracts, and then the relevant full-text articles. Disagreements were resolved through consensus. RESULTS: The primary search yielded 127 potential studies, of which 31 studies met the inclusion criteria and were included in the final review. The included studies were published between 1963 and 2009. Twenty were from Europe, seven from North America, two from Asia and two from Oceania. The manner of death had been re-evaluated in 23 studies (40-3,993 cases), and there were six registry studies (195-17,412 cases) and two combined registry and re-evaluation studies. The study conclusions varied, from findings of fairly reliable to poor suicide statistics. Thirteen studies reported fairly reliable suicide statistics or under-reporting of 0-10%. Of the 31 studies during the 46-year period, 52% found more than 10% under-reporting, and 39% found more than 30% under-reporting or poor suicide statistics. Eleven studies reassessed a nationwide representative sample, although these samples were limited to suicide within subgroups. Only two studies compared data from two countries. CONCLUSIONS: The main finding was that there is a lack of systematic assessment of the reliability of suicide statistics. Few studies have been done, and few countries have been covered. The findings support the general under-reporting of suicide. In particular, nationwide studies and comparisons between countries are lacking.
| S-EPMC3350416 | biostudies-literature
Project description:BACKGROUND:Human height is a simple measure with great applicability. Usually, stadiometers are used to measure height accurately. However, these may be impractical to transport and expensive. Therefore, we developed a portable and low-cost laser height metre (LHM). OBJECTIVE:We aimed to (1) determine intrarater and interrater reliability of our LHM and compare it to a wall-fixed stadiometer, (2) examine its agreement with the same stadiometer, and (3) determine the minimum number of recordings needed to obtain an accurate and reliable height measurement using the LHM. METHODS:We recruited 32 participants (18+ years)-both men and women. Two raters performed assessments on the same day blinded to each other and their reference standard measurements. We calculated intraclass correlation coefficient (ICC), coefficient of variation (CV), standard error of measurement (SEM), and Bland-Altman plots with limits of agreement (LOA). RESULTS:For both the LHM and stadiometer, we found ICC values of 0.99-1.00 (95% CI: 0.997-1.000) for both intrarater and interrater reliability. Regarding LHM intrarater reliability, SEM, CV, and LOA were 0.34 cm, 0.16%, and -1.07 to 0.73 cm, respectively. In terms of LHM interrater reliability, SEM, CV, and LOA were 0.27 cm, 0.12%, and -0.32 to 0.84 cm, respectively. As to agreement with stadiometers using one measurement, the mean difference was -0.14 cm and LOA ranged from -0.81 to 0.77 cm. CONCLUSION:A portable and low-cost LHM, for measuring body height once, showed an excellent reproducibility within and between raters along with an acceptable agreement with a stadiometer thereby representing a suitable alternative.
| S-EPMC7141692 | biostudies-literature
Project description:The aim of the study was to examine the test-retest reliability and agreement across methods for assessing individual force-velocity (FV) profiles of the lower limbs in athletes. Using a multicenter approach, 27 male athletes completed all measurements for the main analysis, with up to 82 male and female athletes on some measurements. The athletes were tested twice before and twice after a 2- to 6-month period of regular training and sport participation. The double testing sessions were separated by ~1 week. Individual FV-profiles were acquired from incremental loading protocols in squat jump (SJ), countermovement jump (CMJ) and leg press. A force plate, linear encoder and a flight time calculation method were used for measuring force and velocity during SJ and CMJ. A linear regression was fitted to the average force and velocity values for each individual test to extrapolate the FV-variables: theoretical maximal force (F0), velocity (V0), power (Pmax), and the slope of the FV-profile (SFV). Despite strong linearity (R2>0.95) for individual FV-profiles, the SFV was unreliable for all measurement methods assessed during vertical jumping (coefficient of variation (CV): 14-30%, interclass correlation coefficient (ICC): 0.36-0.79). Only the leg press exercise, of the four FV-variables, showed acceptable reliability (CV:3.7-8.3%, ICC:0.82-0.98). The agreement across methods for F0 and Pmax ranged from (Pearson r): 0.56-0.95, standard error of estimate (SEE%): 5.8-18.8, and for V0 and SFV r: -0.39-0.78, SEE%: 12.2-37.2. With a typical error of 1.5 cm (5-10% CV) in jump height, SFV and V0 cannot be accurately obtained, regardless of the measurement method, using a loading range corresponding to 40-70% of F0. Efforts should be made to either reduce the variation in jumping performance or to assess loads closer to the FV-intercepts. Coaches and researchers should be aware of the poor reliability of the FV-variables obtained from vertical jumping, and of the differences across measurement methods.
| S-EPMC7850492 | biostudies-literature
Project description:BackgroundThe present study takes place within a geriatric program, aiming at improving the diagnosis and management of geriatric syndromes in primary care. Within this program it was of prime importance to be able to rely on a robust and reproducible geriatric consultation to use as a gold standard for evaluating a primary care brief assessment tool. The specific objective of the present study was thus assessing the agreement and reliability of a comprehensive geriatric consultation.MethodThe study was conducted at the outpatient clinic of the Service of Geriatric Medicine, University of Lausanne, Switzerland. All community-dwelling older persons aged 70 years and above were eligible. Patients were excluded if they hadn't a primary care physician, they were unable to speak French, or they were already assessed by a geriatrician within the last 12 months. A set of 9 geriatricians evaluated 20 patients. Each patient was assessed twice within a 2-month delay. Geriatric consultations were based on a structured evaluation process, leading to rating the following geriatric conditions: functional, cognitive, visual, and hearing impairment, mood disorders, risk of fall, osteoporosis, malnutrition, and urinary incontinence. Reliability and agreement estimates on each of these items were obtained using a three-way Intraclass Correlation and a three-way Observed Disagreement index. The latter allowed a decomposition of overall disagreement into disagreements due to each source of error variability (visit, rater and random).ResultsAgreement ranged between 0.62 and 0.85. For most domains, geriatrician-related error variability explained an important proportion of disagreement. Reliability ranged between 0 and 0.8. It was poor/moderate for visual impairment, malnutrition and risk of fall, and good/excellent for functional/cognitive/hearing impairment, osteoporosis, incontinence and mood disorders.ConclusionsSix out of nine items of the geriatric consultation described in this study (functional/cognitive/hearing impairment, osteoporosis, incontinence and mood disorders) present a good to excellent reliability and can safely be used as a reference (gold standard) to evaluate the diagnostic performance of a primary care brief assessment tool. More objective/significant measures are needed to improve reliability of malnutrition, visual impairment, and risk of fall assessment before they can serve as a safe gold standard of a primary care tool.
| S-EPMC5517926 | biostudies-literature
Project description:The ability to reproducibly match tinnitus loudness and pitch is important to research and clinical management. Here we examine agreement and reliability of tinnitus loudness matching and pitch likeness ratings when using a computer-based method to measure the tinnitus spectrum and estimate a dominant tinnitus pitch, using tonal or narrowband sounds. Group level data indicated a significant effect of time between test session 1 and 2 for loudness matching, likely procedural or perceptual learning, which needs to be accounted in study design. Pitch likeness rating across multiple frequencies appeared inherently more variable and with no systematic effect of time. Dominant pitch estimates reached a level of clinical acceptability when sessions were spaced two weeks apart. However when dominant tinnitus pitch assessments were separated by three months, acceptable agreement was achieved only for group mean data, not for individual estimates. This has implications for prescription of some sound-based interventions that rely on accurate measures of individual dominant tinnitus pitch.
| S-EPMC4257689 | biostudies-literature
Project description:Relatively little is known about reliability of longitudinal diffusion-tensor imaging (DTI) measurements despite growing interest in using DTI to track change in white matter structure. The purpose of this study is to quantify within- and between session scan-rescan reliability of DTI-derived measures that are commonly used to describe the characteristics of neural white matter in the context of neural plasticity research. DTI data were acquired from 16 cognitively healthy older adults (mean age 68.4). We used the Tract-Based Spatial Statistics (TBSS) approach implemented in FSL, evaluating how different DTI preprocessing choices affect reliability indices. Test-Retest reliability, quantified as ICC averaged across the voxels of the TBSS skeleton, ranged from 0.524 to 0.798 depending on the specific DTI-derived measure and the applied preprocessing steps. The two main preprocessing steps that we found to improve TBSS reliability were (a) the use of a common individual template and (b) smoothing DTI data using a 1-voxel median filter. Overall our data indicate that small choices in the preprocessing pipeline have a significant effect on test-retest reliability, therefore influencing the power to detect change within a longitudinal study. Furthermore, differences in the data processing pipeline limit the comparability of results across studies.
| S-EPMC6869410 | biostudies-literature
Project description:The aim of this study was to evaluate the reliability and agreement between the test and retest of the 10-repetition maximum (10-RM) test for leg press and bench press in breast cancer survivors (BCS). Thirty-one BCS participated in this study, age 54.87 ± 5.7 years. All performed 10-RM tests and retests for the leg press 45° and the bench press. For reliability analyses, an intraclass coefficient correlation (ICC) and coefficient of variation (CV) were performed. The limits of agreement were calculated using a Bland-Altman plot with 95% CIs. For absolute and relative error of measurement, we used standard error of measurement and minimally detectable change. The result showed a high reliability for the bench press and leg press; ICC of 0.94 and 0.98, respectively. CV was <10% for both exercises. The systematic error were 1.5 kg (10%) and 6.1 (8%) for the bench press and leg press, respectively. The standard errors of measurements were 0.96 kg (6.08%) and 4.11 kg (5.27%) for the bench press and leg press, respectively. The minimally detectable changes were 2.72 kg (17.20%) and 5.62 kg (7.21%) for the bench press and leg press, respectively. In breast cancer survivors, the muscular strength measurement for the 10-RM test showed a high to very high rate of reliability and agreement, with acceptable error of measurement.
| S-EPMC6775190 | biostudies-literature
Project description:BACKGROUND:Expert psychiatrists conducting work disability evaluations often disagree on work capacity (WC) when assessing the same patient. More structured and standardised evaluations focusing on function could improve agreement. The RELY studies aimed to establish the inter-rater reproducibility (reliability and agreement) of 'functional evaluations' in patients with mental disorders applying for disability benefits and to compare the effect of limited versus intensive expert training on reproducibility. METHODS:We performed two multi-centre reproducibility studies on standardised functional WC evaluation (RELY 1 and 2). Trained psychiatrists interviewed 30 and 40 patients respectively and determined WC using the Instrument for Functional Assessment in Psychiatry (IFAP). Three psychiatrists per patient estimated WC from videotaped evaluations. We analysed reliability (intraclass correlation coefficients [ICC]) and agreement ('standard error of measurement' [SEM] and proportions of comparisons within prespecified limits) between expert evaluations of WC. Our primary outcome was WC in alternative work (WCalternative.work), 100-0%. Secondary outcomes were WC in last job (WClast.job), 100-0%; patients' perceived fairness of the evaluation, 10-0, higher is better; usefulness to psychiatrists. RESULTS:Inter-rater reliability for WCalternative.work was fair in RELY 1 (ICC 0.43; 95%CI 0.22-0.60) and RELY 2 (ICC 0.44; 0.25-0.59). Agreement was low in both studies, the 'standard error of measurement' for WCalternative.work was 24.6 percentage points (20.9-28.4) and 19.4 (16.9-22.0) respectively. Using a 'maximum acceptable difference' of 25 percentage points WCalternative.work between two experts, 61.6% of comparisons in RELY 1, and 73.6% of comparisons in RELY 2 fell within these limits. Post-hoc secondary analysis for RELY 2 versus RELY 1 showed a significant change in SEMalternative.work (- 5.2 percentage points WCalternative.work [95%CI - 9.7 to - 0.6]), and in the proportions on the differences ≤ 25 percentage points WCalternative.work between two experts (p = 0.008). Patients perceived the functional evaluation as fair (RELY 1: mean 8.0; RELY 2: 9.4), psychiatrists as useful. CONCLUSIONS:Evidence from non-randomised studies suggests that intensive training in functional evaluation may increase agreement on WC between experts, but fell short to reach stakeholders' expectations. It did not alter reliability. Isolated efforts in training psychiatrists may not suffice to reach the expected level of agreement. A societal discussion about achievable goals and readiness to consider procedural changes in WC evaluations may deserve considerations.
| S-EPMC6607597 | biostudies-literature