Dataset Information

Repeatability analysis improves the reliability of behavioral data.

ABSTRACT: Reliability of data has become a major concern in the course of the reproducibility crisis. Especially when studying animal behavior, confounding factors such as novelty of the test apparatus can lead to a wide variability of data which may mask treatment effects and consequently lead to misinterpretation. Habituation to the test situation is a common practice to circumvent novelty induced increases in variance and to improve the reliability of the respective measurements. However, there is a lack of published empirical knowledge regarding reasonable habituation procedures and a method validation seems to be overdue. This study aimed at setting up a simple strategy to increase reliability of behavioral data measured in a familiar test apparatus. Therefore, exemplary data from mice tested in an Open Field (OF) arena were used to elucidate the potential of habituation and how reliability of measures can be confirmed by means of a repeatability analysis using the software R. On seven consecutive days, male C57BL/6J, BALB/cJ and 129S1/SvImJ mice were tested in an OF arena once daily and individual mouse behavior was recorded. A repeatability analysis was conducted with regard to repeated trials of habituation. Our data analysis revealed that monitoring animal behavior during habituation is important to determine when individual differences of the measurements are stable. Repeatability values from distance travelled and average activity increased over the habituation period, revealing that around 60% of the variance of the data can be explained by individual differences between mice. The first day of habituation was significantly different from the following 6 days. A three-day habituation period appeared to be sufficient in this study. Overall, these results emphasize the importance of habituation and in depth analysis of habituation data to define the correct starting point of the experiment for improving the reliability and reproducibility of experimental data.

SUBMITTER: Rudeck J

PROVIDER: S-EPMC7117744 | biostudies-literature | 2020

REPOSITORIES: biostudies-literature

ACCESS DATA

Publications

Repeatability analysis improves the reliability of behavioral data.

Rudeck Juliane J Vogl Silvia S Banneke Stefanie S Schönfelder Gilbert G Lewejohann Lars L

PloS one 20200402 4

Reliability of data has become a major concern in the course of the reproducibility crisis. Especially when studying animal behavior, confounding factors such as novelty of the test apparatus can lead to a wide variability of data which may mask treatment effects and consequently lead to misinterpretation. Habituation to the test situation is a common practice to circumvent novelty induced increases in variance and to improve the reliability of the respective measurements. However, there is a la ...[more]

PMID: 32240211

Similar Datasets

Project description:Purpose:Researchers have long noted speech recognition variability that is not explained by the pure-tone audiogram. Previous work (Souza, Wright, Blackburn, Tatman, & Gallun, 2015) demonstrated that a small number of listeners with sensorineural hearing loss utilized different types of acoustic cues to identify speechlike stimuli, specifically the extent to which the participant relied upon spectral (or temporal) information for identification. Consistent with recent calls for data rigor and reproducibility, the primary aims of this study were to replicate the pattern of cue use in a larger cohort and to verify stability of the cue profiles over time. Method:Cue-use profiles were measured for adults with sensorineural hearing loss using a syllable identification task consisting of synthetic speechlike stimuli in which spectral and temporal dimensions were manipulated along continua. For the first set, a static spectral shape varied from alveolar to palatal, and a temporal envelope rise time varied from affricate to fricative. For the second set, formant transitions varied from labial to alveolar and a temporal envelope rise time varied from approximant to stop. A discriminant feature analysis was used to determine to what degree spectral and temporal information contributed to stimulus identification. A subset of participants completed a 2nd visit using the same stimuli and procedures. Results:When spectral information was static, most participants were more influenced by spectral than by temporal information. When spectral information was dynamic, participants demonstrated a balanced distribution of cue-use patterns, with nearly equal numbers of individuals influenced by spectral or temporal cues. Individual cue profile was repeatable over a period of several months. Conclusion:In combination with previously published data, these results indicate that listeners with sensorineural hearing loss are influenced by different cues to identify speechlike sounds and that those patterns are stable over time.

Project description:BackgroundMRI is the imaging modality of choice for diagnosis and intervention assessment in neurological disease. Its full potential has not been realized due in part to challenges in harmonizing advanced techniques across multiple sites.PurposeTo develop a method for the assessment of reliability and repeatability of advanced multisite-multisession neuroimaging studies and specifically to assess the reliability of an advanced MRI protocol, including multiband fMRI and diffusion tensor MRI, in a multisite setting.Study typeProspective.PopulationTwice repeated measurement of a single subject with stable relapsing-remitting multiple sclerosis (MS) at seven institutions.Field strength/sequenceA 3 T MRI protocol included higher spatial resolution anatomical scans, a variable flip-angle longitudinal relaxation rate constant (R1 ≡ 1/T1 ) measurement, quantitative magnetization transfer imaging, diffusion tensor imaging, and a resting-state fMRI (rsFMRI) series.AssessmentMultiple methods of assessing intrasite repeatability and intersite reliability were evaluated for imaging metrics derived from each sequence.Statistical testsStudent's t-test, Pearson's r, and intraclass correlation coefficient (ICC) (2,1) were employed to assess repeatability and reliability. Two new statistical metrics are introduced that frame reliability and repeatability in the respective units of the measurements themselves.ResultsIntrasite repeatability was excellent for quantitative R1 , magnetization transfer ratio (MTR), and diffusion-weighted imaging (DWI) based metrics (r > 0.95). rsFMRI metrics were less repeatable (r = 0.8). Intersite reliability was excellent for R1 , MTR, and DWI (ICC >0.9), and moderate for rsFMRI metrics (ICC∼0.4).Data conclusionFrom most reliable to least, using a new reliability metric introduced here, MTR > R1 > DWI > rsFMRI; for repeatability, MTR > DWI > R1 > rsFMRI. A graphical method for at-a-glance assessment of reliability and repeatability, effect sizes, and outlier identification in multisite-multisession neuroimaging studies is introduced.Level of evidence1 Technical Efficacy: Stage 2 J. Magn. Reson. Imaging 2019;50:878-888.

Project description:The aim of this study is to investigate the inter and intra-rater reliability, repeatability, and reproducibility of pulmonary transit time (PTT) measurement in patients using contrast enhanced ultrasound (CEUS), as an indirect measure of preload and left ventricular function.Mean transit times (MTT) were measured by drawing a region of interest (ROI) in right and left cardiac ventricle in the CEUS loops. Acoustic intensity dilution curves were obtained from the ROIs. MTTs were calculated by applying model-based fitting on the dilution curves. PTT was calculated as the difference of the MTTs. Eight raters with different levels of experience measured the PTT (time moment 1) and repeated the measurement within a week (time moment 2). Reliability and agreement were assessed using intra-class correlations (ICC) and Bland-Altman analysis. Repeatability was tested by estimating the variance of means (ANOVA) of three injections in each patient at different doses. Reproducibility was tested by the ICC of the two time moments.Fifteen patients with heart failure were included. The mean PTT was 11.8 ± 3.1 s at time moment 1 and 11.7 ± 2.9 s at time moment 2. The inter-rater reliability for PTT was excellent (ICC = 0.94). The intra-rater reliability per rater was between 0.81-0.99. Bland-Altman analysis revealed a bias of 0.10 s within the rater groups. Reproducibility for PTT showed an ICC = 0.94 between the two time moments. ANOVA showed no significant difference between the means of the three different doses F = 0.048 (P = 0.95). The mean and standard deviation for PTT estimates at three different doses was 11.6 ± 3.3 s.PTT estimation using CEUS shows a high inter- and intra-rater reliability, repeatability at three different doses, and reproducibility by ROI drawing. This makes the minimally invasive PTT measurement using contrast echocardiography ready for clinical evaluation in patients with heart failure and for preload estimation.

Project description:Neurological impairments can limit the implementation of conventional cardiopulmonary exercise testing (CPET) and cardiovascular training strategies. A promising approach to provoke cardiovascular stress while facilitating task-specific exercise in people with disabilities is feedback-controlled robot-assisted end-effector-based stair climbing (RASC). The aim of this study was to evaluate the feasibility, reliability, and repeatability of augmented RASC-based CPET in able-bodied subjects, with a view towards future research and applications in neurologically impaired populations.Twenty able-bodied subjects performed a familiarisation session and 2 consecutive incremental CPETs using augmented RASC. Outcome measures focussed on standard cardiopulmonary performance parameters and on accuracy of work rate tracking (RMSEP-root mean square error). Criteria for feasibility were cardiopulmonary responsiveness and technical implementation. Relative and absolute test-retest reliability were assessed by intraclass correlation coefficients (ICC), standard error of the measurement (SEM), and minimal detectable change (MDC). Mean differences, limits of agreement, and coefficients of variation (CoV) were estimated to assess repeatability.All criteria for feasibility were achieved. Mean V'O2peak was 106±9% of predicted V'O2max and mean HRpeak was 99±3% of predicted HRmax. 95% of the subjects achieved at least 1 criterion for V'O2max, and the detection of the sub-maximal ventilatory thresholds was successful (ventilatory anaerobic threshold 100%, respiratory compensation point 90% of the subjects). Excellent reliability was found for peak cardiopulmonary outcome measures (ICC ? 0.890, SEM ? 0.60%, MDC ? 1.67%). Repeatability for the primary outcomes was good (CoV ? 0.12).RASC-based CPET with feedback-guided exercise intensity demonstrated comparable or higher peak cardiopulmonary performance variables relative to predicted values, achieved the criteria for V'O2max, and allowed determination of sub-maximal ventilatory thresholds. The reliability and repeatability were found to be high. There is potential for augmented RASC to be used for exercise testing and prescription in populations with neurological impairments who would benefit from repetitive task-specific training.

Dataset Information

Repeatability analysis improves the reliability of behavioral data.

Publications

Repeatability analysis improves the reliability of behavioral data.

Similar Datasets

OmicsDI is part of the ELIXIR infrastructure

Tweets