Dataset Information

We need to talk about reliability: making better use of test-retest studies for study design and interpretation.

ABSTRACT: Neuroimaging, in addition to many other fields of clinical research, is both time-consuming and expensive, and recruitable patients can be scarce. These constraints limit the possibility of large-sample experimental designs, and often lead to statistically underpowered studies. This problem is exacerbated by the use of outcome measures whose accuracy is sometimes insufficient to answer the scientific questions posed. Reliability is usually assessed in validation studies using healthy participants, however these results are often not easily applicable to clinical studies examining different populations. I present a new method and tools for using summary statistics from previously published test-retest studies to approximate the reliability of outcomes in new samples. In this way, the feasibility of a new study can be assessed during planning stages, and before collecting any new data. An R package called relfeas also accompanies this article for performing these calculations. In summary, these methods and tools will allow researchers to avoid performing costly studies which are, by virtue of their design, unlikely to yield informative conclusions.

SUBMITTER: Matheson GJ

PROVIDER: S-EPMC6536112 | biostudies-other | 2019

REPOSITORIES: biostudies-other

ACCESS DATA

Publications

We need to talk about reliability: making better use of test-retest studies for study design and interpretation.

Matheson Granville J GJ

PeerJ 20190524

Neuroimaging, in addition to many other fields of clinical research, is both time-consuming and expensive, and recruitable patients can be scarce. These constraints limit the possibility of large-sample experimental designs, and often lead to statistically underpowered studies. This problem is exacerbated by the use of outcome measures whose accuracy is sometimes insufficient to answer the scientific questions posed. Reliability is usually assessed in validation studies using healthy participant ...[more]

PMID: 31179173

Similar Datasets

Project description:Neural correlates of decision making under risk are being increasingly utilized as biomarkers of risk for substance abuse and other psychiatric disorders, treatment outcomes, and brain development. This research relies on the basic assumption that fMRI measures of decision making represent stable, trait-like individual differences. However, reliability needs to be established for each individual construct. Here we assessed long-term test-retest reliability (TRR) of regional brain activations related to decision making under risk using the Balloon Analogue Risk Taking task (BART) and identified regions with good TRRs and familial influences, an important prerequisite for the use of fMRI measures in genetic studies. A secondary goal was to examine the factors potentially affecting fMRI TRRs in one particular risk task, including the magnitude of neural activation, data analytical approaches, different methods of defining boundaries of a region, and participant motion. For the average BOLD response, reliabilities ranged across brain regions from poor to good (ICCs of 0 to 0.8, with a mean ICC of 0.17) and highest reliabilities were observed for parietal, occipital, and temporal regions. Among the regions that were of a priori theoretical importance due to their reported associations with decision making, the activation of left anterior insula and right caudate during the decision period showed the highest reliabilities (ICCs of 0.54 and 0.63, respectively). Among the regions with highest reliabilities, the right fusiform, right rostral anterior cingulate and left superior parietal regions also showed high familiality as indicated by intrapair monozygotic twin correlations (ranging from 0.66 to 0.69). Overall, regions identified by modeling the average BOLD response to a specific event type (rather than its modulation by a parametric regressor), regions including significantly activated vertices (compared to a whole parcel), and regions with greater magnitude of task-related activations showed greater reliabilities. Participant motion had a moderate negative effect on TRR. Regions activated during decision period rather than outcome period of risky decisions showed the greatest TRR and familiality. Regions with reliable activations can be utilized as neural markers of individual differences or endophenotypes in future clinical neuroscience and genetic studies of risk-taking.

Project description:BACKGROUND:Concussion diagnosis is typically made through clinical examination and supported by performance on clinical assessment tools. Performance on commonly implemented and emerging assessment tools is known to vary between administrations, in the absence of concussion. OBJECTIVE:To evaluate the test-retest reliability of commonly implemented and emerging concussion assessment tools across a large nationally representative sample of student-athletes. METHODS:Participants (n = 4874) from the Concussion Assessment, Research, and Education Consortium completed annual baseline assessments on two or three occasions. Each assessment included measures of self-reported concussion symptoms, motor control, brief and extended neurocognitive function, reaction time, oculomotor/oculovestibular function, and quality of life. Consistency between years 1 and 2 and 1 and 3 were estimated using intraclass correlation coefficients or Kappa and effect sizes (Cohen's d). Clinical interpretation guidelines were also generated using confidence intervals to account for non-normally distributed data. RESULTS:Reliability for the self-reported concussion symptoms, motor control, and brief and extended neurocognitive assessments from year 1 to 2 ranged from 0.30 to 0.72 while effect sizes ranged from 0.01 to 0.28 (i.e., small). The reliability for these same measures ranged from 0.34 to 0.66 for the year 1-3 interval with effect sizes ranging from 0.05 to 0.42 (i.e., small to less than medium). The year 1-2 reliability for the reaction time, oculomotor/oculovestibular function, and quality-of-life measures ranged from 0.28 to 0.74 with effect sizes from 0.01 to 0.38 (i.e., small to less than medium effects). CONCLUSIONS:This investigation noted less than optimal reliability for most common and emerging concussion assessment tools. Despite this finding, their use is still necessitated by the absence of a gold standard diagnostic measure, with the ultimate goal of developing more refined and sound tools for clinical use. Clinical interpretation guidelines are provided for the clinician to apply with a degree of certainty in application.

Project description:Neural and behavioral mechanisms during approach-avoidance conflict decision-making are relevant across various psychiatric disorders, particularly anxiety disorders. Studies using approach-avoidance conflict paradigms in healthy adults have identified preliminary neural mechanisms, but findings must be replicated and demonstrated as reliable before further application. This study sought to replicate previous findings and examine test-retest reliability of behavioral (approach behavior, reaction time) and neural (regions of interest [ROIs]) responses during an approach-avoidance conflict task conducted during functional magnetic resonance imaging (fMRI). Thirty healthy adults completed an approach-avoidance conflict task during fMRI on two occasions (mean interval: 17 days; range: 11-32). Effects of task condition during three task phases (decision-making, affective outcome and monetary reward) and intraclass correlation coefficients (ICCs) were calculated across time points. Results replicated that approach behavior was modulated by conflict during decision-making. ROI activations were replicated such that dorsal anterior cingulate cortex (dACC) was modulated by conflict during decision-making, and dACC, striatum, and anterior insula were modulated by valence during affective outcomes (p's <.0083). Approach behavior during conflict demonstrated excellent reliability (ICCs ≥.77). Activation of dACC during conflict decision-making and anterior insula during negative outcomes demonstrated fair reliability (ICCs = .51 and .54), and dACC and striatum activation demonstrated good reliability during negative outcomes (ICCs = .63 and .69). Two additional ROIs (amygdala, left dorsolateral prefrontal cortex) showed good reliability during negative outcomes (ICCs ≥.60). These results characterize several specific behavioral and neuroimaging responses that are replicable and sufficiently reliable during approach-avoidance conflict decision-making to support future utility.

Project description:BackgroundNeurocognitive testing is an important concussion evaluation tool, but for neurocognitive tests to be useful, their psychometric properties must be well established. Test-retest reliability of computerized neurocognitive tests can influence their clinical utility. The reliability for a commonly used computerized neurocognitive test, CNS Vital Signs, is not well established. The purpose of this study was to examine test-retest reliability and reliable change indices for CNS Vital Signs in a healthy, physically active college population.HypothesisCNS Vital Signs yields acceptable test-retest reliability, with greater reliability between the second and third test administration compared with between the first and second administration.Study designCohort study.Level of evidenceLevel 3.MethodsForty healthy, active volunteers (16 men, 24 women; mean age, 21.05 ± 2.17 years) reported to a clinical laboratory for 3 sessions, 1 week apart. At each session, participants were administered CNS Vital Signs. Outcomes included standard scores for the following CNS Vital Signs domains: verbal memory, visual memory, psychomotor speed, cognitive flexibility, complex attention, processing speed, reaction time, executive functioning, and reasoning.ResultsParticipants performed significantly better on the second session and/or third session than they did on the first testing session on 6 of 9 neurocognitive domains. Pearson r test-retest correlations between sessions ranged from 0.11 to 0.87. Intraclass correlation coefficients ranged from 0.10 to 0.86.ConclusionClinicians should consider using reliable change indices to account for practice effects, identify meaningful score changes due to pathology, and inform clinical decisions.Clinical relevanceThis study highlights the importance of clinicians understanding the psychometric properties of computerized neurocognitive tests when using them in the management of sport-related concussion. If CNS Vital Signs is administered twice within a small time frame (such as 1 week), athletes should be expected to improve between the first and second administration.

Dataset Information

We need to talk about reliability: making better use of test-retest studies for study design and interpretation.

Publications

We need to talk about reliability: making better use of test-retest studies for study design and interpretation.

Similar Datasets

OmicsDI is part of the ELIXIR infrastructure

Tweets