Dataset Information

Assessing reliability in neuroimaging research through intra-class effect decomposition (ICED).

ABSTRACT: Magnetic resonance imaging has become an indispensable tool for studying associations of structural and functional properties of the brain with behavior in humans. However, generally recognized standards for assessing and reporting the reliability of these techniques are still lacking. Here, we introduce a new approach for assessing and reporting reliability, termed intra-class effect decomposition (ICED). ICED uses structural equation modeling of data from a repeated-measures design to decompose reliability into orthogonal sources of measurement error that are associated with different characteristics of the measurements, for example, session, day, or scanning site. This allows researchers to describe the magnitude of different error components, make inferences about error sources, and inform them in planning future studies. We apply ICED to published measurements of myelin content and resting state functional connectivity. These examples illustrate how longitudinal data can be leveraged separately or conjointly with cross-sectional data to obtain more precise estimates of reliability.

SUBMITTER: Brandmaier AM

PROVIDER: S-EPMC6044907 | biostudies-other | 2018 Jul

REPOSITORIES: biostudies-other

ACCESS DATA

Publications

Assessing reliability in neuroimaging research through intra-class effect decomposition (ICED).

Brandmaier Andreas M AM Wenger Elisabeth E Bodammer Nils C NC Kühn Simone S Raz Naftali N Lindenberger Ulman U

eLife 20180702

Magnetic resonance imaging has become an indispensable tool for studying associations of structural and functional properties of the brain with behavior in humans. However, generally recognized standards for assessing and reporting the reliability of these techniques are still lacking. Here, we introduce a new approach for assessing and reporting reliability, termed intra-class effect decomposition (ICED). ICED uses structural equation modeling of data from a repeated-measures design to decompos ...[more]

PMID: 29963984

Similar Datasets

Project description:Introduction"Lean" is a set of management principles which focus on increasing value and efficiency by reducing or avoiding waste (e.g., overproduction, defects, inventory, transportation, waiting, motion, over processing). It has been applied to manufacturing, education, and health care, leading to optimized process flow, increased efficiency and increased team empowerment. However, to date, it has not been applied to neuroimaging research.MethodsLean principles, such as Value stream mapping (e.g. a tool with which steps in the workflow can be identified on which to focus improvement efforts), 5S (e.g. an organizational method to boost workplace efficiency and efficacy) and Root-cause analysis (e.g. a problem-solving approach to identify key points of failure in a system) were applied to an ongoing, large neuroimaging study that included seven research visits per participant. All team members participated in a half-day exercise in which the entire project flow was visualized and areas of inefficiency were identified. Changes focused on removing obstacles, standardization, optimal arrangement of equipment and root-cause-analysis. A process for continuous improvement was also implemented. Total time of an experiment was recorded before implementation of Lean for two participants and after implementation of Lean for two participants. Satisfaction of team members was assessed anonymously on a 5-item Likert scale, ranging from much worsened to much improved.ResultsAll team members (N = 6) considered the overall experience of conducting an experiment much improved after implementation of Lean. Five out of six team members indicated a much-improved reduction in time, with the final team member considering this somewhat improved. Average experiment time was reduced by 13% after implementation of Lean (from 142 and 147 minutes to 124 and 128 minutes).DiscussionLean principles can be successfully applied to neuroimaging research. Training in Lean principles for junior research scientists is recommended.

Project description:Interdisciplinary research collaboration is needed to perform transformative science and accelerate innovation. The Science of Team Science strives to investigate, evaluate, and foster team science, including institutional policies that may promote or hinder collaborative interdisciplinary research and the resources and infrastructure needed to promote team science within and across institutions. Social network analysis (SNA) has emerged as a useful method to measure interdisciplinary science through the evaluation of several types of collaboration networks, including co-authorship networks. Likewise, research administrators are responsible for conducting rigorous evaluation of policies and initiatives. Within this paper, we present a case study using SNA to evaluate inter-programmatic collaboration (evidenced by co-authoring scientific papers) from 2007-2014 among scientists who are members of four formal research programs at an NCI-designated Cancer Center, the Markey Cancer Center (MCC) at the University of Kentucky. We evaluate change in network descriptives over time and implement separable temporal exponential-family random graph models (STERGMs) to estimate the effect of author and network variables on the tendency to form a co-authorship tie. We measure the diversity of the articles published over time (Blau's Index) to understand whether the changes in the co-authorship network are reflected in the diversity of articles published by research members. Over the 8-year period, we found increased inter-programmatic collaboration among research members as evidenced by co-authorship of published scientific papers. Over time, MCC Members collaborated more with others outside of their research program and outside their initial dense co-authorship groups, however tie formation continues to be driven by co-authoring with individuals of the same research program and academic department. Papers increased in diversity over time on all measures with the exception of author gender. This inter-programmatic research was fostered by policy changes in cancer center administration encouraging interdisciplinary research through both informal (e.g., annual retreats, seminar series) and formal (e.g., requiring investigators from more than two research programs on applications for pilot funding) means. Within this cancer center, interdisciplinary co-authorship increased over time as policies encouraging this collaboration were implemented. Yet, there is room for improvement in creating more interdisciplinary and diverse ties between research program members.

Project description:The usual method for assessing the reliability of survey data has been to conduct reinterviews a short interval (such as one to two weeks) after an initial interview and to use these data to estimate relatively simple statistics, such as gross difference rates (GDRs). More sophisticated approaches have also been used to estimate reliability. These include estimates from multi-trait, multi-method experiments, models applied to longitudinal data, and latent class analyses. To our knowledge, no prior study has systematically compared these different methods for assessing reliability. The Population Assessment of Tobacco and Health Reliability and Validity (PATH-RV) Study, done on a national probability sample, assessed the reliability of answers to the Wave 4 questionnaire from the PATH Study. Respondents in the PATH-RV were interviewed twice about two weeks apart. We examined whether the classic survey approach yielded different conclusions from the more sophisticated methods. We also examined two ex ante methods for assessing problems with survey questions and item nonresponse rates and response times to see how strongly these related to the different reliability estimates. We found that kappa was highly correlated with both GDRs and over-time correlations, but the latter two statistics were less highly correlated, particularly for adult respondents; estimates from longitudinal analyses of the same items in the main PATH study were also highly correlated with the traditional reliability estimates. The latent class analysis results, based on fewer items, also showed a high level of agreement with the traditional measures. The other methods and indicators had at best weak relationships with the reliability estimates derived from the reinterview data. Although the Question Understanding Aid seems to tap a different factor from the other measures, for adult respondents, it did predict item nonresponse and response latencies and thus may be a useful adjunct to the traditional measures.

Dataset Information

Assessing reliability in neuroimaging research through intra-class effect decomposition (ICED).

Publications

Assessing reliability in neuroimaging research through intra-class effect decomposition (ICED).

Similar Datasets

OmicsDI is part of the ELIXIR infrastructure

Tweets