Dataset Information

Evaluation of Reproducibility of Brain Volumetry between Commercial Software, Inbrain and Established Research Purpose Method, FreeSurfer.

ABSTRACT:

Background and purpose

We aimed to determine the intermethod reproducibility between the commercial software Inbrain (MIDAS IT) and the established research-purpose method FreeSurfer, as well as the effect of MRI resolution and the pathological condition of subjects on their intermethod reproducibility.

Methods

This study included 45 healthy volunteers and 85 patients with mild cognitive impairment (MCI). In 43 of the 85 patients with MCI, three-dimensional, T1-weighted MRI data were obtained at an in-plane resolution of 1.2 mm. The data of the remaining 42 patients with MCI and the healthy volunteers were obtained at an in-plane resolution of 1.0 mm. The within-subject coefficient of variation (CoV), intraclass correlation coefficient (ICC), and effect size were calculated, and means were compared using paired t-tests. The parameters obtained at 1.0-mm and 1.2-mm resolutions in patients with MCI were compared to evaluate the effect of the in-plane resolution on the intermethod reproducibility. The parameters obtained at a 1.0-mm in-plane resolution in patients with MCI and healthy volunteers were used to analyze the effect of subject condition on intermethod reproducibility.

Results

Overall the two methods showed excellent reproducibility across all regions of the brain (CoV=0.5-3.9, ICC=0.93 to >0.99). In the subgroup of healthy volunteers, the intermethod reliability was only good in some regions (frontal, temporal, cingulate, and insular). The intermethod reproducibility was better in the 1.0-mm group than the 1.2-mm group in all regions other than the nucleus accumbens.

Conclusions

Inbrain and FreeSurfer showed good-to-excellent intermethod reproducibility for volumetric measurements. Nevertheless, some noticeable differences were found based on subject condition, image resolution, and brain region.

SUBMITTER: Lee J

PROVIDER: S-EPMC8053534 | biostudies-literature |

REPOSITORIES: biostudies-literature

ACCESS DATA

Similar Datasets

Project description:BackgroundThe inter-scanner reproducibility of brain volumetry is important in multi-site neuroimaging studies, where the reliability of automated brain segmentation (ABS) tools plays an important role. This study aimed to evaluate the influence of ABS tools on the consistency and reproducibility of the quantified brain volumetry from different scanners.MethodsWe included fifteen healthy volunteers who were scanned with 3D isotropic brain T1-weighted sequence on three different 3.0 Tesla MRI scanners (GE, Siemens and Philips). For each individual, the time span between image acquisitions on different scanners was limited to 1 h. All the T1-weighted images were processed with FreeSurfer v6.0, FSL v5.0 and AccuBrain® with default settings to obtain volumetry of brain tissues (e.g. gray matter) and substructures (e.g. basal ganglia structures) if available. Coefficient of variation (CV) was calculated to test inter-scanner variability in brain volumetry of various structures as quantified by these ABS tools.ResultsThe mean inter-scanner CV values per brain structure among three MRI scanners ranged from 6.946 to 12.29% (mean, 9.577%) for FreeSurfer, 7.245 to 20.98% (mean, 12.60%) for FSL and 1.348 to 8.800% (mean value, 3.546%) for AccuBrain®. In addition, AccuBrain® and FreeSurfer achieved the lowest mean values of region-specific CV between GE and Siemens scanners (from 0.818 to 5.958% for AccuBrain®, and from 0.903 to 7.977% for FreeSurfer), while FSL-FIRST had the lowest mean values of region-specific CV between GE and Philips scanners (from 2.603 to 16.310%). AccuBrain® also had the lowest mean values of region-specific CV between Siemens and Philips scanners (from 1.138 to 6.615%).ConclusionThere is a large discrepancy in the inter-scanner reproducibility of brain volumetry when using different processing software. Image acquisition protocols and selection of ABS tool for brain volumetry quantification have impact on the robustness of results in multi-site studies.

Project description:PurposeTo compare two artificial intelligence software packages performing normative brain volumetry and explore whether they could differently impact dementia diagnostics in a clinical context.MethodsSixty patients (20 Alzheimer's disease, 20 frontotemporal dementia, 20 mild cognitive impairment) and 20 controls were included retrospectively. One MRI per subject was processed by software packages from two proprietary manufacturers, producing two quantitative reports per subject. Two neuroradiologists assigned forced-choice diagnoses using only the normative volumetry data in these reports. They classified the volumetric profile as "normal," or "abnormal", and if "abnormal," they specified the most likely dementia subtype. Differences between the packages' clinical impact were assessed by comparing (1) agreement between diagnoses based on software output; (2) diagnostic accuracy, sensitivity, and specificity; and (3) diagnostic confidence. Quantitative outputs were also compared to provide context to any diagnostic differences.ResultsDiagnostic agreement between packages was moderate, for distinguishing normal and abnormal volumetry (K = .41-.43) and for specific diagnoses (K = .36-.38). However, each package yielded high inter-observer agreement when distinguishing normal and abnormal profiles (K = .73-.82). Accuracy, sensitivity, and specificity were not different between packages. Diagnostic confidence was different between packages for one rater. Whole brain intracranial volume output differed between software packages (10.73%, p < .001), and normative regional data interpreted for diagnosis correlated weakly to moderately (rs = .12-.80).ConclusionDifferent artificial intelligence software packages for quantitative normative assessment of brain MRI can produce distinct effects at the level of clinical interpretation. Clinics should not assume that different packages are interchangeable, thus recommending internal evaluation of packages before adoption.

Project description:Electrical brain stimulation (EBS) is a trendy new technique used to change brain function and treat neurological, psychiatric and psychological disorders. We were curious whether the published literature, which is dominated by positive results, reflects the experience of researchers using EBS. Specifically, we wanted to know whether researchers are able to reproduce published EBS effects and whether they engage in, but fail to report, questionable research practices. We invited 976 researchers to complete an online survey. We also audited 100 randomly-selected published EBS papers. A total of 154 researchers completed the survey. Survey respondents had a median of 3 [1 to 6, IQR] published EBS papers (1180 total) and 2 [1 to 3] unpublished ones (380 total). With anodal and cathodal EBS, the two most widely used techniques, 45-50% of researchers reported being able to routinely reproduce published results. When asked about how study sample size was determined, 69% of respondents reported using the sample size of published studies, while 61% had used power calculations, and 32% had based their decision on pilot data. In contrast, our audit found only 6 papers where power calculations were used and a single paper in which pilot data was used. When asked about questionable research practices, survey respondents were aware of other researchers who selectively reported study outcomes (41%) and experimental conditions (36%), adjusted statistical analysis to optimise results (43%), and engaged in other shady practices (20%). Fewer respondents admitted to engaging in these practices themselves, although 25% admitted to adjusting statistical analysis to optimize results. There was strong agreement that such practices should be reported in research papers; however, our audit found only two such admissions. The present survey confirms that questionable research practices and poor reproducibility are present in EBS studies. The belief that EBS is effective needs to be replaced by a more rigorous approach so that reproducible brain stimulation methods can be devised and applied.

Project description:ObjectivesTo compare the cross-sectional robustness of commonly used volumetric software and effects of lesion filling in multiple sclerosis (MS).MethodsNine MS patients (six females; age 38±13 years, disease duration 7.3±5.2 years) were scanned twice with repositioning on three MRI scanners (Siemens Aera 1.5T, Avanto 1.5T, Trio 3.0T) the same day. Volumetric T1-weighted images were processed with FreeSurfer, FSL-SIENAX, SPM and SPM-CAT before and after 3D FLAIR lesion filling with LST. The whole-brain, grey matter (GM) and white matter (WM) volumes were calculated with and without normalisation to the intracranial volume or FSL-SIENAX scaling factor. Robustness was assessed using the coefficient of variation (CoV).ResultsVariability in volumetrics was lower within than between scanners (CoV 0.17-0.96% vs. 0.65-5.0%, p<0.001). All software provided similarly robust segmentations of the brain volume on the same scanner (CoV 0.17-0.28%, p=0.076). Normalisation improved inter-scanner reproducibility in FreeSurfer and SPM-based methods, but the FSL-SIENAX scaling factor did not improve robustness. Generally, SPM-based methods produced the most consistent volumetrics, while FreeSurfer was more robust for WM volumes on different scanners. FreeSurfer had more robust normalised brain and GM volumes on different scanners than FSL-SIENAX (p=0.004). MS lesion filling changed the output of FSL-SIENAX, SPM and SPM-CAT but not FreeSurfer.ConclusionsConsistent use of the same scanner is essential and normalisation to the intracranial volume is recommended for multiple scanners. Based on robustness, SPM-based methods are particularly suitable for cross-sectional volumetry. FreeSurfer poses a suitable alternative with WM segmentations less sensitive to MS lesions.Key points• The same scanner should be used for brain volumetry. If different scanners are used, the intracranial volume normalisation improves the FreeSurfer and SPM robustness (but not the FSL scaling factor). • FreeSurfer, FSL and SPM all provide robust measures of the whole brain volume on the same MRI scanner. SPM-based methods overall provide the most robust segmentations (except white matter segmentations on different scanners where FreeSurfer is more robust). • MS lesion filling with Lesion Segmentation Toolbox changes the output of FSL-SIENAX and SPM. FreeSurfer output is not affected by MS lesion filling since it already takes white matter hypointensities into account and is therefore particularly suitable for MS brain volumetry.