Dataset Information

Crowdsourced MRI quality metrics and expert quality annotations for training of humans and machines.

ABSTRACT: The neuroimaging community is steering towards increasingly large sample sizes, which are highly heterogeneous because they can only be acquired by multi-site consortia. The visual assessment of every imaging scan is a necessary quality control step, yet arduous and time-consuming. A sizeable body of evidence shows that images of low quality are a source of variability that may be comparable to the effect size under study. We present the MRIQC Web-API, an open crowdsourced database that collects image quality metrics extracted from MR images and corresponding manual assessments by experts. The database is rapidly growing, and currently contains over 100,000 records of image quality metrics of functional and anatomical MRIs of the human brain, and over 200 expert ratings. The resource is designed for researchers to share image quality metrics and annotations that can readily be reused in training human experts and machine learning algorithms. The ultimate goal of the database is to allow the development of fully automated quality control tools that outperform expert ratings in identifying subpar images.

SUBMITTER: Esteban O

PROVIDER: S-EPMC6472378 | biostudies-literature | 2019 Apr

REPOSITORIES: biostudies-literature

ACCESS DATA

Publications

Crowdsourced MRI quality metrics and expert quality annotations for training of humans and machines.

Esteban Oscar O Blair Ross W RW Nielson Dylan M DM Varada Jan C JC Marrett Sean S Thomas Adam G AG Poldrack Russell A RA Gorgolewski Krzysztof J KJ

Scientific data 20190411 1

The neuroimaging community is steering towards increasingly large sample sizes, which are highly heterogeneous because they can only be acquired by multi-site consortia. The visual assessment of every imaging scan is a necessary quality control step, yet arduous and time-consuming. A sizeable body of evidence shows that images of low quality are a source of variability that may be comparable to the effect size under study. We present the MRIQC Web-API, an open crowdsourced database that collects ...[more]

PMID: 30975998

Similar Datasets

Project description:BackgroundTailored, web-assisted interventions can reach many smokers. Content from other smokers (peers) through crowdsourcing could enhance relevance.PurposeTo evaluate whether peers can generate tailored messages encouraging other smokers to use a web-assisted tobacco intervention (Decide2Quit.org).MethodsPhase 1: In 2009, smokers wrote messages in response to scenarios for peer advice. These smoker-to-smoker (S2S) messages were coded to identify themes. Phase 2: resulting S2S messages, and comparison expert messages, were then e-mailed to newly registered smokers. In 2012, subsequent Decide2Quit.org visits following S2S or expert-written e-mails were compared.ResultsPhase 1: a total of 39 smokers produced 2886 messages (message themes: attitudes and expectations, improvements in quality of life, seeking help, and behavioral strategies). For not-ready-to-quit scenarios, S2S messages focused more on expectations around a quit attempt and how quitting would change an individual's quality of life. In contrast, for ready-to-quit scenarios, S2S messages focused on behavioral strategies for quitting. Phase 2: In multivariable analysis, S2S messages were more likely to generate a return visit (OR=2.03, 95% CI=1.74, 2.35), compared to expert messages. A significant effect modification of this association was found, by time-from-registration and message codes (both interaction terms p<0.01). In stratified analyses, S2S codes that were related more to "social" and "real-life" aspects of smoking were driving the main association of S2S and increased return visits.ConclusionsS2S peer messages that increased longitudinal engagement in a web-assisted tobacco intervention were successfully collected and delivered. S2S messages expanded beyond the biomedical model to enhance relevance of messages.Trial registrationThis study is registered at www.clinicaltrials.gov NCT00797628 (web-delivered provider intervention for tobacco control [QUIT-PRIMO]) and NCT01108432 (DPBRN Hygienists Internet Quality Improvement in Tobacco Cessation [HiQuit]).

Project description:Recent advancements in in situ methods, such as multiplexed in situ RNA hybridization and in situ RNA sequencing, have deepened our understanding of the way biological processes are spatially organized in tissues. Automated image processing and spot-calling algorithms for analyzing in situ transcriptomics images have many parameters which need to be tuned for optimal detection. Having ground truth datasets (images where there is very high confidence on the accuracy of the detected spots) is essential for evaluating these algorithms and tuning their parameters. We present a first-in-kind open-source toolkit and framework for in situ transcriptomics image analysis that incorporates crowdsourced annotations, alongside expert annotations, as a source of ground truth for the analysis of in situ transcriptomics images. The kit includes tools for preparing images for crowdsourcing annotation to optimize crowdsourced workers' ability to annotate these images reliably, performing quality control (QC) on worker annotations, extracting candidate parameters for spot-calling algorithms from sample images, tuning parameters for spot-calling algorithms, and evaluating spot-calling algorithms and worker performance. These tools are wrapped in a modular pipeline with a flexible structure that allows users to take advantage of crowdsourced annotations from any source of their choice. We tested the pipeline using real and synthetic in situ transcriptomics images and annotations from the Amazon Mechanical Turk system obtained via Quanti.us. Using real images from in situ experiments and simulated images produced by one of the tools in the kit, we studied worker sensitivity to spot characteristics and established rules for annotation QC. We explored and demonstrated the use of ground truth generated in this way for validating spot-calling algorithms and tuning their parameters, and confirmed that consensus crowdsourced annotations are a viable substitute for expert-generated ground truth for these purposes.

Project description:Formalin-fixed, paraffin-embedded (FFPE) tissues have many advantages for identification of risk biomarkers, including wide availability and potential for extended follow-up endpoints. However, RNA derived from archival FFPE samples has limited quality. Here we identified parameters that determine which FFPE samples have the potential for successful RNA extraction, library preparation, and generation of usable RNAseq data. We optimized library preparation protocols designed for use with FFPE samples using seven FFPE and Fresh Frozen replicate pairs, and tested optimized protocols using a study set of 130 FFPE biopsies from women with benign breast disease. Metrics from RNA extraction and preparation procedures were collected and compared with bioinformatics sequencing summary statistics. Finally, a decision tree model was built to learn the relationship between pre-sequencing lab metrics and qc pass/fail status as determined by bioinformatics metrics.. Samples that failed bioinformatics qc tended to have low median sample-wise correlation within the cohort (Spearman correlation < 0.75), low number of reads mapped to gene regions (< 25 million), or low number of detectable genes (11,400 # of detected genes with TPM > 4). The median RNA concentration and pre-capture library Qubit values for qc failed samples were 18.9 ng/ul and 2.08 ng/ul respectively, which were significantly lower than those of qc pass samples (40.8 ng/ul and 5.82 ng/ul). We built a decision tree model based on input RNA concentration, input library qubit values, and achieved an F score of 0.848 in predicting QC status (pass/fail) of FFPE samples. We provide a bioinformatics quality control recommendation for FFPE samples from breast tissue by evaluating bioinformatic and sample metrics. Our results suggest a minimum concentration of 25 ng/ul FFPE-extracted RNA for library preparation and 1.7 ng/ul pre-capture library output to achieve adequate RNA-seq data for downstream bioinformatics analysis.

Dataset Information

Crowdsourced MRI quality metrics and expert quality annotations for training of humans and machines.

Publications

Crowdsourced MRI quality metrics and expert quality annotations for training of humans and machines.

Similar Datasets

OmicsDI is part of the ELIXIR infrastructure