Browse
Submit Data
Databases
API
Help

Dataset Information

4 Views

0 Connections

0 Citations

0 Reanalyses

0 Downloads

Omics score: 0

Dataset decay and the problem of sequential analyses on open datasets.

ABSTRACT: Open data allows researchers to explore pre-existing datasets in new ways. However, if many researchers reuse the same dataset, multiple statistical testing may increase false positives. Here we demonstrate that sequential hypothesis testing on the same dataset by multiple researchers can inflate error rates. We go on to discuss a number of correction procedures that can reduce the number of false positives, and the challenges associated with these correction procedures.

SUBMITTER: Thompson WH

PROVIDER: S-EPMC7237204 | biostudies-literature | 2020 May

REPOSITORIES: biostudies-literature

ACCESS DATA

Json Xml

Publications

Dataset decay and the problem of sequential analyses on open datasets.

Thompson William Hedley WH Wright Jessey J Bissett Patrick G PG Poldrack Russell A RA

eLife 20200519

Open data allows researchers to explore pre-existing datasets in new ways. However, if many researchers reuse the same dataset, multiple statistical testing may increase false positives. Here we demonstrate that sequential hypothesis testing on the same dataset by multiple researchers can inflate error rates. We go on to discuss a number of correction procedures that can reduce the number of false positives, and the challenges associated with these correction procedures. ...[more]

PMID: 32425159

Similar Datasets

The Amsterdam Open MRI Collection, a set of multimodal MRI datasets for individual difference analyses.

Project description:We present the Amsterdam Open MRI Collection (AOMIC): three datasets with multimodal (3 T) MRI data including structural (T1-weighted), diffusion-weighted, and (resting-state and task-based) functional BOLD MRI data, as well as detailed demographics and psychometric variables from a large set of healthy participants (N = 928, N = 226, and N = 216). Notably, task-based fMRI was collected during various robust paradigms (targeting naturalistic vision, emotion perception, working memory, face perception, cognitive conflict and control, and response inhibition) for which extensively annotated event-files are available. For each dataset and data modality, we provide the data in both raw and preprocessed form (both compliant with the Brain Imaging Data Structure), which were subjected to extensive (automated and manual) quality control. All data is publicly available from the OpenNeuro data sharing platform.

| S-EPMC7979787 | biostudies-literature

Development of brain networks for social functions: Confirmatory analyses in a large open source dataset.

Project description:Human observers show robust activity in distinct brain networks during movie-viewing. For example, scenes that emphasize characters' thoughts evoke activity in the "Theory of Mind" (ToM) network, whereas scenes that emphasize characters' bodily sensations evoke activity in the "Pain Matrix." A prior exploratory fMRI study used a naturalistic movie-viewing stimulus to study the developmental origins of this functional dissociation, and the links between cortical and cognitive changes in children's social development (Richardson et al., 2018). To replicate and extend this work, the current study utilized a large publicly available dataset (n = 241, ages 5-20 years) (Alexander et al., 2017) who viewed "The Present" (Frey, 2014) and completed a resting state scan (n = 200) while undergoing fMRI. This study provides confirmatory evidence that 1) ToM and pain networks are functionally dissociated early in development, 2) selectivity increases with age, and in ToM regions, with a behavioral index of social reasoning. Additionally, while inter-region correlations are similar when measured during the movie and at rest, only inter-region correlations measured during movie-viewing correlated with functional maturity. This study demonstrates the scientific benefits of open source data in developmental cognitive neuroscience, and provides insight into the relationship between functional and intrinsic properties of the developing brain.

| S-EPMC6969289 | biostudies-literature

Open University Learning Analytics dataset.

Project description:Learning Analytics focuses on the collection and analysis of learners' data to improve their learning experience by providing informed guidance and to optimise learning materials. To support the research in this area we have developed a dataset, containing data from courses presented at the Open University (OU). What makes the dataset unique is the fact that it contains demographic data together with aggregated clickstream data of students' interactions in the Virtual Learning Environment (VLE). This enables the analysis of student behaviour, represented by their actions. The dataset contains the information about 22 courses, 32,593 students, their assessment results, and logs of their interactions with the VLE represented by daily summaries of student clicks (10,655,280 entries). The dataset is freely available at https://analyse.kmi.open.ac.uk/open_dataset under a CC-BY 4.0 license.

| S-EPMC5704676 | biostudies-literature

A benchmark dataset for the multiple depot vehicle scheduling problem.

Project description:This data article presents a description of a benchmark dataset for the multiple depot vehicle scheduling problem (MDVSP). The MDVSP is to assign vehicles from different depots to timetabled trips to minimize the total cost of empty travel and waiting. The dataset has been developed to evaluate the heuristics of the MDVSP that are presented in "A new formulation and a column generation-based heuristic for the multiple depot vehicle scheduling problem" (Kulkarni et al., 2018). The dataset contains 60 problem instances of varying size. Researchers can use the dataset to evaluate the future algorithms for the MDVSP and compare the performance with the existing algorithms. The dataset includes a program that can be used to generate new problem instances of the MDVSP.

| S-EPMC6312793 | biostudies-literature

Handling class imbalance problem in miRNA dataset associated with cancer.

Project description:MiRNAs are small (~22nt long) non-coding RNA sequences; binds to the complementarity target sites in 3' Untranslated Region (UTR) of mRNA sequences but not restricted to other mRNA regions viz., 5' UTR and Coding sequences (CDS). Complementarity binding of miRNA to mRNA target sites either results in complete degradation of the mRNA itself or it may regulate the mRNA as an oncogene or as a tumor suppressor gene. However, the exact mechanism involved in identifying a miRNA to be associated with cancer is still unclear. Further, with the outburst in the number of miRNAs sequences recorded every year in miRBase, the gap is still widening mainly due to the laborious and economically unfavorable experimental procedures associated with the functional annotation. Motivated by the fact, we constructed a two-step support vector machine-based predictive model - miRSEQ and miRINT. However, the major pitfall during the construction of the model is the class imbalance problem. Hence, in order to overcome class imbalance problem, in the present study we empirically compare the effectiveness of two different methods viz., Synthetic Minority Oversampling Technique (SMOTE) and cost-senstive learning method. Performance measures were evaluated in terms of Precision and Recall. Based on our result, it was observed that for miRNA dataset with high class imbalance utilized for predicting association of cancer, cost-sensitive method outperformed the oversampling method.

| S-EPMC4349932 | biostudies-literature

CompNovo datasets - Software dataset

Project description:Not available

2009-12-18 | PRD000081 | Pride

The Problem with Big Data: Operating on Smaller Datasets to Bridge the Implementation Gap.

Project description:Big datasets have the potential to revolutionize public health. However, there is a mismatch between the political and scientific optimism surrounding big data and the public's perception of its benefit. We suggest a systematic and concerted emphasis on developing models derived from smaller datasets to illustrate to the public how big data can produce tangible benefits in the long term. In order to highlight the immediate value of a small data approach, we produced a proof-of-concept model predicting hospital length of stay. The results demonstrate that existing small datasets can be used to create models that generate a reasonable prediction, facilitating health-care delivery. We propose that greater attention (and funding) needs to be directed toward the utilization of existing information resources in parallel with current efforts to create and exploit "big data."

| S-EPMC5130981 | biostudies-literature

DataMed - an open source discovery index for finding biomedical datasets.

Project description:OBJECTIVE:Finding relevant datasets is important for promoting data reuse in the biomedical domain, but it is challenging given the volume and complexity of biomedical data. Here we describe the development of an open source biomedical data discovery system called DataMed, with the goal of promoting the building of additional data indexes in the biomedical domain. MATERIALS AND METHODS:DataMed, which can efficiently index and search diverse types of biomedical datasets across repositories, is developed through the National Institutes of Health-funded biomedical and healthCAre Data Discovery Index Ecosystem (bioCADDIE) consortium. It consists of 2 main components: (1) a data ingestion pipeline that collects and transforms original metadata information to a unified metadata model, called DatA Tag Suite (DATS), and (2) a search engine that finds relevant datasets based on user-entered queries. In addition to describing its architecture and techniques, we evaluated individual components within DataMed, including the accuracy of the ingestion pipeline, the prevalence of the DATS model across repositories, and the overall performance of the dataset retrieval engine. RESULTS AND CONCLUSION:Our manual review shows that the ingestion pipeline could achieve an accuracy of 90% and core elements of DATS had varied frequency across repositories. On a manually curated benchmark dataset, the DataMed search engine achieved an inferred average precision of 0.2033 and a precision at 10 (P@10, the number of relevant results in the top 10 search results) of 0.6022, by implementing advanced natural language processing and terminology services. Currently, we have made the DataMed system publically available as an open source package for the biomedical community.

| S-EPMC7378878 | biostudies-literature

Are open set classification methods effective on large-scale datasets?

Project description:Supervised classification methods often assume the train and test data distributions are the same and that all classes in the test set are present in the training set. However, deployed classifiers often require the ability to recognize inputs from outside the training set as unknowns. This problem has been studied under multiple paradigms including out-of-distribution detection and open set recognition. For convolutional neural networks, there have been two major approaches: 1) inference methods to separate knowns from unknowns and 2) feature space regularization strategies to improve model robustness to novel inputs. Up to this point, there has been little attention to exploring the relationship between the two approaches and directly comparing performance on large-scale datasets that have more than a few dozen categories. Using the ImageNet ILSVRC-2012 large-scale classification dataset, we identify novel combinations of regularization and specialized inference methods that perform best across multiple open set classification problems of increasing difficulty level. We find that input perturbation and temperature scaling yield significantly better performance on large-scale datasets than other inference methods tested, regardless of the feature space regularization strategy. Conversely, we find that improving performance with advanced regularization schemes during training yields better performance when baseline inference techniques are used; however, when advanced inference methods are used to detect open set classes, the utility of these combersome training paradigms is less evident.

| S-EPMC7473573 | biostudies-literature

Dataset of metaheuristics for the flow shop scheduling problem with maintenance activities integrated.

Project description:This data article presents a flow shop scheduling problem in which machines are not available during the whole planning horizon and the periods of unavailability are due to random faults. The experimental dataset consists of two problems with different sizes. In the largest one, about 2400 problems were analysed and compared with two diffuse metaheuristics: Genetic Algorithm (GA) and Harmony Search (HS). In the smallest, about 600 problems were analysed comparing the solution obtained with an exhaustive algorithm with those obtained by means of GA and HS. This dataset represents a test-bed for further works, allowing a comparison between the solution quality and the computation time obtained with different optimization methods. The substantial computational effort spent to generate the dataset undoubtedly represents a significant asset for the scientific community.

| S-EPMC8049987 | biostudies-literature

OmicsDI is part of the ELIXIR infrastructure

OmicsDI is an Elixir interoperability service. Learn more ›

OmicsDI Databases

PRIDE
PeptideAtlas
MassIVE
JPOST Repository
Physiome Model Repository

EGA
EVA
ENA
LINCS
PAXDB
Cell Collective

MetaboLights
Metabolomics Workbench
MetabolomeExpress
GNPS
BioModels
FAIRDOMHub

ArrayExpress
dbGaP
ExpressionAtlas
GEO
NODE

Information

Databases
Help
API
Contact us
Code on GitHub
Terms of use
Submit Data