Reproducible biomedical benchmarking in the cloud: lessons from crowd-sourced data challenges.
Ontology highlight
ABSTRACT:
SUBMITTER: Ellrott K
PROVIDER: S-EPMC6737594 | biostudies-literature | 2019 Sep
REPOSITORIES: biostudies-literature
Publications
Reproducible biomedical benchmarking in the cloud: lessons from crowd-sourced data challenges.
Ellrott Kyle K Buchanan Alex A Creason Allison A Mason Michael M Schaffter Thomas T Hoff Bruce B Eddy James J Chilton John M JM Yu Thomas T Stuart Joshua M JM Saez-Rodriguez Julio J Stolovitzky Gustavo G Boutros Paul C PC Guinney Justin J
Genome biology 20190910 1
Challenges are achieving broad acceptance for addressing many biomedical questions and enabling tool assessment. But ensuring that the methods evaluated are reproducible and reusable is complicated by the diversity of software architectures, input and output file formats, and computing environments. To mitigate these problems, some challenges have leveraged new virtualization and compute methods, requiring participants to submit cloud-ready software packages. We review recent data challenges wit ...[more]
Similar Datasets
Project description:In 2013, in response to an epidemic of ash dieback disease in England the previous year, we launched a Facebook-based game called Fraxinus to enable non-scientists to contribute to genomics studies of the pathogen that causes the disease and the ash trees that are devastated by it. Over a period of 51 weeks players were able to match computational alignments of genetic sequences in 78% of cases, and to improve them in 15% of cases. We also found that most players were only transiently interested in the game, and that the majority of the work done was performed by a small group of dedicated players. Based on our experiences we have built a linear model for the length of time that contributors are likely to donate to a crowd-sourced citizen science project. This model could serve a guide for the design and implementation of future crowd-sourced citizen science initiatives.
| S-EPMC4517073 | biostudies-literature
Project description:IntroductionTo mitigate the COVID-19 pandemic and prevent overwhelming the healthcare system, social-distancing policies such as school closure, stay-at-home orders, and indoor dining closure have been utilized worldwide. These policies function by reducing the rate of close contact within populations and result in decreased human mobility. Adherence to social distancing can substantially reduce disease spread. Thus, quantifying human mobility and social-distancing compliance, especially at high temporal resolution, can provide great insight into the impact of social distancing policies.MethodsWe used the movement of individuals around New York City (NYC), measured via traffic levels, as a proxy for human mobility and the impact of social-distancing policies (i.e., work from home policies, school closure, indoor dining closure etc.). By data mining Google traffic in real-time, and applying image processing, we derived high resolution time series of traffic in NYC. We used time series decomposition and generalized additive models to quantify changes in rush hour/non-rush hour, and weekday/weekend traffic, pre-pandemic and following the roll-out of multiple social distancing interventions.ResultsMobility decreased sharply on March 14, 2020 following declaration of the pandemic. However, levels began rebounding by approximately April 13, almost 2 months before stay-at-home orders were lifted, indicating premature increase in mobility, which we term social-distancing fatigue. We also observed large impacts on diurnal traffic congestion, such that the pre-pandemic bi-modal weekday congestion representing morning and evening rush hour was dramatically altered. By September, traffic congestion rebounded to approximately 75% of pre-pandemic levels.ConclusionUsing crowd-sourced traffic congestion data, we described changes in mobility in Manhattan, NYC, during the COVID-19 pandemic. These data can be used to inform human mobility changes during the current pandemic, in planning for responses to future pandemics, and in understanding the potential impact of large-scale traffic interventions such as congestion pricing policies.
| S-EPMC8403631 | biostudies-literature
Project description:BackgroundBacterial resistance to antibiotics is a growing health problem that is projected to cause more deaths than cancer by 2050. Consequently, novel antibiotics are urgently needed. Since more than half of the available antibiotics target the structurally conserved bacterial ribosomes, factors involved in protein synthesis are thus prime targets for the development of novel antibiotics. However, experimental identification of these potential antibiotic target proteins can be labor-intensive and challenging, as these proteins are likely to be poorly characterized and specific to few bacteria. Here, we use a bioinformatics approach to identify novel components of protein synthesis.ResultsIn order to identify these novel proteins, we established a Large-Scale Transcriptomic Analysis Pipeline in Crowd (LSTrAP-Crowd), where 285 individuals processed 26 terabytes of RNA-sequencing data of the 17 most notorious bacterial pathogens. In total, the crowd processed 26,269 RNA-seq experiments and used the data to construct gene co-expression networks, which were used to identify more than a hundred uncharacterized genes that were transcriptionally associated with protein synthesis. We provide the identity of these genes together with the processed gene expression data.ConclusionsWe identified genes related to protein synthesis in common bacterial pathogens and thus provide a resource of potential antibiotic development targets for experimental validation. The data can be used to explore additional vulnerabilities of bacteria, while our approach demonstrates how the processing of gene expression data can be easily crowd-sourced.
| S-EPMC7470450 | biostudies-literature
Project description:The Internet has dramatically expanded citizens' access to and ability to engage with political information. On many websites, any user can contribute and edit "crowd-sourced" information about important political figures. One of the most prominent examples of crowd-sourced information on the Internet is Wikipedia, a free and open encyclopedia created and edited entirely by users, and one of the world's most accessed websites. While previous studies of crowd-sourced information platforms have found them to be accurate, few have considered biases in what kinds of information are included. We report the results of four randomized field experiments that sought to explore what biases exist in the political articles of this collaborative website. By randomly assigning factually true but either positive or negative and cited or uncited information to the Wikipedia pages of U.S. senators, we uncover substantial evidence of an editorial bias toward positivity on Wikipedia: Negative facts are 36% more likely to be removed by Wikipedia editors than positive facts within 12 hours and 29% more likely within 3 days. Although citations substantially increase an edit's survival time, the editorial bias toward positivity is not eliminated by inclusion of a citation. We replicate this study on the Wikipedia pages of deceased as well as recently retired but living senators and find no evidence of an editorial bias in either. Our results demonstrate that crowd-sourced information is subject to an editorial bias that favors the politically active.
| S-EPMC4558055 | biostudies-literature
Project description:Survey data sets are often wider than they are long. This high ratio of variables to observations raises concerns about overfitting during prediction, making informed variable selection important. Recent applications in computer science have sought to incorporate human knowledge into machine-learning methods to address these problems. The authors implement such a "human-in-the-loop" approach in the Fragile Families Challenge. The authors use surveys to elicit knowledge from experts and laypeople about the importance of different variables to different outcomes. This strategy offers the option to subset the data before prediction or to incorporate human knowledge as scores in prediction models, or both together. The authors find that human intervention is not obviously helpful. Human-informed subsetting reduces predictive performance, and considered alone, approaches incorporating scores perform marginally worse than approaches that do not. However, incorporating human knowledge may still improve predictive performance, and future research should consider new ways of doing so.
| S-EPMC8112737 | biostudies-literature
Project description:Collecting large datasets of amputee gait data is notoriously difficult. Additionally, collecting data on less prevalent amputations or on gait activities other than level walking and running on hard surfaces is rarely attempted. However, with the wealth of user-generated content on the Internet, the scope for collecting amputee gait data from alternative sources other than traditional gait labs is intriguing. Here we investigate the potential of YouTube videos to provide gait data on amputee walking. We use an example dataset of trans-femoral amputees level walking at self-selected speeds to collect temporal gait parameters and calculate gait asymmetry. We compare our YouTube data with typical literature values, and show that our methodology produces results that are highly comparable to data collected in a traditional manner. The similarity between the results of our novel methodology and literature values lends confidence to our technique. Nevertheless, clear challenges with the collection and interpretation of crowd-sourced gait data remain, including long term access to datasets, and a lack of validity and reliability studies in this area.
| S-EPMC5072704 | biostudies-literature
Project description:Seafloor mapping can offer important insights for marine management, spatial planning, and research in marine geology, ecology, and oceanography. Here, we present a method for generating regional bathymetry and geomorphometry maps from crowd-sourced depth soundings (Olex AS) for a small fraction of the cost of multibeam data collection over the same area. Empirical Bayesian Kriging was used to generate a continuous bathymetric surface from incomplete and, in some areas, sparse Olex coverage on the Newfoundland and Labrador shelves of eastern Canada. The result is a 75m bathymetric grid that provides over 100x finer spatial resolution than previously available for the majority of the 672,900 km2 study area. The interpolated bathymetry was tested for accuracy against independent depth data provided by Fisheries and Oceans Canada (Spearman correlation = 0.99, p<0.001). Quantitative terrain attributes were generated to better understand seascape characteristics at multiple spatial scales, including slope, rugosity, aspect, and bathymetric position index. Landform classification was carried out using the geomorphons algorithm and a novel method for the identification of previously unmapped tributary canyons at the continental shelf edge are also presented to illustrate some of many potential benefits of crowd-sourced regional seafloor mapping.
| S-EPMC6557478 | biostudies-literature
Project description:Noise is a major source of pollution with a strong impact on health. Noise assessment is therefore a very important issue to reduce its impact on humans. To overcome the limitations of the classical method of noise assessment (such as simulation tools or noise observatories), alternative approaches have been developed, among which is collaborative noise measurement via a smartphone. Following this approach, the NoiseCapture application was proposed, in an open science framework, providing free access to a considerable amount of information and offering interesting perspectives of spatial and temporal noise analysis for the scientific community. After more than 3 years of operation, the amount of collected data is considerable. Its exploitation for a sound environment analysis, however, requires one to consider the intrinsic limits of each collected information, defined, for example, by the very nature of the data, the measurement protocol, the technical performance of the smartphone, the absence of calibration, the presence of anomalies in the collected data, etc. The purpose of this article is thus to provide enough information, in terms of quality, consistency, and completeness of the data, so that everyone can exploit the database, in full control.
| S-EPMC8345695 | biostudies-literature
Project description:BackgroundVideo-based assessment by experts may structurally measure surgical performance using procedure-specific competency assessment tools (CATs). A CAT for minimally invasive esophagectomy (MIE-CAT) was developed and validated previously. However, surgeon's time is scarce and video assessment is time-consuming and labor intensive. This study investigated non-procedure-specific assessment of MIE video clips by MIE experts and crowdsourcing, collective surgical performance evaluation by anonymous and untrained laypeople, to assist procedure-specific expert review.MethodsTwo surgical performance scoring frameworks were used to assess eight MIE videos. First, global performance was assessed with the non-procedure-specific Global Operative Assessment of Laparoscopic Skills (GOALS) of 64 procedural phase-based video clips < 10 min. Each clip was assessed by two MIE experts and > 30 crowd workers. Second, the same experts assessed procedure-specific performance with the MIE-CAT of the corresponding full-length video. Reliability and convergent validity of GOALS for MIE were investigated using hypothesis testing with correlations (experience, blood loss, operative time, and MIE-CAT).ResultsLess than 75% of hypothesized correlations between GOALS scores and experience of the surgical team (r < 0.3), blood loss (r = - 0.82 to 0.02), operative time (r = - 0.42 to 0.07), and the MIE-CAT scores (r = - 0.04 to 0.76) were met for both crowd workers and experts. Interestingly, experts' GOALS and MIE-CAT scores correlated strongly (r = 0.40 to 0.79), while crowd workers' GOALS and experts' MIE-CAT scores correlations were weak (r = - 0.04 to 0.49). Expert and crowd worker GOALS scores correlated poorly (ICC ≤ 0.42).ConclusionGOALS assessments by crowd workers lacked convergent validity and showed poor reliability. It is likely that MIE is technically too difficult to assess for laypeople. Convergent validity of GOALS assessments by experts could also not be established. GOALS might not be comprehensive enough to assess detailed MIE performance. However, expert's GOALS and MIE-CAT scores strongly correlated indicating video clip (instead of full-length video) assessments could be useful to shorten assessment time.
| S-EPMC10520122 | biostudies-literature
Project description:Accurate maps of species ranges are essential to inform conservation, but time-consuming to produce and update. Given the pace of change of knowledge about species distributions and shifts in ranges under climate change and land use, a need exists for timely mapping approaches that enable batch processing employing widely available data. We develop a systematic approach of batch-processing range maps and derived Area of Habitat maps for terrestrial bird species with published ranges below 125,000 km2 in Central and South America. (Area of Habitat is the habitat available to a species within its range.) We combine existing range maps with the rapidly expanding crowd-sourced eBird data of presences and absences from frequently surveyed locations, plus readily accessible, high resolution satellite data on forest cover and elevation to map the Area of Habitat available to each species. Users can interrogate the maps produced to see details of the observations that contributed to the ranges. Previous estimates of Areas of Habitat were constrained within the published ranges and thus were, by definition, smaller—typically about 30%. This reflects how little habitat within suitable elevation ranges exists within the published ranges. Our results show that on average, Areas of Habitat are 12% larger than published ranges, reflecting the often-considerable extent that eBird records expand the known distributions of species. Interestingly, there are substantial differences between threatened and non-threatened species. Some 40% of Critically Endangered, 43% of Endangered, and 55% of Vulnerable species have Areas of Habitat larger than their published ranges, compared with 31% for Near Threatened and Least Concern species. The important finding for conservation is that threatened species are generally more widespread than previously estimated.
| S-EPMC8612558 | biostudies-literature