Dataset Information

Quantifying machine influence over human forecasters.

ABSTRACT: Crowdsourcing human forecasts and machine learning models each show promise in predicting future geopolitical outcomes. Crowdsourcing increases accuracy by pooling knowledge, which mitigates individual errors. On the other hand, advances in machine learning have led to machine models that increase accuracy due to their ability to parameterize and adapt to changing environments. To capitalize on the unique advantages of each method, recent efforts have shown improvements by "hybridizing" forecasts-pairing human forecasters with machine models. This study analyzes the effectiveness of such a hybrid system. In a perfect world, independent reasoning by the forecasters combined with the analytic capabilities of the machine models should complement each other to arrive at an ultimately more accurate forecast. However, well-documented biases describe how humans often mistrust and under-utilize such models in their forecasts. In this work, we present a model that can be used to estimate the trust that humans assign to a machine. We use forecasts made in the absence of machine models as prior beliefs to quantify the weights placed on the models. Our model can be used to uncover other aspects of forecasters' decision-making processes. We find that forecasters trust the model rarely, in a pattern that suggests they treat machine models similarly to expert advisors, but only the best forecasters trust the models when they can be expected to perform well. We also find that forecasters tend to choose models that conform to their prior beliefs as opposed to anchoring on the model forecast. Our results suggest machine models can improve the judgment of a human pool but highlight the importance of accounting for trust and cognitive biases involved in the human judgment process.

SUBMITTER: Abeliuk A

PROVIDER: S-EPMC7524768 | biostudies-literature | 2020 Sep

REPOSITORIES: biostudies-literature

ACCESS DATA

Publications

Quantifying machine influence over human forecasters.

Abeliuk Andrés A Benjamin Daniel M DM Morstatter Fred F Galstyan Aram A

Scientific reports 20200929 1

Crowdsourcing human forecasts and machine learning models each show promise in predicting future geopolitical outcomes. Crowdsourcing increases accuracy by pooling knowledge, which mitigates individual errors. On the other hand, advances in machine learning have led to machine models that increase accuracy due to their ability to parameterize and adapt to changing environments. To capitalize on the unique advantages of each method, recent efforts have shown improvements by "hybridizing" forecast ...[more]

PMID: 32994447

Similar Datasets

Project description:Realistically parameterized and dynamically simulated food-webs are useful tool to explore the importance of the functional diversity of ecosystems, and in particular relations between the dynamics of species and the whole community. We present a stochastic dynamical food web simulation for the Kelian River (Borneo). The food web was constructed for six different locations, arrayed along a gradient of increasing human perturbation (mostly resulting from gold mining activities) along the river. Along the river, the relative importance of grazers, filterers and shredders decreases with increasing disturbance downstream, while predators become more dominant in governing eco-dynamics. Human activity led to increased turbidity and sedimentation which adversely impacts primary productivity. Since the main difference between the study sites was not the composition of the food webs (structure is quite similar) but the strengths of interactions and the abundance of the trophic groups, a dynamical simulation approach seemed to be useful to better explain human influence. In the pristine river (study site 1), when comparing a structural version of our model with the dynamical model we found that structurally central groups such as omnivores and carnivores were not the most important ones dynamically. Instead, primary consumers such as invertebrate grazers and shredders generated a greater dynamical response. Based on the dynamically most important groups, bottom-up control is replaced by the predominant top-down control regime as distance downstream and human disturbance increased. An important finding, potentially explaining the poor structure to dynamics relationship, is that indirect effects are at least as important as direct ones during the simulations. We suggest that our approach and this simulation framework could serve systems-based conservation efforts. Quantitative indicators on the relative importance of trophic groups and the mechanistic modeling of eco-dynamics could greatly contribute to understanding various aspects of functional diversity.

Project description:Evaluating microbial community composition through next-generation sequencing has become increasingly accessible. However, metagenomic sequencing data sets provide researchers with only a snapshot of a dynamic ecosystem and do not provide information about the total microbial number, or load, of a sample. Additionally, DNA can be detected long after a microorganism is dead, making it unsafe to assume that all microbial sequences detected in a community came from living organisms. By combining relic DNA removal by propidium monoazide (PMA) with microbial quantification with flow cytometry, we present a novel workflow to quantify live microbial load in parallel with metagenomic sequencing. We applied this method to unstimulated saliva samples, which can easily be collected longitudinally and standardized by passive collection time. We found that the number of live microorganisms detected in saliva was inversely correlated with salivary flow rate and fluctuated by an order of magnitude throughout the day in healthy individuals. In an acute perturbation experiment, alcohol-free mouthwash resulted in a massive decrease in live bacteria, which would have been missed if we did not consider dead cell signal. While removing relic DNA from saliva samples did not greatly impact the microbial composition, it did increase our resolution among samples collected over time. These results provide novel insight into the dynamic nature of host-associated microbiomes and underline the importance of applying scale-invariant tools in the analysis of next-generation sequencing data sets.IMPORTANCE Human microbiomes are dynamic ecosystems often composed of hundreds of unique microbial taxa. To detect fluctuations over time in the human oral microbiome, we developed a novel workflow to quantify live microbial cells with flow cytometry in parallel with next-generation sequencing, and applied this method to over 150 unstimulated, timed saliva samples. Microbial load was inversely correlated with salivary flow rate and fluctuated by an order of magnitude within a single participant throughout the day. Removing relic DNA improved our ability to distinguish samples over time and revealed that the percentage of sequenced bacteria in a given saliva sample that are alive can range from nearly 0% up to 100% throughout a typical day. These findings highlight the dynamic ecosystem of the human oral microbiome and the benefit of removing relic DNA signals in longitudinal microbiome study designs.

Dataset Information

Quantifying machine influence over human forecasters.

Publications

Quantifying machine influence over human forecasters.

Similar Datasets

OmicsDI is part of the ELIXIR infrastructure

Tweets