Dataset Information

The Missing Link Between Memory and Reinforcement Learning.

ABSTRACT: Reinforcement learning systems usually assume that a value function is defined over all states (or state-action pairs) that can immediately give the value of a particular state or action. These values are used by a selection mechanism to decide which action to take. In contrast, when humans and animals make decisions, they collect evidence for different alternatives over time and take action only when sufficient evidence has been accumulated. We have previously developed a model of memory processing that includes semantic, episodic and working memory in a comprehensive architecture. Here, we describe how this memory mechanism can support decision making when the alternatives cannot be evaluated based on immediate sensory information alone. Instead we first imagine, and then evaluate a possible future that will result from choosing one of the alternatives. Here we present an extended model that can be used as a model for decision making that depends on accumulating evidence over time, whether that information comes from the sequential attention to different sensory properties or from internal simulation of the consequences of making a particular choice. We show how the new model explains both simple immediate choices, choices that depend on multiple sensory factors and complicated selections between alternatives that require forward looking simulations based on episodic and semantic memory structures. In this framework, vicarious trial and error is explained as an internal simulation that accumulates evidence for a particular choice. We argue that a system like this forms the "missing link" between more traditional ideas of semantic and episodic memory, and the associative nature of reinforcement learning.

SUBMITTER: Balkenius C

PROVIDER: S-EPMC7758424 | biostudies-literature | 2020

REPOSITORIES: biostudies-literature

ACCESS DATA

Publications

The Missing Link Between Memory and Reinforcement Learning.

Balkenius Christian C Tjøstheim Trond A TA Johansson Birger B Wallin Annika A Gärdenfors Peter P

Frontiers in psychology 20201210

Reinforcement learning systems usually assume that a value function is defined over all states (or state-action pairs) that can immediately give the value of a particular state or action. These values are used by a selection mechanism to decide which action to take. In contrast, when humans and animals make decisions, they collect evidence for different alternatives over time and take action only when sufficient evidence has been accumulated. We have previously developed a model of memory proces ...[more]

PMID: 33362625

Similar Datasets

Project description:The origin and deep evolution of retroviruses remain largely unclear. It has been proposed that retroviruses might have originated from a Ty3/Gypsy retrotransposon, but all known Ty3/Gypsy retrotransposons are only distantly related to retroviruses. Retroviruses and some plant Athila/Tat elements (within Ty3/Gypsy retrotransposons) independently evolved a dual RNase H domain and an env/env-like gene. Here, we reported the discovery of a novel lineage of retrotransposons, designated Odin retrotransposons, in the genomes of eight sea anemones (order Actinaria) within the Cnidaria phylum. Odin retrotransposons exhibited unique genome features, encoding a dual RNase H domain (like retroviruses) but no env gene (like most Ty3/Gypsy retrotransposons). Phylogenetic analyses based on reverse transcriptase showed that Odin retrotransposons formed a sister group to lokiretroviruses, and lokiretroviruses and Odin retrotransposons together were sister to canonical retroviruses. Moreover, phylogenetic analyses based on RNase H and integrase also supported the hypothesis that Odin retrotransposons were sisters to lokiretroviruses. Lokiretroviruses and canonical retroviruses did not form a monophyletic group, indicating that lokiretroviruses and canonical retroviruses might represent two distinct virus families. Taken together, the discovery of Odin retrotransposons narrowed down the evolutionary gaps between retrotransposons and canonical retroviruses and lokiretroviruses. IMPORTANCE The origin of retroviruses remains largely unclear. In this study, we discovered a novel retrotransposon lineage, Odin retrotransposons, within the genomes of sea anemones (order Actinaria). In contrast to retroviruses and most retrotransposons, Odin retrotransposons encode a dual RNase H domain but no env gene. Phylogenetic analyses showed that Odin retrotransposons were sisters to lokiretroviruses, and lokiretroviruses and Odin retrotransposons were sisters to retroviruses, establishing an evolutionary framework to decipher the origin of retroviruses (canonical retroviruses and lokiretroviruses). Our results provided insights into the diversity and deep evolution of LTR retrotransposons closely related to retroviruses.

Project description:BackgroundFunctional traits are the primary biotic component driving organism influence on ecosystem functions; in consequence, traits are widely used in ecological research. However, most animal trait-based studies use easy-to-measure characteristics of species that are at best only weakly associated with functions. Animal-mediated pollination is a key ecosystem function and is likely to be influenced by pollinator traits, but to date no one has identified functional traits that are simple to measure and have good predictive power.MethodsHere, we show that a simple, easy to measure trait (hairiness) can predict pollinator effectiveness with high accuracy. We used a novel image analysis method to calculate entropy values for insect body surfaces as a measure of hairiness. We evaluated the power of our method for predicting pollinator effectiveness by regressing pollinator hairiness (entropy) against single visit pollen deposition (SVD) and pollen loads on insects. We used linear models and AICC model selection to determine which body regions were the best predictors of SVD and pollen load.ResultsWe found that hairiness can be used as a robust proxy of SVD. The best models for predicting SVD for the flower species Brassica rapa and Actinidia deliciosa were hairiness on the face and thorax as predictors (R2 = 0.98 and 0.91 respectively). The best model for predicting pollen load for B. rapa was hairiness on the face (R2 = 0.81).DiscussionWe suggest that the match between pollinator body region hairiness and plant reproductive structure morphology is a powerful predictor of pollinator effectiveness. We show that pollinator hairiness is strongly linked to pollination-an important ecosystem function, and provide a rigorous and time-efficient method for measuring hairiness. Identifying and accurately measuring key traits that drive ecosystem processes is critical as global change increasingly alters ecological communities, and subsequently, ecosystem functions worldwide.

Dataset Information

The Missing Link Between Memory and Reinforcement Learning.

Publications

The Missing Link Between Memory and Reinforcement Learning.

Similar Datasets

OmicsDI is part of the ELIXIR infrastructure

Tweets