Browse
Submit Data
Databases
API
Help

Dataset Information

45 Views

0 Connections

0 Citations

0 Reanalyses

0 Downloads

Omics score: 0

Experience replay is associated with efficient nonlocal learning.

ABSTRACT: To make effective decisions, people need to consider the relationship between actions and outcomes. These are often separated by time and space. The neural mechanisms by which disjoint actions and outcomes are linked remain unknown. One promising hypothesis involves neural replay of nonlocal experience. Using a task that segregates direct from indirect value learning, combined with magnetoencephalography, we examined the role of neural replay in human nonlocal learning. After receipt of a reward, we found significant backward replay of nonlocal experience, with a 160-millisecond state-to-state time lag, which was linked to efficient learning of action values. Backward replay and behavioral evidence of nonlocal learning were more pronounced for experiences of greater benefit for future behavior. These findings support nonlocal replay as a neural mechanism for solving complex credit assignment problems during learning.

SUBMITTER: Liu Y

PROVIDER: S-EPMC7610948 | biostudies-literature | 2021 May

REPOSITORIES: biostudies-literature

ACCESS DATA

Json Xml

Publications

Experience replay is associated with efficient nonlocal learning.

Liu Yunzhe Y Mattar Marcelo G MG Behrens Timothy E J TEJ Daw Nathaniel D ND Dolan Raymond J RJ

Science (New York, N.Y.) 20210501 6544

To make effective decisions, people need to consider the relationship between actions and outcomes. These are often separated by time and space. The neural mechanisms by which disjoint actions and outcomes are linked remain unknown. One promising hypothesis involves neural replay of nonlocal experience. Using a task that segregates direct from indirect value learning, combined with magnetoencephalography, we examined the role of neural replay in human nonlocal learning. After receipt of a reward ...[more]

PMID: 34016753

Similar Datasets

Map-based experience replay: a memory-efficient solution to catastrophic forgetting in reinforcement learning.

Project description:Deep reinforcement learning (RL) agents often suffer from catastrophic forgetting, forgetting previously found solutions in parts of the input space when training new data. Replay memories are a common solution to the problem by decorrelating and shuffling old and new training samples. They naively store state transitions as they arrive, without regard for redundancy. We introduce a novel cognitive-inspired replay memory approach based on the Grow-When-Required (GWR) self-organizing network, which resembles a map-based mental model of the world. Our approach organizes stored transitions into a concise environment-model-like network of state nodes and transition edges, merging similar samples to reduce the memory size and increase pair-wise distance among samples, which increases the relevancy of each sample. Overall, our study shows that map-based experience replay allows for significant memory reduction with only small decreases in performance.

| S-EPMC10333526 | biostudies-literature

Hippocampal replay of extended experience.

Project description:During pauses in exploration, ensembles of place cells in the rat hippocampus re-express firing sequences corresponding to recent spatial experience. Such "replay" co-occurs with ripple events: short-lasting (approximately 50-120 ms), high-frequency (approximately 200 Hz) oscillations that are associated with increased hippocampal-cortical communication. In previous studies, rats exploring small environments showed replay anchored to the rat's current location and compressed in time into a single ripple event. Here, we show, using a neural decoding approach, that firing sequences corresponding to long runs through a large environment are replayed with high fidelity and that such replay can begin at remote locations on the track. Extended replay proceeds at a characteristic virtual speed of approximately 8 m/s and remains coherent across trains of ripple events. These results suggest that extended replay is composed of chains of shorter subsequences, which may reflect a strategy for the storage and flexible expression of memories of prolonged experience.

| S-EPMC4364032 | biostudies-literature

A model of hippocampal replay driven by experience and environmental structure facilitates spatial learning.

Project description:Replay of neuronal sequences in the hippocampus during resting states and sleep play an important role in learning and memory consolidation. Consistent with these functions, replay sequences have been shown to obey current spatial constraints. Nevertheless, replay does not necessarily reflect previous behavior and can construct never-experienced sequences. Here, we propose a stochastic replay mechanism that prioritizes experiences based on three variables: 1. Experience strength, 2. experience similarity, and 3. inhibition of return. Using this prioritized replay mechanism to train reinforcement learning agents leads to far better performance than using random replay. Its performance is close to the state-of-the-art, but computationally intensive, algorithm by Mattar & Daw (2018). Importantly, our model reproduces diverse types of replay because of the stochasticity of the replay mechanism and experience-dependent differences between the three variables. In conclusion, a unified replay mechanism generates diverse replay statistics and is efficient in driving spatial learning.

| S-EPMC10076035 | biostudies-literature

Prioritized experience replay based on dynamics priority

Project description: Not available

| S-EPMC10933423 | biostudies-literature

Hippocampal replay of experience at real-world speeds.

Project description:Representations related to past experiences play a critical role in memory and decision-making processes. The rat hippocampus expresses these types of representations during sharp-wave ripple (SWR) events, and previous work identified a minority of SWRs that contain 'replay' of spatial trajectories at ∼20x the movement speed of the animal. Efforts to understand replay typically make multiple assumptions about which events to examine and what sorts of representations constitute replay. We therefore lack a clear understanding of both the prevalence and the range of representational dynamics associated with replay. Here, we develop a state space model that uses a combination of movement dynamics of different speeds to capture the spatial content and time evolution of replay during SWRs. Using this model, we find that the large majority of replay events contain spatially coherent, interpretable content. Furthermore, many events progress at real-world, rather than accelerated, movement speeds, consistent with actual experiences.

| S-EPMC8476125 | biostudies-literature

Clustering-based Failed goal Aware Hindsight Experience Replay.

Project description:In a multi-goal reinforcement learning environment, an agent learns a policy to perform tasks with multiple goals from experiences gained through exploration. In environments with sparse binary rewards, the replay buffer contains few successful experiences, posing a challenge for sampling efficiency. To address this, Hindsight Experience Replay (HER) generates successful experiences, named hindsight experiences, from unsuccessful ones. However, uniform sampling of experiences for the process of HER can lead to inefficient scenarios of generating hindsight experience. In this paper, a novel method called Failed goal Aware HER (FAHER) is proposed to enhance sampling efficiency. This method considers the properties of achieved goals with respect to failed goals during sampling. To account for these properties, a cluster model is used to cluster episodes in the replay buffer, and experiences are subsequently sampled in the manner of HER. The proposed method is validated through experiments on three robotic control tasks from the OpenAI Gym. The experimental results demonstrate that the proposed method is more sample-efficient and achieves improved performance over baseline approaches.

| S-EPMC11784800 | biostudies-literature

The role of experience in prioritizing hippocampal replay.

Project description:During sleep, recent memories are replayed by the hippocampus, leading to their consolidation, with a higher priority given to salient experiences. To examine the role of replay in the selective strengthening of memories, we recorded large ensembles of hippocampal place cells while male rats ran repeated spatial trajectories on two linear tracks, differing in either their familiarity or number of laps run. We observed that during sleep, the rate of replay events for a given track increased proportionally with the number of spatial trajectories run by the animal. In contrast, the rate of sleep replay events decreased if the animal was more familiar with the track. Furthermore, we find that the cumulative number of awake replay events occurring during behavior, influenced by both the novelty and duration of an experience, predicts which memories are prioritized for sleep replay, providing a more parsimonious neural correlate for the selective strengthening of memories.

| S-EPMC10710481 | biostudies-literature

Hindsight Experience Replay Improves Reinforcement Learning for Control of a MIMO Musculoskeletal Model of the Human Arm.

Project description:High-level spinal cord injuries often result in paralysis of all four limbs, leading to decreased patient independence and quality of life. Coordinated functional electrical stimulation (FES) of paralyzed muscles can be used to restore some motor function in the upper extremity. To coordinate functional movements, FES controllers should be developed to exploit the complex characteristics of human movement and produce the intended movement kinematics and/or kinetics. Here, we demonstrate the ability of a controller trained using reinforcement learning to generate desired movements of a horizontal planar musculoskeletal model of the human arm with 2 degrees of freedom and 6 actuators. The controller is given information about the kinematics of the arm, but not the internal state of the actuators. In particular, we demonstrate that a technique called "hindsight experience replay" can improve controller performance while also decreasing controller training time.

| S-EPMC8630802 | biostudies-literature

Experience-driven rate modulation is reinstated during hippocampal replay.

Project description:Replay, the sequential reactivation within a neuronal ensemble, is a central hippocampal mechanism postulated to drive memory processing. While both rate and place representations are used by hippocampal place cells to encode behavioral episodes, replay has been largely defined by only the latter - based on the fidelity of sequential activity across neighboring place fields. Here, we show that dorsal CA1 place cells in rats can modulate their firing rate between replay events of two different contexts. This experience-dependent phenomenon mirrors the same pattern of rate modulation observed during behavior and can be used independently from place information within replay sequences to discriminate between contexts. Our results reveal the existence of two complementary neural representations available for memory processes.

| S-EPMC9489210 | biostudies-literature

Hippocampal replay is not a simple function of experience.

Project description:Replay of behavioral sequences in the hippocampus during sharp wave ripple complexes (SWRs) provides a potential mechanism for memory consolidation and the learning of knowledge structures. Current hypotheses imply that replay should straightforwardly reflect recent experience. However, we find these hypotheses to be incompatible with the content of replay on a task with two distinct behavioral sequences (A and B). We observed forward and backward replay of B even when rats had been performing A for >10 min. Furthermore, replay of nonlocal sequence B occurred more often when B was infrequently experienced. Neither forward nor backward sequences preferentially represented highly experienced trajectories within a session. Additionally, we observed the construction of never-experienced novel-path sequences. These observations challenge the idea that sequence activation during SWRs is a simple replay of recent experience. Instead, replay reflected all physically available trajectories within the environment, suggesting a potential role in active learning and maintenance of the cognitive map.

| S-EPMC4460981 | biostudies-literature

OmicsDI is part of the ELIXIR infrastructure

OmicsDI is an Elixir interoperability service. Learn more ›

Tweets

OmicsDI Databases

PRIDE
PeptideAtlas
MassIVE
JPOST Repository
Physiome Model Repository

EGA
EVA
ENA
LINCS
PAXDB
Cell Collective

MetaboLights
Metabolomics Workbench
MetabolomeExpress
GNPS
BioModels
FAIRDOMHub

ArrayExpress
dbGaP
ExpressionAtlas
GEO
NODE

Information

Databases
Help
API
Contact us
Code on GitHub
Terms of use
Submit Data