Dataset Information

Lateral intraparietal cortex and reinforcement learning during a mixed-strategy game.

ABSTRACT: Activity of the neurons in the lateral intraparietal cortex (LIP) displays a mixture of sensory, motor, and memory signals. Moreover, they often encode signals reflecting the accumulation of sensory evidence that certain eye movements might lead to a desirable outcome. However, when the environment changes dynamically, animals are also required to combine the information about its previously chosen actions and their outcomes appropriately to update continually the desirabilities of alternative actions. Here, we investigated whether LIP neurons encoded signals necessary to update an animal's decision-making strategies adaptively during a computer-simulated matching-pennies game. Using a reinforcement learning algorithm, we estimated the value functions that best predicted the animal's choices on a trial-by-trial basis. We found that, immediately before the animal revealed its choice, approximately 18% of LIP neurons changed their activity according to the difference in the value functions for the two targets. In addition, a somewhat higher fraction of LIP neurons displayed signals related to the sum of the value functions, which might correspond to the state value function or an average rate of reward used as a reference point. Similar to the neurons in the prefrontal cortex, many LIP neurons also encoded the signals related to the animal's previous choices. Thus, the posterior parietal cortex might be a part of the network that provides the substrate for forming appropriate associations between actions and outcomes.

SUBMITTER: Seo H

PROVIDER: S-EPMC2743508 | biostudies-other | 2009 Jun

REPOSITORIES: biostudies-other

ACCESS DATA

Publications

Lateral intraparietal cortex and reinforcement learning during a mixed-strategy game.

Seo Hyojung H Barraclough Dominic J DJ Lee Daeyeol D

The Journal of neuroscience : the official journal of the Society for Neuroscience 20090601 22

Activity of the neurons in the lateral intraparietal cortex (LIP) displays a mixture of sensory, motor, and memory signals. Moreover, they often encode signals reflecting the accumulation of sensory evidence that certain eye movements might lead to a desirable outcome. However, when the environment changes dynamically, animals are also required to combine the information about its previously chosen actions and their outcomes appropriately to update continually the desirabilities of alternative a ...[more]

PMID: 19494150

Similar Datasets

Project description:The decisions we make are sometimes influenced by interactions with other agents. Previous studies have suggested that the prefrontal cortex plays an important role in decision-making and that the dopamine system underlies processes of motivation, motor preparation, and reinforcement learning. However, the physiological mechanisms underlying how the prefrontal cortex and the dopaminergic system are involved in decision-making remain largely unclear. The present study aimed to determine how decision strategies influence event-related potentials (ERPs). We also tested the effect of levodopa, a dopamine precursor, on decision-making and ERPs in a randomized double-blind placebo-controlled investigation. The subjects performed a matching-pennies task against an opposing virtual computer player by choosing between right and left targets while their ERPs were recorded. According to the rules of the matching-pennies task, the subject won the trial when they chose the same side as the opponent, and lost otherwise. We set three different task rules: (1) with the alternation (ALT) rule, the computer opponent made alternating choices of right and left in sequential trials; (2) with the random (RAND) rule, the opponent randomly chose between right and left; and (3) with the GAME rule, the opponent analyzed the subject's past choices to predict the subject's next choice, and then chose the opposite side. A sustained medial ERP became more negative toward the time of the subject's target choice. A biphasic potential appeared when the opponent's choice was revealed after the subject's response. The ERPs around the subject's choice were greater in RAND and GAME than in ALT, and the negative peak was enhanced by levodopa. In addition to these medial ERPs, we observed lateral frontal ERPs tuned to the choice direction. The signals emerged around the choice period selectively in RAND and GAME when levodopa was administered. These results suggest that decision processes are modulated by the dopamine system when a complex and strategic decision is required, which may reflect decision updating with dopaminergic prediction error signals.

Project description:Recent models of spiking neuronal networks have been trained to perform behaviors in static environments using a variety of learning rules, with varying degrees of biological realism. Most of these models have not been tested in dynamic visual environments where models must make predictions on future states and adjust their behavior accordingly. The models using these learning rules are often treated as black boxes, with little analysis on circuit architectures and learning mechanisms supporting optimal performance. Here we developed visual/motor spiking neuronal network models and trained them to play a virtual racket-ball game using several reinforcement learning algorithms inspired by the dopaminergic reward system. We systematically investigated how different architectures and circuit-motifs (feed-forward, recurrent, feedback) contributed to learning and performance. We also developed a new biologically-inspired learning rule that significantly enhanced performance, while reducing training time. Our models included visual areas encoding game inputs and relaying the information to motor areas, which used this information to learn to move the racket to hit the ball. Neurons in the early visual area relayed information encoding object location and motion direction across the network. Neuronal association areas encoded spatial relationships between objects in the visual scene. Motor populations received inputs from visual and association areas representing the dorsal pathway. Two populations of motor neurons generated commands to move the racket up or down. Model-generated actions updated the environment and triggered reward or punishment signals that adjusted synaptic weights so that the models could learn which actions led to reward. Here we demonstrate that our biologically-plausible learning rules were effective in training spiking neuronal network models to solve problems in dynamic environments. We used our models to dissect the circuit architectures and learning rules most effective for learning. Our model shows that learning mechanisms involving different neural circuits produce similar performance in sensory-motor tasks. In biological networks, all learning mechanisms may complement one another, accelerating the learning capabilities of animals. Furthermore, this also highlights the resilience and redundancy in biological systems.

Dataset Information

Lateral intraparietal cortex and reinforcement learning during a mixed-strategy game.

Publications

Lateral intraparietal cortex and reinforcement learning during a mixed-strategy game.

Similar Datasets

OmicsDI is part of the ELIXIR infrastructure

Tweets