Learning Reward Uncertainty in the Basal Ganglia.
Ontology highlight
ABSTRACT:
SUBMITTER: Mikhael JG
PROVIDER: S-EPMC5010205 | biostudies-literature | 2016 Sep
REPOSITORIES: biostudies-literature
Publications
Learning Reward Uncertainty in the Basal Ganglia.
Mikhael John G JG Bogacz Rafal R
PLoS computational biology 20160902 9
Learning the reliability of different sources of rewards is critical for making optimal choices. However, despite the existence of detailed theory describing how the expected reward is learned in the basal ganglia, it is not known how reward uncertainty is estimated in these circuits. This paper presents a class of models that encode both the mean reward and the spread of the rewards, the former in the difference between the synaptic weights of D1 and D2 neurons, and the latter in their sum. In ...[more]
Similar Datasets
Project description:To accurately predict rewards associated with states or actions, the variability of observations has to be taken into account. In particular, when the observations are noisy, the individual rewards should have less influence on tracking of average reward, and the estimate of the mean reward should be updated to a smaller extent after each observation. However, it is not known how the magnitude of the observation noise might be tracked and used to control prediction updates in the brain reward system. Here, we introduce a new model that uses simple, tractable learning rules that track the mean and standard deviation of reward, and leverages prediction errors scaled by uncertainty as the central feedback signal. We show that the new model has an advantage over conventional reinforcement learning models in a value tracking task, and approaches a theoretic limit of performance provided by the Kalman filter. Further, we propose a possible biological implementation of the model in the basal ganglia circuit. In the proposed network, dopaminergic neurons encode reward prediction errors scaled by standard deviation of rewards. We show that such scaling may arise if the striatal neurons learn the standard deviation of rewards and modulate the activity of dopaminergic neurons. The model is consistent with experimental findings concerning dopamine prediction error scaling relative to reward magnitude, and with many features of striatal plasticity. Our results span across the levels of implementation, algorithm, and computation, and might have important implications for understanding the dopaminergic prediction error signal and its relation to adaptive and effective learning.
| S-EPMC9182698 | biostudies-literature
Project description:The basal ganglia are equipped with inhibitory and disinhibitory mechanisms that enable a subject to choose valuable objects and actions. Notably, a value can be determined flexibly by recent experience or stably by prolonged experience. Recent studies have revealed that the head and tail of the caudate nucleus selectively and differentially process flexible and stable values of visual objects. These signals are sent to the superior colliculus through different parts of the substantia nigra so that the animal looks preferentially at high-valued objects, but in different manners. Thus, relying on short-term value memories, the caudate head circuit allows the subject's gaze to move expectantly to recently valued objects. Relying on long-term value memories, the caudate tail circuit allows the subject's gaze to move automatically to previously valued objects. The basal ganglia also contain an equivalent parallel mechanism for action values. Such flexible-stable parallel mechanisms for object and action values create a highly adaptable system for decision making.
| S-EPMC4148825 | biostudies-literature
Project description:BackgroundBesides motor function, the basal ganglia have been implicated in feedback learning. In patients with chronic basal ganglia infarcts, deficits in reward-based reversal learning have previously been described.MethodsWe re-examined the acquisition and reversal of stimulus-stimulus-reward associations and acquired equivalence in eleven patients with acute basal ganglia stroke (8 men, 3 women; 57.8±13.3 years), whose performance was compared eleven healthy subjects of comparable age, sex distribution and education, who were recruited outside the hospital. Eleven hospitalized patients with a similar vascular risk profile as the stroke patients but without stroke history served as clinical control group.ResultsIn a neuropsychological assessment 7±3 days post-stroke, verbal and spatial short-term and working memory and inhibition control did not differ between groups. Compared with healthy subjects, control patients with vascular risk factors exhibited significantly reduced performance in the reversal phase (F[2,30] = 3.47; p = 0.044; post-hoc comparison between risk factor controls and healthy controls: p = 0.030), but not the acquisition phase (F[2,30] = 1.01; p = 0.376) and the acquired equivalence (F[2,30] = 1.04; p = 0.367) tasks. In all tasks, the performance of vascular risk factor patients closely resembled that of basal ganglia stroke patients. Correlation studies revealed a significant association of the number of vascular risk factors with reversal learning (r = -0.33, p = 0.012), but not acquisition learning (r = -0.20, p = 0.121) or acquired equivalence (r = -0.22, p = 0.096).ConclusionsThe previously reported impairment of reward-based learning may be attributed to vascular risk factors and associated diseases, which are enriched in stroke patients. This study emphasizes the necessity of appropriate control subjects in cognition studies.
| S-EPMC4862682 | biostudies-literature
Project description:Goal-directed decision making in biological systems is broadly based on associations between conditional and unconditional stimuli. This can be further classified as classical conditioning (correlation-based learning) and operant conditioning (reward-based learning). A number of computational and experimental studies have well established the role of the basal ganglia in reward-based learning, where as the cerebellum plays an important role in developing specific conditioned responses. Although viewed as distinct learning systems, recent animal experiments point toward their complementary role in behavioral learning, and also show the existence of substantial two-way communication between these two brain structures. Based on this notion of co-operative learning, in this paper we hypothesize that the basal ganglia and cerebellar learning systems work in parallel and interact with each other. We envision that such an interaction is influenced by reward modulated heterosynaptic plasticity (RMHP) rule at the thalamus, guiding the overall goal directed behavior. Using a recurrent neural network actor-critic model of the basal ganglia and a feed-forward correlation-based learning model of the cerebellum, we demonstrate that the RMHP rule can effectively balance the outcomes of the two learning systems. This is tested using simulated environments of increasing complexity with a four-wheeled robot in a foraging task in both static and dynamic configurations. Although modeled with a simplified level of biological abstraction, we clearly demonstrate that such a RMHP induced combinatorial learning mechanism, leads to stabler and faster learning of goal-directed behaviors, in comparison to the individual systems. Thus, in this paper we provide a computational model for adaptive combination of the basal ganglia and cerebellum learning systems by way of neuromodulated plasticity for goal-directed decision making in biological and bio-mimetic organisms.
| S-EPMC4211401 | biostudies-literature
Project description:Recent studies have shown that the cerebellum and the basal ganglia are interconnected at subcortical levels. However, a subcortical basal ganglia connection to the inferior olive (IO), being the source of the olivocerebellar climbing fiber system, is not known. We have used classical tracing with CTb, retrograde transneuronal infection with wildtype rabies virus, conditional tracing with genetically modified rabies virus, and examination of material made available by the Allen Brain Institute, to study potential basal ganglia connections to the inferior olive in rats and mice. We show in both species that parvalbumin-positive, and therefore GABAergic, neurons in the entopeduncular nucleus, representing the rodent equivalent of the internal part of the globus pallidus, innervate a group of cells that surrounds the fasciculus retroflexus and that are collectively known as the area parafascicularis prerubralis. As these neurons supply a direct excitatory input to large parts of the inferior olivary complex, we propose that the entopeduncular nucleus, as a main output station of the basal ganglia, provides an inhibitory influence on olivary excitability. As such, this connection may influence olivary involvement in cerebellar learning and/or could be involved in transmission of reward properties that have recently been established for olivocerebellar signaling.
| S-EPMC10196041 | biostudies-literature
Project description:We learn complex skills such as speech and dance through a gradual process of trial and error. Cortical-basal ganglia circuits have an important yet unresolved function in this trial-and-error skill learning; influential 'actor-critic' models propose that basal ganglia circuits generate a variety of behaviours during training and learn to implement the successful behaviours in their repertoire. Here we show that the anterior forebrain pathway (AFP), a cortical-basal ganglia circuit, contributes to skill learning even when it does not contribute to such 'exploratory' variation in behavioural performance during training. Blocking the output of the AFP while training Bengalese finches to modify their songs prevented the gradual improvement that normally occurs in this complex skill during training. However, unblocking the output of the AFP after training caused an immediate transition from naive performance to excellent performance, indicating that the AFP covertly gained the ability to implement learned skill performance without contributing to skill practice. In contrast, inactivating the output nucleus of the AFP during training completely prevented learning, indicating that learning requires activity within the AFP during training. Our results suggest a revised model of skill learning: basal ganglia circuits can monitor the consequences of behavioural variation produced by other brain regions and then direct those brain regions to implement more successful behaviours. The ability of the AFP to identify successful performances generated by other brain regions indicates that basal ganglia circuits receive a detailed efference copy of premotor activity in those regions. The capacity of the AFP to implement successful performances that were initially produced by other brain regions indicates precise functional connections between basal ganglia circuits and the motor regions that directly control performance.
| S-EPMC3377745 | biostudies-other
Project description:Learning vocal behaviors, like speech and birdsong, is thought to rely on continued performance evaluation. Whether candidate performance evaluation circuits in the brain are sufficient to guide vocal learning is not known. Here, we test the sufficiency of VTA projections to the vocal basal ganglia in singing zebra finches, a songbird species that learns to produce a complex and stereotyped multi-syllabic courtship song during development. We optogenetically manipulate VTA axon terminals in singing birds contingent on how the pitch of an individual song syllable is naturally performed. We find that optical inhibition and excitation of VTA terminals are each sufficient to reliably guide learned changes in song. Inhibition and excitation have opponent effects on future performances of targeted song syllables, consistent with positive and negative reinforcement of performance outcomes. These findings define a central role for reinforcement mechanisms in learning vocalizations and demonstrate minimal circuit elements for learning vocal behaviors. VIDEO ABSTRACT.
| S-EPMC5918681 | biostudies-literature
Project description:BackgroundChildhood adversity increases the risk of psychopathology, but the neurobiological mechanisms underlying this vulnerability are not well-understood. In animal models, early adversity is associated with dysfunction in basal ganglia regions involved in reward processing, but this relationship has not been established in humans.MethodsFunctional magnetic resonance imaging was used to examine basal ganglia responses to: 1) cues signaling possible monetary rewards and losses; and 2) delivery of monetary gains and penalties, in 13 young adults who experienced maltreatment before age 14 years and 31 nonmaltreated control subjects.ResultsRelative to control subjects, individuals exposed to childhood adversity reported elevated symptoms of anhedonia and depression, rated reward cues less positively, and displayed a weaker response to reward cues in the left globus pallidus. There were no group differences in right hemisphere basal ganglia response to reward cues or in basal ganglia response to loss cues, no-incentive cues, gains, or penalties.ConclusionsResults indicate that childhood adversity in humans is associated with blunted subjective responses to reward-predicting cues as well as dysfunction in left basal ganglia regions implicated in reward-related learning and motivation. This dysfunction might serve as a diathesis that contributes to the multiple negative outcomes and psychopathologies associated with childhood adversity. The findings suggest that interventions that target motivation and goal-directed action might be useful for reducing the negative consequences of childhood adversity.
| S-EPMC2883459 | biostudies-literature
Project description:Music can trigger emotional responses in a more direct way than any other stimulus. In particular, music-evoked pleasure involves brain networks that are part of the reward system. Furthermore, rhythmic music stimulates the basal ganglia and may trigger involuntary movements to the beat. In the present study, we created a continuously playing rhythmic, dance floor-like composition where the ambient noise from the MR scanner was incorporated as an additional instrument of rhythm. By treating this continuous stimulation paradigm as a variant of resting-state, the data was analyzed with stochastic dynamic causal modeling (sDCM), which was used for exploring functional dependencies and interactions between core areas of auditory perception, rhythm processing, and reward processing. The sDCM model was a fully connected model with the following areas: auditory cortex, putamen/pallidum, and ventral striatum/nucleus accumbens of both hemispheres. The resulting estimated parameters were compared to ordinary resting-state data, without an additional continuous stimulation. Besides reduced connectivity within the basal ganglia, the results indicated a reduced functional connectivity of the reward system, namely the right ventral striatum/nucleus accumbens from and to the basal ganglia and auditory network while listening to rhythmic music. In addition, the right ventral striatum/nucleus accumbens demonstrated also a change in its hemodynamic parameter, reflecting an increased level of activation. These converging results may indicate that the dopaminergic reward system reduces its functional connectivity and relinquishing its constraints on other areas when we listen to rhythmic music.
| S-EPMC5368249 | biostudies-literature
Project description:Maturation of basal ganglia (BG) and frontoparietal circuitry parallels developmental gains in working memory (WM). Neurobiological models posit that adult WM performance is enhanced by communication between reward-sensitive BG and frontoparietal regions, via increased stability in the maintenance of goal-relevant neural patterns. It is not known whether this reward-driven pattern stability mechanism may have a role in WM development. In 34 young adolescents (12.16-14.72 years old) undergoing fMRI, reward-sensitive BG regions were localized using an incentive processing task. WM-sensitive regions were localized using a delayed-response WM task. Functional connectivity analyses were used to examine the stability of goal-relevant functional connectivity patterns during WM delay periods between and within reward-sensitive BG and WM-sensitive frontoparietal regions. Analyses revealed that more stable goal-relevant connectivity patterns between reward-sensitive BG and WM-sensitive frontoparietal regions were associated with both greater adolescent age and WM ability. Computational lesion models also revealed that functional connections to WM-sensitive frontoparietal regions from reward-sensitive BG uniquely increased the stability of goal-relevant functional connectivity patterns within frontoparietal regions. Findings suggested (1) the extent to which goal-relevant communication patterns within reward-frontoparietal circuitry are maintained increases with adolescent development and WM ability and (2) communication from reward-sensitive BG to frontoparietal regions enhances the maintenance of goal-relevant neural patterns in adolescents' WM. The maturation of reward-driven stability of goal-relevant neural patterns may provide a putative mechanism for understanding the developmental enhancement of WM.
| S-EPMC8500599 | biostudies-literature