Browse
Submit Data
Databases
API
Help

Dataset Information

0 Views

0 Connections

0 Citations

0 Reanalyses

0 Downloads

Omics score: 0

Subcortical Substrates of Explore-Exploit Decisions in Primates.

ABSTRACT: The explore-exploit dilemma refers to the challenge of deciding when to forego immediate rewards and explore new opportunities that could lead to greater rewards in the future. While motivational neural circuits facilitate learning based on past choices and outcomes, it is unclear whether they also support computations relevant for deciding when to explore. We recorded neural activity in the amygdala and ventral striatum of rhesus macaques as they solved a task that required them to balance novelty-driven exploration with exploitation of what they had already learned. Using a partially observable Markov decision process (POMDP) model to quantify explore-exploit trade-offs, we identified that the ventral striatum and amygdala differ in how they represent the immediate value of exploitative choices and the future value of exploratory choices. These findings show that subcortical motivational circuits are important in guiding explore-exploit decisions.

SUBMITTER: Costa VD

PROVIDER: S-EPMC6687547 | biostudies-literature | 2019 Aug

REPOSITORIES: biostudies-literature

ACCESS DATA

Json Xml

Publications

Subcortical Substrates of Explore-Exploit Decisions in Primates.

Costa Vincent D VD Mitz Andrew R AR Averbeck Bruno B BB

Neuron 20190610 3

The explore-exploit dilemma refers to the challenge of deciding when to forego immediate rewards and explore new opportunities that could lead to greater rewards in the future. While motivational neural circuits facilitate learning based on past choices and outcomes, it is unclear whether they also support computations relevant for deciding when to explore. We recorded neural activity in the amygdala and ventral striatum of rhesus macaques as they solved a task that required them to balance nove ...[more]

PMID: 31196672

Similar Datasets

The dynamics of explore-exploit decisions reveal a signal-to-noise mechanism for random exploration.

Project description:Growing evidence suggests that behavioral variability plays a critical role in how humans manage the tradeoff between exploration and exploitation. In these decisions a little variability can help us to overcome the desire to exploit known rewards by encouraging us to randomly explore something else. Here we investigate how such 'random exploration' could be controlled using a drift-diffusion model of the explore-exploit choice. In this model, variability is controlled by either the signal-to-noise ratio with which reward is encoded (the 'drift rate'), or the amount of information required before a decision is made (the 'threshold'). By fitting this model to behavior, we find that while, statistically, both drift and threshold change when people randomly explore, numerically, the change in drift rate has by far the largest effect. This suggests that random exploration is primarily driven by changes in the signal-to-noise ratio with which reward information is represented in the brain.

| S-EPMC7862437 | biostudies-literature

The neurocomputational bases of explore-exploit decision-making.

Project description:Flexible decision-making requires animals to forego immediate rewards (exploitation) and try novel choice options (exploration) to discover if they are preferable to familiar alternatives. Using the same task and a partially observable Markov decision process (POMDP) model to quantify the value of choices, we first determined that the computational basis for managing explore-exploit tradeoffs is conserved across monkeys and humans. We then used fMRI to identify where in the human brain the immediate value of exploitative choices and relative uncertainty about the value of exploratory choices were encoded. Consistent with prior neurophysiological evidence in monkeys, we observed divergent encoding of reward value and uncertainty in prefrontal and parietal regions, including frontopolar cortex, and parallel encoding of these computations in motivational regions including the amygdala, ventral striatum, and orbitofrontal cortex. These results clarify the interplay between prefrontal and motivational circuits that supports adaptive explore-exploit decisions in humans and nonhuman primates.

| S-EPMC9167768 | biostudies-literature

Disturbance modifies payoffs in the explore-exploit trade-off.

Project description:Decision-making agents face a fundamental trade-off between exploring new opportunities with risky outcomes versus exploiting familiar options with more certain but potentially suboptimal outcomes. Although mediation of this trade-off is essential to adaptive behavior and has for decades been assumed to modulate performance, the empirical consequences of human exploratory strategies are unknown beyond laboratory or theoretical settings. Leveraging 540,000 vessel position records from 2494 commercial fishing trips along with corresponding revenues, here we find that during undisturbed conditions, there was no relationship between exploration and performance, contrary to theoretical predictions. However, during a major disturbance event which closed the most-utilized fishing grounds, explorers benefited significantly from less-impacted revenues and were also more likely to continue fishing. We conclude that in stochastic natural systems characterized by non-stationary rewards, the role of exploration in buffering against disturbance may be greater than previously thought in humans.

| S-EPMC6662700 | biostudies-literature

Attention-deficit/hyperactivity disorder and the explore/exploit trade-off.

Project description:The ability to maximize rewards and minimize the costs of obtaining them is vital to making advantageous explore/exploit decisions. Exploratory decisions are theorized to be greater among individuals with attention-deficit/hyperactivity disorder (ADHD), potentially due to deficient catecholamine transmission. Here, we examined the effects of ADHD status and methylphenidate, a common ADHD medication, on explore/exploit decisions using a 6-armed bandit task. We hypothesized that ADHD participants would make more exploratory decisions than controls, and that MPH would reduce group differences. On separate study days, adults with (n = 26) and without (n = 23) ADHD completed the bandit task at baseline, and after methylphenidate or placebo in counter-balanced order. Explore/exploit decisions were modeled using reinforcement learning algorithms. ADHD participants made more exploratory decisions (i.e., chose options without the highest expected reward value) and earned fewer points than controls in all three study days, and methylphenidate did not affect these outcomes. Baseline exploratory choices were positively associated with hyperactive ADHD symptoms across all participants. These results support several theoretical models of increased exploratory choices in ADHD and suggest the unexplained variance in ADHD decisions may be due to less value tracking. The inability to suppress actions with little to no reward value may be a key feature of hyperactive ADHD symptoms.

| S-EPMC8027173 | biostudies-literature

Cortical substrates for exploratory decisions in humans.

Project description:Decision making in an uncertain environment poses a conflict between the opposing demands of gathering and exploiting information. In a classic illustration of this 'exploration-exploitation' dilemma, a gambler choosing between multiple slot machines balances the desire to select what seems, on the basis of accumulated experience, the richest option, against the desire to choose a less familiar option that might turn out more advantageous (and thereby provide information for improving future decisions). Far from representing idle curiosity, such exploration is often critical for organisms to discover how best to harvest resources such as food and water. In appetitive choice, substantial experimental evidence, underpinned by computational reinforcement learning (RL) theory, indicates that a dopaminergic, striatal and medial prefrontal network mediates learning to exploit. In contrast, although exploration has been well studied from both theoretical and ethological perspectives, its neural substrates are much less clear. Here we show, in a gambling task, that human subjects' choices can be characterized by a computationally well-regarded strategy for addressing the explore/exploit dilemma. Furthermore, using this characterization to classify decisions as exploratory or exploitative, we employ functional magnetic resonance imaging to show that the frontopolar cortex and intraparietal sulcus are preferentially active during exploratory decisions. In contrast, regions of striatum and ventromedial prefrontal cortex exhibit activity characteristic of an involvement in value-based exploitative decision making. The results suggest a model of action selection under uncertainty that involves switching between exploratory and exploitative behavioural modes, and provide a computationally precise characterization of the contribution of key decision-related brain systems to each of these functions.

| S-EPMC2635947 | biostudies-literature

Differential reinforcement encoding along the hippocampal long axis helps resolve the explore-exploit dilemma.

Project description:When making decisions, should one exploit known good options or explore potentially better alternatives? Exploration of spatially unstructured options depends on the neocortex, striatum, and amygdala. In natural environments, however, better options often cluster together, forming structured value distributions. The hippocampus binds reward information into allocentric cognitive maps to support navigation and foraging in such spaces. Here we report that human posterior hippocampus (PH) invigorates exploration while anterior hippocampus (AH) supports the transition to exploitation on a reinforcement learning task with a spatially structured reward function. These dynamics depend on differential reinforcement representations in the PH and AH. Whereas local reward prediction error signals are early and phasic in the PH tail, global value maximum signals are delayed and sustained in the AH body. AH compresses reinforcement information across episodes, updating the location and prominence of the value maximum and displaying goal cell-like ramping activity when navigating toward it.

| S-EPMC7589536 | biostudies-literature

Biases in the Explore-Exploit Tradeoff in Addictions: The Role of Avoidance of Uncertainty.

Project description:We focus on exploratory decisions across disorders of compulsivity, a potential dimensional construct for the classification of mental disorders. Behaviors associated with the pathological use of alcohol or food, in alcohol use disorders (AUD) or binge-eating disorder (BED), suggest a disturbance in explore-exploit decision-making, whereby strategic exploratory decisions in an attempt to improve long-term outcomes may diminish in favor of more repetitive or exploitatory choices. We compare exploration vs exploitation across disorders of natural (obesity with and without BED) and drug rewards (AUD). We separately acquired resting state functional MRI data using a novel multi-echo planar imaging sequence and independent components analysis from healthy individuals to assess the neural correlates underlying exploration. Participants with AUD showed reduced exploratory behavior across gain and loss environments, leading to lower-yielding exploitatory choices. Obese subjects with and without BED did not differ from healthy volunteers but when compared with each other or to AUD subjects, BED had enhanced exploratory behaviors particularly in the loss domain. All subject groups had decreased exploration or greater uncertainty avoidance to losses compared with rewards. More exploratory decisions in the context of reward were associated with frontal polar and ventral striatal connectivity. For losses, exploration was associated with frontal polar and precuneus connectivity. We further implicate the relevance and dimensionality of constructs of compulsivity across disorders of both natural and drug rewards.

| S-EPMC4650253 | biostudies-literature

Neural substrates of norm compliance in perceptual decisions.

Project description:Societal norms exert a powerful influence on our decisions. Behaviours motivated by norms, however, do not always concur with the responses mandated by decision relevant information potentially generating a conflict. To probe the interplay between normative and informational influences, we examined how prosocial norms impact on perceptual decisions subjects made in the context of a simultaneous presentation of social information. Participants displayed a bias in their perceptual decisions towards that mandated by social information. However, normative prescriptions modulated this bias bi-directionally depending on whether norms mandated a decision in accord or contrary to the contextual social information. At a neural level, the addition of a norms increased activity in prefrontal cortex and modulated functional connectivity between prefrontal and parietal areas. The bi-directional effect of our norms was captured by differential activations when participants decided against the social information. When norms indicated a decision in line with social information, non-compliance modulated lateral prefrontal cortex activity. By contrast, when norms mandated a decision against social information norm compliance increased activity in the anterior cingulate cortex. Hence, social norms changed the balance between a reliance on perceptual and social information by modulating brain activity in regions associated with response inhibition and conflict monitoring.

| S-EPMC5820261 | biostudies-literature

Imprecise action selection in substance use disorder: Evidence for active learning impairments when solving the explore-exploit dilemma.

Project description:BackgroundSubstance use disorders (SUDs) are a major public health risk. However, mechanisms accounting for continued patterns of poor choices in the face of negative life consequences remain poorly understood.MethodsWe use a computational (active inference) modeling approach, combined with multiple regression and hierarchical Bayesian group analyses, to examine how treatment-seeking individuals with one or more SUDs (alcohol, cannabis, sedatives, stimulants, hallucinogens, and/or opioids; N = 147) and healthy controls (HCs; N = 54) make choices to resolve uncertainty within a gambling task. A subset of SUDs (N = 49) and HCs (N = 51) propensity-matched on age, sex, and verbal IQ were also compared to replicate larger group findings.ResultsResults indicate that: (a) SUDs show poorer task performance than HCs (p = 0.03, Cohen's d = 0.33), with model estimates revealing less precise action selection mechanisms (p = 0.004, d = 0.43), a lower learning rate from losses (p = 0.02, d = 0.36), and a greater learning rate from gains (p = 0.04, d = 0.31); and (b) groups do not differ significantly in goal-directed information seeking.ConclusionsFindings suggest a pattern of inconsistent behavior in response to positive outcomes in SUDs combined with a tendency to attribute negative outcomes to chance. Specifically, individuals with SUDs fail to settle on a behavior strategy despite sufficient evidence of its success. These learning impairments could help account for difficulties in adjusting behavior and maintaining optimal decision-making during and after treatment.

| S-EPMC7502502 | biostudies-literature

Modulation of ventromedial orbitofrontal cortical glutamatergic activity affects the explore-exploit balance and influences value-based decision-making.

Project description:The balance between exploration and exploitation is essential for decision-making. The present study investigated the role of ventromedial orbitofrontal cortex (vmOFC) glutamate neurons in mediating value-based decision-making by first using optogenetics to manipulate vmOFC glutamate activity in rats during a probabilistic reversal learning (PRL) task. Rats that received vmOFC activation during informative feedback completed fewer reversals and exhibited reduced reward sensitivity relative to rats. Analysis with a Q-learning computational model revealed that increased vmOFC activity did not affect the learning rate but instead promoted maladaptive exploration. By contrast, vmOFC inhibition increased the number of completed reversals and increased exploitative behavior. In a separate group of animals, calcium activity of vmOFC glutamate neurons was recorded using fiber photometry. Complementing our results above, we found that suppression of vmOFC activity during the latter part of rewarded trials was associated with improved PRL performance, greater win-stay responding and selecting the correct choice on the next trial. These data demonstrate that excessive vmOFC activity during reward feedback disrupted value-based decision-making by increasing the maladaptive exploration of lower-valued options. Our findings support the premise that pharmacological interventions that normalize aberrant vmOFC glutamate activity during reward feedback processing may attenuate deficits in value-based decision-making.

| S-EPMC10183731 | biostudies-literature

OmicsDI is part of the ELIXIR infrastructure

OmicsDI is an Elixir interoperability service. Learn more ›

Tweets

OmicsDI Databases

PRIDE
PeptideAtlas
MassIVE
JPOST Repository
Physiome Model Repository

EGA
EVA
ENA
LINCS
PAXDB
Cell Collective

MetaboLights
Metabolomics Workbench
MetabolomeExpress
GNPS
BioModels
FAIRDOMHub

ArrayExpress
dbGaP
ExpressionAtlas
GEO
NODE

Information

Databases
Help
API
Contact us
Code on GitHub
Terms of use
Submit Data