The neurocomputational bases of explore-exploit decision-making.
Ontology highlight
ABSTRACT:
SUBMITTER: Hogeveen J
PROVIDER: S-EPMC9167768 | biostudies-literature | 2022 Jun
REPOSITORIES: biostudies-literature
Publications
The neurocomputational bases of explore-exploit decision-making.
Hogeveen Jeremy J Mullins Teagan S TS Romero John D JD Eversole Elizabeth E Rogge-Obando Kimberly K Mayer Andrew R AR Costa Vincent D VD
Neuron 20220406 11
Flexible decision-making requires animals to forego immediate rewards (exploitation) and try novel choice options (exploration) to discover if they are preferable to familiar alternatives. Using the same task and a partially observable Markov decision process (POMDP) model to quantify the value of choices, we first determined that the computational basis for managing explore-exploit tradeoffs is conserved across monkeys and humans. We then used fMRI to identify where in the human brain the immed ...[more]
Similar Datasets
Project description:Every decision that we make involves a conflict between exploiting our current knowledge of an action's value or exploring alternative courses of action that might lead to a better, or worse outcome. The sub-cortical nuclei that make up the basal ganglia have been proposed as a neural circuit that may contribute to resolving this explore-exploit 'dilemma'. To test this hypothesis, we examined the effects of neuromodulating the basal ganglia's output nucleus, the globus pallidus interna, in patients who had undergone deep brain stimulation (DBS) for isolated dystonia. Neuromodulation enhanced the number of exploratory choices to the lower value option in a 2-armed bandit probabilistic reversal-learning task. Enhanced exploration was explained by a reduction in the rate of evidence accumulation (drift rate) in a reinforcement learning drift diffusion model. We estimated the functional connectivity profile between the stimulating DBS electrode and the rest of the brain using a normative functional connectome derived from heathy controls. Variation in the extent of neuromodulation induced exploration between patients was associated with functional connectivity from the stimulation electrode site to a distributed brain functional network. We conclude that the basal ganglia's output nucleus, the globus pallidus interna, can adaptively modify decision choice when faced with the dilemma to explore or exploit.
| S-EPMC9940911 | biostudies-literature
Project description:In many everyday decisions, individuals choose between trialling something novel or something they know well. Deciding when to try a new option or stick with an option that is already known to you, known as the "explore/exploit" dilemma, is an important feature of cognition that characterises a range of decision-making contexts encountered by humans. Recent evidence has suggested preferences in explore/exploit biases are associated with psychopathology, although this has typically been examined within individual disorders. The current review examined whether explore/exploit decision-making represents a promising transdiagnostic target for psychosis, depression, and anxiety. A systematic search of academic databases was conducted, yielding a total of 29 studies. Studies examining psychosis were mostly consistent in showing that individuals with psychosis explored more compared with individuals without psychosis. The literature on anxiety and depression was more heterogenous; some studies found that anxiety and depression were associated with more exploration, whereas other studies demonstrated reduced exploration in anxiety and depression. However, examining a subset of studies that employed case-control methods, there was some evidence that both anxiety and depression also were associated with increased exploration. Due to the heterogeneity across the literature, we suggest that there is insufficient evidence to conclude whether explore/exploit decision-making is a transdiagnostic target for psychosis, depression, and anxiety. However, alongside our advisory groups of lived experience advisors, we suggest that this context of decision-making is a promising candidate that merits further investigation using well-powered, longitudinal designs. Such work also should examine whether biases in explore/exploit choices are amenable to intervention.
| S-EPMC11390819 | biostudies-literature
Project description:The balance between exploration and exploitation is essential for decision-making. The present study investigated the role of ventromedial orbitofrontal cortex (vmOFC) glutamate neurons in mediating value-based decision-making by first using optogenetics to manipulate vmOFC glutamate activity in rats during a probabilistic reversal learning (PRL) task. Rats that received vmOFC activation during informative feedback completed fewer reversals and exhibited reduced reward sensitivity relative to rats. Analysis with a Q-learning computational model revealed that increased vmOFC activity did not affect the learning rate but instead promoted maladaptive exploration. By contrast, vmOFC inhibition increased the number of completed reversals and increased exploitative behavior. In a separate group of animals, calcium activity of vmOFC glutamate neurons was recorded using fiber photometry. Complementing our results above, we found that suppression of vmOFC activity during the latter part of rewarded trials was associated with improved PRL performance, greater win-stay responding and selecting the correct choice on the next trial. These data demonstrate that excessive vmOFC activity during reward feedback disrupted value-based decision-making by increasing the maladaptive exploration of lower-valued options. Our findings support the premise that pharmacological interventions that normalize aberrant vmOFC glutamate activity during reward feedback processing may attenuate deficits in value-based decision-making.
| S-EPMC10183731 | biostudies-literature
Project description:We examined the neurocomputational mechanisms in which male adolescents make food and physical activity decisions and how those processes are influenced by body weight and physical activity levels. After physical activity and dietary assessments, thirty-eight males ages 14-18 completed the behavioral rating and fMRI decision tasks for food and physical activity items. The food and physical activity self-control decisions were significantly correlated with each other. In both, taste- or enjoyment-oriented processes were negatively associated with successful self-control decisions, while health-oriented processes were positively associated. The correlation between taste/enjoyment and healthy attribute ratings predicted actual laboratory food intake and physical activities (2-week activity monitoring). fMRI data showed the decision values of both food and activity are encoded in the ventromedial prefrontal cortex, suggesting both decisions share common reward value-related circuits at the time of choice. Compared to the group with overweight/obese, the group with normal weight showed stronger brain activations in the cognitive control, multisensory integration, and motor control regions during physical activity decisions. For both food and physical activity, self-controlled decisions utilize similar computational and neurobiological mechanisms, which may provide insights into how to promote healthy food and physical activity decisions.
| S-EPMC10105706 | biostudies-literature
Project description:The survival and well-being of humans require solving the patch-switching problem: we must decide when to stop collecting rewards in a current patch and travel somewhere else where gains may be higher. Previous studies suggested that frontal regions are underpinned by several processes in the context of foraging decisions such as tracking task difficulty, and/or the value of exploring the environment. To dissociate between these processes, participants completed an fMRI patch-switching learning task inspired by behavioral ecology. By analyzing >11,000 trials collected across 21 participants, we found that the activation in the cingulate cortex was closely related to several patch-switching-related variables including the decision to leave the current patch, the encounter of a new patch, the harvest value, and the relative forage value. Learning-induced changes in the patch-switching threshold were tracked by activity within frontoparietal regions including the superior frontal gyrus and angular gyrus. Our findings suggest that frontoparietal regions shape patch-switching learning apart from encoding classical non-learning foraging processes. These findings provide a novel neurobiological understanding of how learning emerges neurocomputationally shaping patch-switching behavior with implications in real-life choices such as job selection and pave the way for future studies to probe the causal role of these neurobiological mechanisms.
| S-EPMC10267616 | biostudies-literature
Project description:In this article, we describe a province-wide collaborative project in which we adapted the Explore Transplant (ET) education program for use in Ontario, Canada, to develop Explore Transplant Ontario (ETO). Kidney transplantation (KT), especially living donor kidney transplantation (LDKT), is the best treatment for many patients with end-stage kidney disease (ESKD), with the best patient survival and quality of life and also reduced health care costs. Yet KT and LDKT are underutilized both internationally and in Canada. Research has demonstrated that patients with ESKD who receive personalized transplant education are more likely to complete the transplant evaluation process and to receive LDKT compared with patients who do not receive this education.Research expertise of the lead authors and Medline search of studies assessing the impact of education interventions on access to KT and LDKT.The ET program, developed by Dr Amy Waterman, has been used in thousands of patients with ESKD in the United States to enhance KT and LDKT knowledge. To adapt this program for use in Ontario, we convened a working group, including patient representatives, nephrologists, transplant coordinators, dialysis nurses, and patient educators from all Ontario KT centers and selected dialysis units. In an iterative process concluding in a consensus workshop, the working group reviewed and edited the text of the original ET program and suggested changes to the videos.The adapted program reflects the Ontario health care environment and responds to the specific needs of patients with chronic kidney disease (CKD) in the province. The videos feature Ontario transplant nephrologists, transplant coordinators, and patients, representative of the ethnic diversity in Ontario, sharing their transplant experience and expertise. Despite the changes, ETO is consistent with the quality and style of the original ET program. At the end of this article, we summarize subsequent steps to test and utilize ETO. Those projects, specifically the ETO pilot study and a multicomponent quality improvement initiative to increase utilization of KT and LDKT across Ontario, will be described in full in future papers.This article describes a provincial initiative; therefore, our findings may not be fully generalizable without further considerations. The adapted education program has not yet been tested in large trial for effectiveness.As a program grounded in the theoretical model of behavior change, ETO places patients with ESKD at the center of a complex process of navigating renal replacement therapy modalities and acknowledges a broad range of patient values, priorities, and states of readiness to pursue KT.
| S-EPMC6058418 | biostudies-other
Project description:Humans adjust their behavioral strategies based on feedback, a process that may depend on intrinsic preferences and contextual factors such as visual salience. In this study, we hypothesized that decision-making based on visual salience is influenced by habitual and goal-directed processes, which can be evidenced by changes in attention and subjective valuation systems. To test this hypothesis, we conducted a series of studies to investigate the behavioral and neural mechanisms underlying visual salience-driven decision-making. We first established the baseline behavioral strategy without salience in Experiment 1 (n = 21). We then highlighted the utility or performance dimension of the chosen outcome using colors in Experiment 2 (n = 30). We demonstrated that the difference in staying frequency increased along the salient dimension, confirming a salience effect. Furthermore, the salience effect was abolished when directional information was removed in Experiment 3 (n = 28), suggesting that the salience effect is feedback-specific. To generalize our findings, we replicated the feedback-specific salience effects using eye-tracking and text emphasis. The fixation differences between the chosen and unchosen values were enhanced along the feedback-specific salient dimension in Experiment 4 (n = 48) but unchanged after removing feedback-specific information in Experiment 5 (n = 32). Moreover, the staying frequency was correlated with fixation properties, confirming that salience guides attention deployment. Lastly, our neuroimaging study (Experiment 6, n = 25) showed that the striatum subregions encoded salience-based outcome evaluation, while the vmPFC encoded salience-based behavioral adjustments. The connectivity of the vmPFC-ventral striatum accounted for individual differences in utility-driven, whereas the vmPFC-dmPFC for performance-driven behavioral adjustments. Together, our results provide a neurocognitive account of how task-irrelevant visual salience drives decision-making by involving attention and the frontal-striatal valuation systems. PUBLIC SIGNIFICANCE STATEMENT: Humans may use the current outcome to make behavior adjustments. How this occurs may depend on stable individual preferences and contextual factors, such as visual salience. Under the hypothesis that visual salience determines attention and subsequently modulates subjective valuation, we investigated the underlying behavioral and neural bases of visual-context-guided outcome evaluation and behavioral adjustments. Our findings suggest that the reward system is orchestrated by visual context and highlight the critical role of attention and the frontal-striatal neural circuit in visual-context-guided decision-making that may involve habitual and goal-directed processes.
| S-EPMC10868706 | biostudies-literature
Project description:Autism spectrum disorder (ASD) and obsessive-compulsive disorder (OCD) often share phenotypes of repetitive behaviors, possibly underpinned by abnormal decision-making. To compare neural correlates underlying decision-making between these disorders, brain activation of boys with ASD (N = 24), OCD (N = 20) and typically developing controls (N = 20) during gambling was compared, and computational modeling compared performance. Patients were unimpaired on number of risky decisions, but modeling showed that both patient groups had lower choice consistency and relied less on reinforcement learning compared to controls. ASD individuals had disorder-specific choice perseverance abnormalities compared to OCD individuals. Neurofunctionally, ASD and OCD boys shared dorsolateral/inferior frontal underactivation compared to controls during decision-making. During outcome anticipation, patients shared underactivation compared to controls in lateral inferior/orbitofrontal cortex and ventral striatum. During reward receipt, ASD boys had disorder-specific enhanced activation in inferior frontal/insular regions relative to OCD boys and controls. Results showed that ASD and OCD individuals shared decision-making strategies that differed from controls to achieve comparable performance to controls. Patients showed shared abnormalities in lateral-(orbito)fronto-striatal reward circuitry, but ASD boys had disorder-specific lateral inferior frontal/insular overactivation, suggesting that shared and disorder-specific mechanisms underpin decision-making in these disorders. Findings provide evidence for shared neurobiological substrates that could serve as possible future biomarkers.
| S-EPMC6919268 | biostudies-literature
Project description:The explore-exploit dilemma refers to the challenge of deciding when to forego immediate rewards and explore new opportunities that could lead to greater rewards in the future. While motivational neural circuits facilitate learning based on past choices and outcomes, it is unclear whether they also support computations relevant for deciding when to explore. We recorded neural activity in the amygdala and ventral striatum of rhesus macaques as they solved a task that required them to balance novelty-driven exploration with exploitation of what they had already learned. Using a partially observable Markov decision process (POMDP) model to quantify explore-exploit trade-offs, we identified that the ventral striatum and amygdala differ in how they represent the immediate value of exploitative choices and the future value of exploratory choices. These findings show that subcortical motivational circuits are important in guiding explore-exploit decisions.
| S-EPMC6687547 | biostudies-literature
Project description:Decision-making agents face a fundamental trade-off between exploring new opportunities with risky outcomes versus exploiting familiar options with more certain but potentially suboptimal outcomes. Although mediation of this trade-off is essential to adaptive behavior and has for decades been assumed to modulate performance, the empirical consequences of human exploratory strategies are unknown beyond laboratory or theoretical settings. Leveraging 540,000 vessel position records from 2494 commercial fishing trips along with corresponding revenues, here we find that during undisturbed conditions, there was no relationship between exploration and performance, contrary to theoretical predictions. However, during a major disturbance event which closed the most-utilized fishing grounds, explorers benefited significantly from less-impacted revenues and were also more likely to continue fishing. We conclude that in stochastic natural systems characterized by non-stationary rewards, the role of exploration in buffering against disturbance may be greater than previously thought in humans.
| S-EPMC6662700 | biostudies-literature