Intrinsically motivated oculomotor exploration guided by uncertainty reduction and conditioned reinforcement in non-human primates.
Ontology highlight
ABSTRACT:
SUBMITTER: Daddaoua N
PROVIDER: S-EPMC4738323 | biostudies-other | 2016 Feb
REPOSITORIES: biostudies-other
Similar Datasets
Project description:Finding algorithms that allow agents to discover a wide variety of skills efficiently and autonomously, remains a challenge of Artificial Intelligence. Intrinsically Motivated Goal Exploration Processes (IMGEPs) have been shown to enable real world robots to learn repertoires of policies producing a wide range of diverse effects. They work by enabling agents to autonomously sample goals that they then try to achieve. In practice, this strategy leads to an efficient exploration of complex environments with high-dimensional continuous actions. Until recently, it was necessary to provide the agents with an engineered goal space containing relevant features of the environment. In this article we show that the goal space can be learned using deep representation learning algorithms, effectively reducing the burden of designing goal spaces. Our results pave the way to autonomous learning agents that are able to autonomously build a representation of the world and use this representation to explore the world efficiently. We present experiments in two environments using population-based IMGEPs. The first experiments are performed on a simple, yet challenging, simulated environment. Then, another set of experiments tests the applicability of those principles on a real-world robotic setup, where a 6-joint robotic arm learns to manipulate a ball inside an arena, by choosing goals in a space learned from its past experience.
| S-EPMC7835425 | biostudies-literature
Project description:Collective motion is found in various animal systems, active suspensions, and robotic or virtual agents. This is often understood by using high-level models that directly encode selected empirical features, such as coalignment and cohesion. Can these features be shown to emerge from an underlying, low-level principle? We find that they emerge naturally under future state maximization (FSM). Here, agents perceive a visual representation of the world around them, such as might be recorded on a simple retina, and then move to maximize the number of different visual environments that they expect to be able to access in the future. Such a control principle may confer evolutionary fitness in an uncertain world by enabling agents to deal with a wide variety of future scenarios. The collective dynamics that spontaneously emerge under FSM resemble animal systems in several qualitative aspects, including cohesion, coalignment, and collision suppression, none of which are explicitly encoded in the model. A multilayered neural network trained on simulated trajectories is shown to represent a heuristic mimicking FSM. Similar levels of reasoning would seem to be accessible under animal cognition, demonstrating a possible route to the emergence of collective motion in social animals directly from the control principle underlying FSM. Such models may also be good candidates for encoding into possible future realizations of artificial "intelligent" matter, able to sense light, process information, and move.
| S-EPMC6681759 | biostudies-literature
Project description:Over the last 20 years, a significant part of the research in exploratory robotics partially switches from looking for the most efficient way of exploring an unknown environment to finding what could motivate a robot to autonomously explore it. Moreover, a growing literature focuses not only on the topological description of a space (dimensions, obstacles, usable paths, etc.) but rather on more semantic components, such as multimodal objects present in it. In the search of designing robots that behave autonomously by embedding life-long learning abilities, the inclusion of mechanisms of attention is of importance. Indeed, be it endogenous or exogenous, attention constitutes a form of intrinsic motivation for it can trigger motor command toward specific stimuli, thus leading to an exploration of the space. The Head Turning Modulation model presented in this paper is composed of two modules providing a robot with two different forms of intrinsic motivations leading to triggering head movements toward audiovisual sources appearing in unknown environments. First, the Dynamic Weighting module implements a motivation by the concept of Congruence, a concept defined as an adaptive form of semantic saliency specific for each explored environment. Then, the Multimodal Fusion and Inference module implements a motivation by the reduction of Uncertainty through a self-supervised online learning algorithm that can autonomously determine local consistencies. One of the novelty of the proposed model is to solely rely on semantic inputs (namely audio and visual labels the sources belong to), in opposition to the traditional analysis of the low-level characteristics of the perceived data. Another contribution is found in the way the exploration is exploited to actively learn the relationship between the visual and auditory modalities. Importantly, the robot-endowed with binocular vision, binaural audition and a rotating head-does not have access to prior information about the different environments it will explore. Consequently, it will have to learn in real-time what audiovisual objects are of "importance" in order to rotate its head toward them. Results presented in this paper have been obtained in simulated environments as well as with a real robot in realistic experimental conditions.
| S-EPMC6160585 | biostudies-other
Project description:The interconnected nuclei of the ventral basal ganglia have long been identified as key regulators of motivated behavior, and dysfunction of this circuit is strongly implicated in mood and substance use disorders. The ventral pallidum (VP) is a central node of the ventral basal ganglia, and recent studies have revealed complex VP cellular heterogeneity and cell- and circuit-specific regulation of reward, aversion, motivation, and drug-seeking behaviors. Although the VP is canonically considered a relay and output structure for this circuit, emerging data indicate that the VP is a central hub in an extensive network for reward processing and the regulation of motivation that extends beyond classically defined basal ganglia borders. VP neurons respond temporally faster and show more advanced reward coding and prediction error processing than neurons in the upstream nucleus accumbens, and regulate the activity of the ventral mesencephalon dopamine system. This review will summarize recent findings in the literature and provide an update on the complex cellular heterogeneity and cell- and circuit-specific regulation of motivated behaviors and reinforcement by the VP with a specific focus on mood and substance use disorders. In addition, we will discuss mechanisms by which stress and drug exposure alter the functioning of the VP and produce susceptibility to neuropsychiatric disorders. Lastly, we will outline unanswered questions and identify future directions for studies necessary to further clarify the central role of VP neurons in the regulation of motivated behaviors. Significance: Research in the last decade has revealed a complex cell- and circuit-specific role for the VP in reward processing and the regulation of motivated behaviors. Novel insights obtained using cell- and circuit-specific interrogation strategies have led to a major shift in our understanding of this region. Here, we provide a comprehensive review of the VP in which we integrate novel findings with the existing literature and highlight the emerging role of the VP as a linchpin of the neural systems that regulate motivation, reward, and aversion. In addition, we discuss the dysfunction of the VP in animal models of neuropsychiatric disorders.
| S-EPMC9932340 | biostudies-literature
Project description:BackgroundNegative symptoms are core features of schizophrenia (SZ); however, the cognitive and neural basis for individual negative symptom domains remains unclear. Converging evidence suggests a role for striatal and prefrontal dopamine in reward learning and the exploration of actions that might produce outcomes that are better than the status quo. The current study examines whether deficits in reinforcement learning and uncertainty-driven exploration predict specific negative symptom domains.MethodsWe administered a temporal decision-making task, which required trial-by-trial adjustment of reaction time to maximize reward receipt, to 51 patients with SZ and 39 age-matched healthy control subjects. Task conditions were designed such that expected value (probability × magnitude) increased, decreased, or remained constant with increasing response times. Computational analyses were applied to estimate the degree to which trial-by-trial responses are influenced by reinforcement history.ResultsIndividuals with SZ showed impaired Go learning but intact NoGo learning relative to control subjects. These effects were most pronounced in patients with higher levels of negative symptoms. Uncertainty-based exploration was substantially reduced in individuals with SZ and selectively correlated with clinical ratings of anhedonia.ConclusionsSchizophrenia patients, particularly those with high negative symptoms, failed to speed reaction times to increase positive outcomes and showed reduced tendency to explore when alternative actions could lead to better outcomes than the status quo. Results are interpreted in the context of current computational, genetic, and pharmacological data supporting the roles of striatal and prefrontal dopamine in these processes.
| S-EPMC3039035 | biostudies-literature
Project description: Not available
| S-EPMC10642204 | biostudies-literature
Project description:Avoiding toxins in food is as important as obtaining nutrition. Conditioned food aversions have been studied in animals as diverse as nematodes and humans [1, 2], but the neural signaling mechanisms underlying this form of learning have been difficult to pinpoint. Honeybees quickly learn to associate floral cues with food [3], a trait that makes them an excellent model organism for studying the neural mechanisms of learning and memory. Here we show that honeybees not only detect toxins but can also learn to associate odors with both the taste of toxins and the postingestive consequences of consuming them. We found that two distinct monoaminergic pathways mediate learned food aversions in the honeybee. As for other insect species conditioned with salt or electric shock reinforcers [4-7], learned avoidances of odors paired with bad-tasting toxins are mediated by dopamine. Our experiments are the first to identify a second, postingestive pathway for learned olfactory aversions that involves serotonin. This second pathway may represent an ancient mechanism for food aversion learning conserved across animal lineages.
| S-EPMC3011020 | biostudies-literature
Project description:Several personality factors have been implicated in vulnerability to addiction by impacting learning and decision making. One such factor is intolerance of uncertainty (IU), the tendency to perceive uncertain situations negatively and avoid them. Conditioned place preference (CPP), which compares preference for contexts paired with reward, has been used to examine the motivation for both drug and non-drug rewards. However, preference for locations associated with non-drug reward, as well as the potential influence of IU, has not been thoroughly studied in individuals with addiction. In the current study, we examined CPP using a computer-based task in a sample of addicted individuals undergoing opioid maintenance treatment and never-addicted controls. Patients were confirmed to have higher IU than controls. In the CPP task, the two groups did not differ in overall time spent in the previously-rewarded context. However, controls were more likely than patients to immediately return to this context. Contrary to our predictions, IU was not a significant predictor of preference for the previously-rewarded context, although higher IU in controls was associated with a higher number of rewards obtained in the task. No such relationship was found in patients.
| S-EPMC5967368 | biostudies-literature
Project description:Gambling disorder (GD) is a behavioral addiction associated with impairments in value-based decision-making and behavioral flexibility and might be linked to changes in the dopamine system. Maximizing long-term rewards requires a flexible trade-off between the exploitation of known options and the exploration of novel options for information gain. This exploration-exploitation trade-off is thought to depend on dopamine neurotransmission. We hypothesized that human gamblers would show a reduction in directed (uncertainty-based) exploration, accompanied by changes in brain activity in a fronto-parietal exploration-related network. Twenty-three frequent, non-treatment seeking gamblers and twenty-three healthy matched controls (all male) performed a four-armed bandit task during functional magnetic resonance imaging (fMRI). Computational modeling using hierarchical Bayesian parameter estimation revealed signatures of directed exploration, random exploration, and perseveration in both groups. Gamblers showed a reduction in directed exploration, whereas random exploration and perseveration were similar between groups. Neuroimaging revealed no evidence for group differences in neural representations of basic task variables (expected value, prediction errors). Our hypothesis of reduced frontal pole (FP) recruitment in gamblers was not supported. Exploratory analyses showed that during directed exploration, gamblers showed reduced parietal cortex and substantia-nigra/ventral-tegmental-area activity. Cross-validated classification analyses revealed that connectivity in an exploration-related network was predictive of group status, suggesting that connectivity patterns might be more predictive of problem gambling than univariate effects. Findings reveal specific reductions of strategic exploration in gamblers that might be linked to altered processing in a fronto-parietal network and/or changes in dopamine neurotransmission implicated in GD.SIGNIFICANCE STATEMENT Wiehler et al. (2021) report that gamblers rely less on the strategic exploration of unknown, but potentially better rewards during reward learning. This is reflected in a related network of brain activity. Parameters of this network can be used to predict the presence of problem gambling behavior in participants.
| S-EPMC7984586 | biostudies-literature
Project description:Across the lifespan, individuals frequently choose between exploiting known rewarding options or exploring unknown alternatives. A large body of work has suggested that children may explore more than adults. However, because novelty and reward uncertainty are often correlated, it is unclear how they differentially influence decision-making across development. Here, children, adolescents, and adults (ages 8-27 years, N = 122) completed an adapted version of a recently developed value-guided decision-making task that decouples novelty and uncertainty. In line with prior studies, we found that exploration decreased with increasing age. Critically, participants of all ages demonstrated a similar bias to select choice options with greater novelty, whereas aversion to reward uncertainty increased into adulthood. Computational modeling of participant choices revealed that whereas adolescents and adults demonstrated attenuated uncertainty aversion for more novel choice options, children's choices were not influenced by reward uncertainty.
| S-EPMC10431916 | biostudies-literature