Dataset Information

Alterations in choice behavior by manipulations of world model.

ABSTRACT: How to compute initially unknown reward values makes up one of the key problems in reinforcement learning theory, with two basic approaches being used. Model-free algorithms rely on the accumulation of substantial amounts of experience to compute the value of actions, whereas in model-based learning, the agent seeks to learn the generative process for outcomes from which the value of actions can be predicted. Here we show that (i) "probability matching"-a consistent example of suboptimal choice behavior seen in humans-occurs in an optimal Bayesian model-based learner using a max decision rule that is initialized with ecologically plausible, but incorrect beliefs about the generative process for outcomes and (ii) human behavior can be strongly and predictably altered by the presence of cues suggestive of various generative processes, despite statistically identical outcome generation. These results suggest human decision making is rational and model based and not consistent with model-free learning.

SUBMITTER: Green CS

PROVIDER: S-EPMC2941269 | biostudies-literature | 2010 Sep

REPOSITORIES: biostudies-literature

ACCESS DATA

Publications

Alterations in choice behavior by manipulations of world model.

Green C S CS Benson C C Kersten D D Schrater P P

Proceedings of the National Academy of Sciences of the United States of America 20100830 37

How to compute initially unknown reward values makes up one of the key problems in reinforcement learning theory, with two basic approaches being used. Model-free algorithms rely on the accumulation of substantial amounts of experience to compute the value of actions, whereas in model-based learning, the agent seeks to learn the generative process for outcomes from which the value of actions can be predicted. Here we show that (i) "probability matching"-a consistent example of suboptimal choice ...[more]

PMID: 20805507

Similar Datasets

Project description:According to a prominent view of sensorimotor processing in primates, selection and specification of possible actions are not sequential operations. Rather, a decision for an action emerges from competition between different movement plans, which are specified and selected in parallel. For action choices which are based on ambiguous sensory input, the frontoparietal sensorimotor areas are considered part of the common underlying neural substrate for selection and specification of action. These areas have been shown capable of encoding alternative spatial motor goals in parallel during movement planning, and show signatures of competitive value-based selection among these goals. Since the same network is also involved in learning sensorimotor associations, competitive action selection (decision making) should not only be driven by the sensory evidence and expected reward in favor of either action, but also by the subject's learning history of different sensorimotor associations. Previous computational models of competitive neural decision making used predefined associations between sensory input and corresponding motor output. Such hard-wiring does not allow modeling of how decisions are influenced by sensorimotor learning or by changing reward contingencies. We present a dynamic neural field model which learns arbitrary sensorimotor associations with a reward-driven Hebbian learning algorithm. We show that the model accurately simulates the dynamics of action selection with different reward contingencies, as observed in monkey cortical recordings, and that it correctly predicted the pattern of choice errors in a control experiment. With our adaptive model we demonstrate how network plasticity, which is required for association learning and adaptation to new reward contingencies, can influence choice behavior. The field model provides an integrated and dynamic account for the operations of sensorimotor integration, working memory and action selection required for decision making in ambiguous choice situations.

Project description:The amygdala is an important neural substrate for the emotional-affective dimension of pain and modulation of pain. The central nucleus (CeA) serves major amygdala output functions and receives nociceptive and affected-related information from the spino-parabrachial and lateral-basolateral amygdala (LA-BLA) networks. The CeA is a major site of extra-hypothalamic expression of corticotropin releasing factor (CRF, also known as corticotropin releasing hormone, CRH), and amygdala CRF neurons form widespread projections to target regions involved in behavioral and descending pain modulation. Here we explored the effects of modulating amygdala neurons on nociceptive processing in the spinal cord and on pain-like behaviors, using optogenetic activation or silencing of BLA to CeA projections and CeA-CRF neurons under normal conditions and in an acute pain model. Extracellular single unit recordings were made from spinal dorsal horn wide dynamic range (WDR) neurons, which respond more strongly to noxious than innocuous mechanical stimuli, in normal and arthritic adult rats (5-6 h postinduction of a kaolin/carrageenan-monoarthritis in the left knee). For optogenetic activation or silencing of CRF neurons, a Cre-inducible viral vector (DIO-AAV) encoding channelrhodopsin 2 (ChR2) or enhanced Natronomonas pharaonis halorhodopsin (eNpHR3.0) was injected stereotaxically into the right CeA of transgenic Crh-Cre rats. For optogenetic activation or silencing of BLA axon terminals in the CeA, a viral vector (AAV) encoding ChR2 or eNpHR3.0 under the control of the CaMKII promoter was injected stereotaxically into the right BLA of Sprague-Dawley rats. For wireless optical stimulation of ChR2 or eNpHR3.0 expressing CeA-CRF neurons or BLA-CeA axon terminals, an LED optic fiber was stereotaxically implanted into the right CeA. Optical activation of CeA-CRF neurons or of BLA axon terminals in the CeA increased the evoked responses of spinal WDR neurons and induced pain-like behaviors (hypersensitivity and vocalizations) under normal condition. Conversely, optical silencing of CeA-CRF neurons or of BLA axon terminals in the CeA decreased the evoked responses of spinal WDR neurons and vocalizations, but not hypersensitivity, in the arthritis pain model. These findings suggest that the amygdala can drive the activity of spinal cord neurons and pain-like behaviors under normal conditions and in a pain model.

Project description:Although gene-environment interactions are known to significantly influence psychopathology-related disease states, only few animal models cover both the genetic background and environmental manipulations. Therefore, we have taken advantage of the bidirectionally inbred high (HAB) and low (LAB) anxiety-related behavior mouse lines to generate HAB × LAB F1 hybrids that intrinsically carry both lines' genetic characteristics, and subsequently raised them in three different environments-standard, enriched (EE) and chronic mild stress (CMS). Assessing genetic correlates of trait anxiety, we focused on two genes already known to play a role in HAB vs. LAB mice, corticotropin releasing hormone receptor type 1 (Crhr1) and high mobility group nucleosomal binding domain 3 (Hmgn3). While EE F1 mice showed decreased anxiety-related and increased explorative behaviors compared to controls, CMS sparked effects in the opposite direction. However, environmental treatments affected the expression of the two genes in distinct ways. Thus, while expression ratios of Hmgn3 between the HAB- and LAB-specific alleles remained equal, total expression resembled the one observed in HAB vs. LAB mice, i.e., decreased after EE and increased after CMS treatment. On the other hand, while total expression of Crhr1 remained unchanged between the groups, the relative expression of HAB- and LAB-specific alleles showed a clear effect following the environmental modifications. Thus, the environmentally driven bidirectional shift of trait anxiety in this F1 model strongly correlated with Hmgn3 expression, irrespective of allele-specific expression patterns that retained the proportions of basic differential HAB vs. LAB expression, making this gene a match for environment-induced modifications. An involvement of Crhr1 in the bidirectional behavioral shift could, however, rather be due to different effects of the HAB- and LAB-specific alleles described here. Both candidate genes therefore deserve attention in the complex regulation of anxiety-related phenotypes including environment-mediated effects.

Dataset Information

Alterations in choice behavior by manipulations of world model.

Publications

Alterations in choice behavior by manipulations of world model.

Similar Datasets

OmicsDI is part of the ELIXIR infrastructure

Tweets