Browse
Submit Data
Databases
API
Help

Dataset Information

28 Views

0 Connections

0 Citations

0 Reanalyses

0 Downloads

Omics score: 0

Does temporal discounting explain unhealthy behavior? A systematic review and reinforcement learning perspective.

ABSTRACT: The tendency to make unhealthy choices is hypothesized to be related to an individual's temporal discount rate, the theoretical rate at which they devalue delayed rewards. Furthermore, a particular form of temporal discounting, hyperbolic discounting, has been proposed to explain why unhealthy behavior can occur despite healthy intentions. We examine these two hypotheses in turn. We first systematically review studies which investigate whether discount rates can predict unhealthy behavior. These studies reveal that high discount rates for money (and in some instances food or drug rewards) are associated with several unhealthy behaviors and markers of health status, establishing discounting as a promising predictive measure. We secondly examine whether intention-incongruent unhealthy actions are consistent with hyperbolic discounting. We conclude that intention-incongruent actions are often triggered by environmental cues or changes in motivational state, whose effects are not parameterized by hyperbolic discounting. We propose a framework for understanding these state-based effects in terms of the interplay of two distinct reinforcement learning mechanisms: a "model-based" (or goal-directed) system and a "model-free" (or habitual) system. Under this framework, while discounting of delayed health may contribute to the initiation of unhealthy behavior, with repetition, many unhealthy behaviors become habitual; if health goals then change, habitual behavior can still arise in response to environmental cues. We propose that the burgeoning development of computational models of these processes will permit further identification of health decision-making phenotypes.

SUBMITTER: Story GW

PROVIDER: S-EPMC3950931 | biostudies-literature |

REPOSITORIES: biostudies-literature

ACCESS DATA

Json Xml

Similar Datasets

Unhealthy diets, obesity and time discounting: a systematic literature review and network analysis.

Project description:There is an increasing policy commitment to address the avoidable burdens of unhealthy diet, overweight and obesity. However, to design effective policies, it is important to understand why people make unhealthy dietary choices. Research from behavioural economics suggests a critical role for time discounting, which describes how people's value of a reward, such as better health, decreases with delay to its receipt. We systematically reviewed the literature on the relationship of time discounting with unhealthy diets, overweight and obesity in Web of Science and PubMed. We identified 41 studies that met our inclusion criteria as they examined the association between time discount rates and (i) unhealthy food consumption; (ii) overweight and (iii) response to dietary and weight loss interventions. Nineteen out of 25 cross-sectional studies found time discount rates positively associated with overweight, obesity and unhealthy diets. Experimental studies indicated that lower time discounting was associated with greater weight loss. Findings varied by how time discount rates were measured; stronger results were observed for food than monetary-based measurements. Network co-citation analysis revealed a concentration of research in nutrition journals. Overall, there is moderate evidence that high time discounting is a significant risk factor for unhealthy diets, overweight and obesity and may serve as an important target for intervention. © 2016 The Authors Obesity Reviews published by John Wiley & Sons Ltd on behalf of International Association for the Study of Obesity (IASO).

| S-EPMC4988386 | biostudies-literature

Hierarchically organized behavior and its neural foundations: a reinforcement learning perspective.

Project description:Research on human and animal behavior has long emphasized its hierarchical structure-the divisibility of ongoing behavior into discrete tasks, which are comprised of subtask sequences, which in turn are built of simple actions. The hierarchical structure of behavior has also been of enduring interest within neuroscience, where it has been widely considered to reflect prefrontal cortical functions. In this paper, we reexamine behavioral hierarchy and its neural substrates from the point of view of recent developments in computational reinforcement learning. Specifically, we consider a set of approaches known collectively as hierarchical reinforcement learning, which extend the reinforcement learning paradigm by allowing the learning agent to aggregate actions into reusable subroutines or skills. A close look at the components of hierarchical reinforcement learning suggests how they might map onto neural structures, in particular regions within the dorsolateral and orbital prefrontal cortex. It also suggests specific ways in which hierarchical reinforcement learning might provide a complement to existing psychological models of hierarchically structured behavior. A particularly important question that hierarchical reinforcement learning brings to the fore is that of how learning identifies new action routines that are likely to provide useful building blocks in solving a wide range of future problems. Here and at many other points, hierarchical reinforcement learning offers an appealing framework for investigating the computational and neural underpinnings of hierarchically structured behavior.

| S-EPMC2783353 | biostudies-literature

Perspective Taking in Deep Reinforcement Learning Agents.

Project description:Perspective taking is the ability to take into account what the other agent knows. This skill is not unique to humans as it is also displayed by other animals like chimpanzees. It is an essential ability for social interactions, including efficient cooperation, competition, and communication. Here we present our progress toward building artificial agents with such abilities. We implemented a perspective taking task inspired by experiments done with chimpanzees. We show that agents controlled by artificial neural networks can learn via reinforcement learning to pass simple tests that require some aspects of perspective taking capabilities. We studied whether this ability is more readily learned by agents with information encoded in allocentric or egocentric form for both their visual perception and motor actions. We believe that, in the long run, building artificial agents with perspective taking ability can help us develop artificial intelligence that is more human-like and easier to communicate with.

| S-EPMC7390927 | biostudies-literature

Temporal-difference reinforcement learning with distributed representations.

Project description:Temporal-difference (TD) algorithms have been proposed as models of reinforcement learning (RL). We examine two issues of distributed representation in these TD algorithms: distributed representations of belief and distributed discounting factors. Distributed representation of belief allows the believed state of the world to distribute across sets of equivalent states. Distributed exponential discounting factors produce hyperbolic discounting in the behavior of the agent itself. We examine these issues in the context of a TD RL model in which state-belief is distributed over a set of exponentially-discounting "micro-Agents", each of which has a separate discounting factor (gamma). Each microAgent maintains an independent hypothesis about the state of the world, and a separate value-estimate of taking actions within that hypothesized state. The overall agent thus instantiates a flexible representation of an evolving world-state. As with other TD models, the value-error (delta) signal within the model matches dopamine signals recorded from animals in standard conditioning reward-paradigms. The distributed representation of belief provides an explanation for the decrease in dopamine at the conditioned stimulus seen in overtrained animals, for the differences between trace and delay conditioning, and for transient bursts of dopamine seen at movement initiation. Because each microAgent also includes its own exponential discounting factor, the overall agent shows hyperbolic discounting, consistent with behavioral experiments.

| S-EPMC2760757 | biostudies-literature

Temporal encoding in deep reinforcement learning agents.

Project description:Neuroscientists have observed both cells in the brain that fire at specific points in time, known as "time cells", and cells whose activity steadily increases or decreases over time, known as "ramping cells". It is speculated that time and ramping cells support temporal computations in the brain and carry mnemonic information. However, due to the limitations in animal experiments, it is difficult to determine how these cells really contribute to behavior. Here, we show that time cells and ramping cells naturally emerge in the recurrent neural networks of deep reinforcement learning models performing simulated interval timing and working memory tasks, which have learned to estimate expected rewards in the future. We show that these cells do indeed carry information about time and items stored in working memory, but they contribute to behavior in large part by providing a dynamic representation on which policy can be computed. Moreover, the information that they do carry depends on both the task demands and the variables provided to the models. Our results suggest that time cells and ramping cells could contribute to temporal and mnemonic calculations, but the way in which they do so may be complex and unintuitive to human observers.

| S-EPMC10724179 | biostudies-literature

Transfer of Temporal Logic Formulas in Reinforcement Learning.

Project description:Transferring high-level knowledge from a source task to a target task is an effective way to expedite reinforcement learning (RL). For example, propositional logic and first-order logic have been used as representations of such knowledge. We study the transfer of knowledge between tasks in which the timing of the events matters. We call such tasks temporal tasks. We concretize similarity between temporal tasks through a notion of logical transferability, and develop a transfer learning approach between different yet similar temporal tasks. We first propose an inference technique to extract metric interval temporal logic (MITL) formulas in sequential disjunctive normal form from labeled trajectories collected in RL of the two tasks. If logical transferability is identified through this inference, we construct a timed automaton for each sequential conjunctive subformula of the inferred MITL formulas from both tasks. We perform RL on the extended state which includes the locations and clock valuations of the timed automata for the source task. We then establish mappings between the corresponding components (clocks, locations, etc.) of the timed automata from the two tasks, and transfer the extended Q-functions based on the established mappings. Finally, we perform RL on the extended state for the target task, starting with the transferred extended Q-functions. Our implementation results show, depending on how similar the source task and the target task are, that the sampling efficiency for the target task can be improved by up to one order of magnitude by performing RL in the extended state space, and further improved by up to another order of magnitude using the transferred extended Q-functions.

| S-EPMC6800702 | biostudies-literature

Risk-taking unmasked: Using risky choice and temporal discounting to explain COVID-19 preventative behaviors.

Project description:To reduce the spread of COVID-19 transmission, government agencies in the United States (US) recommended precautionary guidelines, including wearing masks and social distancing to encourage the prevention of the disease. However, compliance with these guidelines has been inconsistent. This correlational study examined whether individual differences in risky decision-making and motivational propensities predicted compliance with COVID-19 preventative behaviors in a sample of US adults (N = 404). Participants completed an online study from September through December 2020 that included a risky choice decision-making task, temporal discounting task, and measures of appropriate mask-wearing, social distancing, and perceived risk of engaging in public activities. Linear regression results indicated that greater temporal discounting and risky decision-making were associated with less appropriate mask-wearing behavior and social distancing. Additionally, demographic factors, including personal experience with COVID-19 and financial difficulties due to COVID-19, were also associated with differences in COVID-19 preventative behaviors. Path analysis results showed that risky decision-making behavior, temporal discounting, and risk perception collectively predicted 55% of the variance in appropriate mask-wearing behavior. Individual differences in general decision-making patterns are therefore highly predictive of who complies with COVID-19 prevention guidelines.

| S-EPMC8118306 | biostudies-literature

Postreward delays and systematic biases in measures of animal temporal discounting.

Project description:Intertemporal choice tasks, which pit smaller/sooner rewards against larger/later ones, are frequently used to study time preferences and, by extension, impulsivity and self-control. When used in animals, many trials are strung together in sequence and an adjusting buffer is added after the smaller/sooner option to hold the total duration of each trial constant. Choices of the smaller/sooner option are not reward maximizing and so are taken to indicate that the animal is discounting future rewards. However, if animals fail to correctly factor in the duration of the postreward buffers, putative discounting behavior may instead reflect constrained reward maximization. Here, we report three results consistent with this discounting-free hypothesis. We find that (i) monkeys are insensitive to the association between the duration of postreward delays and their choices; (ii) they are sensitive to the length of postreward delays, although they greatly underestimate them; and (iii) increasing the salience of the postreward delay biases monkeys toward the larger/later option, reducing measured discounting rates. These results are incompatible with standard discounting-based accounts but are compatible with an alternative heuristic model. Our data suggest that measured intertemporal preferences in animals may not reflect impulsivity, or even mental discounting of future options, and that standard human and animal intertemporal choice tasks measure unrelated mental processes.

| S-EPMC3780845 | biostudies-other

Gain-loss asymmetry in neural correlates of temporal discounting: An approach-avoidance motivation perspective.

Project description:Gain-loss asymmetry in temporal discounting (i.e., when individuals discount gains more than losses) has been implicated in numerous problematic and addictive behaviors, resulting in enormous personal and societal costs. On the basis of findings from a previous study, we speculated that approach-avoidance motivation would modulate gain-loss asymmetry. To test this speculation, we examined the effects of motivation on gain-loss asymmetry by analyzing functional connectivity. We found that approach and avoidance motivation were negatively associated with functional connectivity between the medial orbitofrontal cortex (MOFC) and the dorsolateral prefrontal cortex and functional connectivity between the MOFC and the posterior parietal cortex (PPC) in the gain domain. Only avoidance motivation was found to be positively associated with functional connectivity between the medial prefrontal cortex (MPFC) and the posterior cingulate cortex (PCC) as well as between the MPFC and the insula in the loss domain. Our findings suggest that the relationships of approach-avoidance motivation and neural correlates yielded an asymmetrical pattern between the gain and loss domains in temporal discounting. Thus, we provide new insight into understanding gain-loss asymmetry in temporal discounting.

| S-EPMC4997255 | biostudies-literature

Navigational Behavior of Humans and Deep Reinforcement Learning Agents.

Project description:Rapid advances in the field of Deep Reinforcement Learning (DRL) over the past several years have led to artificial agents (AAs) capable of producing behavior that meets or exceeds human-level performance in a wide variety of tasks. However, research on DRL frequently lacks adequate discussion of the low-level dynamics of the behavior itself and instead focuses on meta-level or global-level performance metrics. In doing so, the current literature lacks perspective on the qualitative nature of AA behavior, leaving questions regarding the spatiotemporal patterning of their behavior largely unanswered. The current study explored the degree to which the navigation and route selection trajectories of DRL agents (i.e., AAs trained using DRL) through simple obstacle ridden virtual environments were equivalent (and/or different) from those produced by human agents. The second and related aim was to determine whether a task-dynamical model of human route navigation could not only be used to capture both human and DRL navigational behavior, but also to help identify whether any observed differences in the navigational trajectories of humans and DRL agents were a function of differences in the dynamical environmental couplings.

| S-EPMC8493935 | biostudies-literature

OmicsDI is part of the ELIXIR infrastructure

OmicsDI is an Elixir interoperability service. Learn more ›

Tweets

OmicsDI Databases

PRIDE
PeptideAtlas
MassIVE
JPOST Repository
Physiome Model Repository

EGA
EVA
ENA
LINCS
PAXDB
Cell Collective

MetaboLights
Metabolomics Workbench
MetabolomeExpress
GNPS
BioModels
FAIRDOMHub

ArrayExpress
dbGaP
ExpressionAtlas
GEO
NODE

Information

Databases
Help
API
Contact us
Code on GitHub
Terms of use
Submit Data