One-shot learning and behavioral eligibility traces in sequential decision making.
Ontology highlight
ABSTRACT:
SUBMITTER: Lehmann MP
PROVIDER: S-EPMC6897511 | biostudies-literature | 2019 Nov
REPOSITORIES: biostudies-literature
Publications
One-shot learning and behavioral eligibility traces in sequential decision making.
Lehmann Marco P MP Xu He A HA Liakoni Vasiliki V Herzog Michael H MH Gerstner Wulfram W Preuschoff Kerstin K
eLife 20191111
In many daily tasks, we make multiple decisions before reaching a goal. In order to learn such sequences of decisions, a mechanism to link earlier actions to later reward is necessary. Reinforcement learning (RL) theory suggests two classes of algorithms solving this credit assignment problem: In classic temporal-difference learning, earlier actions receive reward information only after multiple repetitions of the task, whereas models with eligibility traces reinforce entire sequences of actions ...[more]
Similar Datasets
Project description:In daily life, people often make consecutive decisions before the ultimate goal is reached (i.e., sequential decision-making). However, this kind of decision-making has been largely overlooked in the literature. The current study investigated whether behavioral preference would change during sequential decisions, and the neural processes underlying the potential changes. For this purpose, we revised the classic balloon analogue risk task and recorded the electroencephalograph (EEG) signals associated with each step of decision-making. Independent component analysis performed on EEG data revealed that four EEG components elicited by periodic feedback in the current step predicted participants' decisions (gamble vs. no gamble) in the next step. In order of time sequence, these components were: bilateral occipital alpha rhythm, bilateral frontal theta rhythm, middle frontal theta rhythm, and bilateral sensorimotor mu rhythm. According to the information flows between these EEG oscillations, we proposed a brain model that describes the temporal dynamics of sequential decision-making. Finally, we found that the tendency to gamble (as well as the power intensity of bilateral frontal theta rhythms) was sensitive to the individual level of trait anxiety in certain steps, which may help understand the role of emotion in decision-making.
| S-EPMC6866314 | biostudies-literature
Project description:People often make decisions in a social environment. The present work examines social influence on people's decisions in a sequential decision-making situation. In the first experimental study, we implemented an information cascade paradigm, illustrating that people infer information from decisions of others and use this information to make their own decisions. We followed a cognitive modeling approach to elicit the weight people give to social as compared to private individual information. The proposed social influence model shows that participants overweight their own private information relative to social information, contrary to the normative Bayesian account. In our second study, we embedded the abstract decision problem of Study 1 in a medical decision-making problem. We examined whether in a medical situation people also take others' authority into account in addition to the information that their decisions convey. The social influence model illustrates that people weight social information differentially according to the authority of other decision makers. The influence of authority was strongest when an authority's decision contrasted with private information. Both studies illustrate how the social environment provides sources of information that people integrate differently for their decisions.
| S-EPMC4718651 | biostudies-literature
Project description:Dual system theories suggest that behavioral control is parsed between a deliberative "model-based" and a more reflexive "model-free" system. A balance of control exerted by these systems is thought to be related to dopamine neurotransmission. However, in the absence of direct measures of human dopamine, it remains unknown whether this reflects a quantitative relation with dopamine either in the striatum or other brain areas. Using a sequential decision task performed during functional magnetic resonance imaging, combined with striatal measures of dopamine using [(18)F]DOPA positron emission tomography, we show that higher presynaptic ventral striatal dopamine levels were associated with a behavioral bias toward more model-based control. Higher presynaptic dopamine in ventral striatum was associated with greater coding of model-based signatures in lateral prefrontal cortex and diminished coding of model-free prediction errors in ventral striatum. Thus, interindividual variability in ventral striatal presynaptic dopamine reflects a balance in the behavioral expression and the neural signatures of model-free and model-based control. Our data provide a novel perspective on how alterations in presynaptic dopamine levels might be accompanied by a disruption of behavioral control as observed in aging or neuropsychiatric diseases such as schizophrenia and addiction.
| S-EPMC4321318 | biostudies-literature
Project description:Many decisions in life are sequential and constrained by a time window. Although mathematically derived optimal solutions exist, it has been reported that humans often deviate from making optimal choices. Here, we used a secretary problem, a classic example of finite sequential decision-making, and investigated the mechanisms underlying individuals' suboptimal choices. Across three independent experiments, we found that a dynamic programming model comprising subjective value function explains individuals' deviations from optimality and predicts the choice behaviors under fewer and more opportunities. We further identified that pupil dilation reflected the levels of decision difficulty and subsequent choices to accept or reject the stimulus at each opportunity. The value sensitivity, a model-based estimate that characterizes each individual's subjective valuation, correlated with the extent to which individuals' physiological responses tracked stimuli information. Our results provide model-based and physiological evidence for subjective valuation in finite sequential decision-making, rediscovering human suboptimality in subjectively optimal decision-making processes.
| S-EPMC8675647 | biostudies-literature
Project description:The role of prediction error (PE) in driving learning is well-established in fields such as classical and instrumental conditioning, reward learning and procedural memory; however, its role in human one-shot declarative encoding is less clear. According to one recent hypothesis, PE reflects the divergence between two probability distributions: one reflecting the prior probability (from previous experiences) and the other reflecting the sensory evidence (from the current experience). Assuming unimodal probability distributions, PE can be manipulated in three ways: (1) the distance between the mode of the prior and evidence, (2) the precision of the prior, and (3) the precision of the evidence. We tested these three manipulations across five experiments, in terms of peoples' ability to encode a single presentation of a scene-item pairing as a function of previous exposures to that scene and/or item. Memory was probed by presenting the scene together with three choices for the previously paired item, in which the two foil items were from other pairings within the same condition as the target item. In Experiment 1, we manipulated the evidence to be either consistent or inconsistent with prior expectations, predicting PE to be larger, and hence memory better, when the new pairing was inconsistent. In Experiments 2a-c, we manipulated the precision of the priors, predicting better memory for a new pairing when the (inconsistent) priors were more precise. In Experiment 3, we manipulated both visual noise and prior exposure for unfamiliar faces, before pairing them with scenes, predicting better memory when the sensory evidence was more precise. In all experiments, the PE hypotheses were supported. We discuss alternative explanations of individual experiments, and conclude the Predictive Interactive Multiple Memory Signals (PIMMS) framework provides the most parsimonious account of the full pattern of results.
| S-EPMC5381756 | biostudies-literature
Project description:Background and objectivesBehavioral stage of change (SoC) algorithms classify patients' readiness for medical treatment decision-making. In the precontemplation stage, patients have no intention to take action within 6 months. In the contemplation stage, action is intended within 6 months. In the preparation stage, patients intend to take action within 30 days. In the action stage, the change has been made. This study examines the influence of SoC on dialysis modality decision-making.Design, setting, participants, & measurementsSoC and relevant covariates were measured, and associations with dialysis decision-making were determined. In-depth interviews were conducted with 16 patients on dialysis to elicit experiences. Qualitative interview data informed the survey design. Surveys were administered to adults with CKD (eGFR?25 ml/min/1.73 m(2)) from August, 2012 to June, 2013. Multivariable logistic regression modeled dialysis decision-making with predictors: SoC, provider connection, and dialysis knowledge score.ResultsFifty-five patients completed the survey (71% women, 39% white, and 59% black), and median annual income was $17,500. In total, 65% of patients were in the precontemplation/contemplation (thinking) and 35% of patients were in the preparation/maintenance (acting) SoC; 62% of patients had made dialysis modality decisions. Doctors explaining modality options, higher dialysis knowledge scores, and fewer lifestyle barriers were associated with acting versus thinking SoC (all P<0.02). Patients making modality decisions had doctors who explained dialysis options (76% versus 43%), were in the acting versus the thinking SoC (50% versus 10%), had higher dialysis knowledge scores (1.4 versus 0.5), and had lower eGFR (13.9 versus 16.8 ml/min/1.73 m(2); all P<0.05). In adjusted analyses, dialysis knowledge was significantly associated with decision-making (odds ratio, 4.2; 95% confidence interval, 1.4 to 12.9; P=0.01), and SoC was of borderline significance (odds ratio, 5.8; 95% confidence interval, 1.0 to 32.6; P=0.05). The model C statistic was 0.87.ConclusionsDialysis decision-making was associated with SoC, dialysis knowledge, and physicians discussing treatment options. Future studies determining ways to assist patients with CKD in making satisfying modality decisions are warranted.
| S-EPMC4317741 | biostudies-literature
Project description:BackgroundSurgical patients incur preventable harm from cognitive and judgment errors made under time constraints and uncertainty regarding patients' diagnoses and predicted response to treatment. Decision analysis and techniques of reinforcement learning theoretically can mitigate these challenges but are poorly understood and rarely used clinically. This review seeks to promote an understanding of decision analysis and reinforcement learning by describing their use in the context of surgical decision-making.MethodsCochrane, EMBASE, and PubMed databases were searched from their inception to June 2019. Included were 41 articles about cognitive and diagnostic errors, decision-making, decision analysis, and machine-learning. The articles were assimilated into relevant categories according to Preferred Reporting Items for Systematic Reviews and Meta-Analyses extension for Scoping Reviews guidelines.ResultsRequirements for time-consuming manual data entry and crude representations of individual patients and clinical context compromise many traditional decision-support tools. Decision analysis methods for calculating probability thresholds can inform population-based recommendations that jointly consider risks, benefits, costs, and patient values but lack precision for individual patient-centered decisions. Reinforcement learning, a machine-learning method that mimics human learning, can use a large set of patient-specific input data to identify actions yielding the greatest probability of achieving a goal. This methodology follows a sequence of events with uncertain conditions, offering potential advantages for personalized, patient-centered decision-making. Clinical application would require secure integration of multiple data sources and attention to ethical considerations regarding liability for errors and individual patient preferences.ConclusionTraditional decision-support tools are ill-equipped to accommodate time constraints and uncertainty regarding diagnoses and the predicted response to treatment, both of which often impair surgical decision-making. Decision analysis and reinforcement learning have the potential to play complementary roles in delivering high-value surgical care through sound judgment and optimal decision-making.
| S-EPMC7390703 | biostudies-literature
Project description:Incremental learning, in which new knowledge is acquired gradually through trial and error, can be distinguished from one-shot learning, in which the brain learns rapidly from only a single pairing of a stimulus and a consequence. Very little is known about how the brain transitions between these two fundamentally different forms of learning. Here we test a computational hypothesis that uncertainty about the causal relationship between a stimulus and an outcome induces rapid changes in the rate of learning, which in turn mediates the transition between incremental and one-shot learning. By using a novel behavioral task in combination with functional magnetic resonance imaging (fMRI) data from human volunteers, we found evidence implicating the ventrolateral prefrontal cortex and hippocampus in this process. The hippocampus was selectively "switched" on when one-shot learning was predicted to occur, while the ventrolateral prefrontal cortex was found to encode uncertainty about the causal association, exhibiting increased coupling with the hippocampus for high-learning rates, suggesting this region may act as a "switch," turning on and off one-shot learning as required.
| S-EPMC4412411 | biostudies-literature
Project description:From ants to humans, the timing of many animal behaviors comes in bursts of activity separated by long periods of inactivity. Recently, mathematical modeling has shown that simple algorithms of priority-driven behavioral choice can result in bursty behavior. To experimentally test this link between decision-making circuitry and bursty dynamics, we have turned to Drosophila melanogaster. We have found that the statistics of intervals between activity periods in endogenous activity-rest switches of wild-type Drosophila are very well described by the Weibull distribution, a common distribution of bursty dynamics in complex systems. The bursty dynamics of wild-type Drosophila walking activity are shown to be determined by this inter-event distribution alone and not by memory effects, thus resembling human dynamics. Further, using mutant flies that disrupt dopaminergic signaling or the mushroom body, circuitry implicated in decision-making, we show that the degree of behavioral burstiness can be modified. These results are thus consistent with the proposed link between decision-making circuitry and bursty dynamics, and highlight the importance of using simple experimental systems to test general theoretical models of behavior. The findings further suggest that analysis of bursts could prove useful for the study and evaluation of decision-making circuitry.
| S-EPMC3121695 | biostudies-literature
Project description:Making optimal decisions in the face of noise requires balancing short-term speed and accuracy. But a theory of optimality should account for the fact that short-term speed can influence long-term accuracy through learning. Here, we demonstrate that long-term learning is an important dynamical dimension of the speed-accuracy trade-off. We study learning trajectories in rats and formally characterize these dynamics in a theory expressed as both a recurrent neural network and an analytical extension of the drift-diffusion model that learns over time. The model reveals that choosing suboptimal response times to learn faster sacrifices immediate reward, but can lead to greater total reward. We empirically verify predictions of the theory, including a relationship between stimulus exposure and learning speed, and a modulation of reaction time by future learning prospects. We find that rats' strategies approximately maximize total reward over the full learning epoch, suggesting cognitive control over the learning process.
| S-EPMC9928425 | biostudies-literature