The value of what's to come: Neural mechanisms coupling prediction error and the utility of anticipation.
Ontology highlight
ABSTRACT:
SUBMITTER: Iigaya K
PROVIDER: S-EPMC7304967 | biostudies-literature |
REPOSITORIES: biostudies-literature
Similar Datasets
Project description:Few studies have explored neural mechanisms of reward learning in ASD despite evidence of behavioral impairments of predictive abilities in ASD. To investigate the neural correlates of reward prediction errors in ASD, 16 adults with ASD and 14 typically developing controls performed a prediction error task during fMRI scanning. Results revealed greater activation in the ASD group in the left paracingulate gyrus during signed prediction errors and the left insula and right frontal pole during thresholded unsigned prediction errors. Findings support atypical neural processing of reward prediction errors in ASD in frontostriatal regions critical for prediction coding and reward learning. Results provide a neural basis for impairments in reward learning that may contribute to traits common in ASD (e.g., intolerance of unpredictability).
| S-EPMC6634058 | biostudies-literature
Project description:Both perceptual inference and motor responses are shaped by learned probabilities. For example, stimulus-induced responses in sensory cortices and preparatory activity in premotor cortex reflect how (un)expected a stimulus is. This is in accordance with predictive coding accounts of brain function, which posit a fundamental role of prediction errors for learning and adaptive behavior. We used functional magnetic resonance imaging and recent advances in computational modeling to investigate how (failures of) learned predictions about visual stimuli influence subsequent motor responses. Healthy volunteers discriminated visual stimuli that were differentially predicted by auditory cues. Critically, the predictive strengths of cues varied over time, requiring subjects to continuously update estimates of stimulus probabilities. This online inference, modeled using a hierarchical Bayesian learner, was reflected behaviorally: speed and accuracy of motor responses increased significantly with predictability of the stimuli. We used nonlinear dynamic causal modeling to demonstrate that striatal prediction errors are used to tune functional coupling in cortical networks during learning. Specifically, the degree of striatal trial-by-trial prediction error activity controls the efficacy of visuomotor connections and thus the influence of surprising stimuli on premotor activity. This finding substantially advances our understanding of striatal function and provides direct empirical evidence for formal learning theories that posit a central role for prediction error-dependent plasticity.
| S-EPMC3044875 | biostudies-literature
Project description:Lithium is one of the most effective mood-stabilizing medications in bipolar disorder. This study was designed to test whether lithium administration may stabilize mood via effects on reward processing. It was hypothesized that lithium administration would modulate reward processing in the striatum and affect both anticipation and outcome computations. Thirty-seven healthy human participants (18 males, 33 with suitable fMRI data) received 11 (±1) days of lithium carbonate or placebo intervention (double-blind), after which they completed the monetary incentive delay task while fMRI data were collected. The monetary incentive delay task is a robust task with excellent test-retest reliability and is well suited to investigate different phases of reward processing within the caudate and nucleus accumbens. To test for correlations with prediction error signals a Rescorla-Wagner reinforcement-learning model was applied. Lithium administration enhanced activity in the caudate during reward anticipation compared to placebo. In contrast, lithium administration reduced caudate and nucleus accumbens activity during reward outcome. This latter effect seems related to learning as reward prediction errors showed a positive correlation with caudate and nucleus accumbens activity during placebo, which was absent after lithium administration. Lithium differentially modulates the anticipation relative to the learning of rewards. This suggests that lithium might reverse dampened reward anticipation while reducing overactive reward updating in patients with bipolar disorder. This specific effect of lithium suggests that a targeted modulation of reward learning may be a viable approach for novel interventions in bipolar disorder.
| S-EPMC7853118 | biostudies-literature
Project description:BackgroundOptimal choices require an accurate neuronal representation of economic value. In economics, utility functions are mathematical representations of subjective value that can be constructed from choices under risk. Utility usually exhibits a nonlinear relationship to physical reward value that corresponds to risk attitudes and reflects the increasing or decreasing marginal utility obtained with each additional unit of reward. Accordingly, neuronal reward responses coding utility should robustly reflect this nonlinearity.ResultsIn two monkeys, we measured utility as a function of physical reward value from meaningful choices under risk (that adhered to first- and second-order stochastic dominance). The resulting nonlinear utility functions predicted the certainty equivalents for new gambles, indicating that the functions' shapes were meaningful. The monkeys were risk seeking (convex utility function) for low reward and risk avoiding (concave utility function) with higher amounts. Critically, the dopamine prediction error responses at the time of reward itself reflected the nonlinear utility functions measured at the time of choices. In particular, the reward response magnitude depended on the first derivative of the utility function and thus reflected the marginal utility. Furthermore, dopamine responses recorded outside of the task reflected the marginal utility of unpredicted reward. Accordingly, these responses were sufficient to train reinforcement learning models to predict the behaviorally defined expected utility of gambles.ConclusionsThese data suggest a neuronal manifestation of marginal utility in dopamine neurons and indicate a common neuronal basis for fundamental explanatory constructs in animal learning theory (prediction error) and economic decision theory (marginal utility).
| S-EPMC4228052 | biostudies-literature
Project description:Dopamine neurons facilitate learning by calculating reward prediction error, or the difference between expected and actual reward. Despite two decades of research, it remains unclear how dopamine neurons make this calculation. Here we review studies that tackle this problem from a diverse set of approaches, from anatomy to electrophysiology to computational modeling and behavior. Several patterns emerge from this synthesis: that dopamine neurons themselves calculate reward prediction error, rather than inherit it passively from upstream regions; that they combine multiple separate and redundant inputs, which are themselves interconnected in a dense recurrent network; and that despite the complexity of inputs, the output from dopamine neurons is remarkably homogeneous and robust. The more we study this simple arithmetic computation, the knottier it appears to be, suggesting a daunting (but stimulating) path ahead for neuroscience more generally.
| S-EPMC6721851 | biostudies-literature
Project description:The functional neuroanatomy and connectivity of reward processing in adults are well documented, with relatively less research on adolescents, a notable gap given this developmental period's association with altered reward sensitivity. Here, a large sample (n?=?1,510) of adolescents performed the monetary incentive delay (MID) task during functional magnetic resonance imaging. Probabilistic maps identified brain regions that were reliably responsive to reward anticipation and receipt, and to prediction errors derived from a computational model. Psychophysiological interactions analyses were used to examine functional connections throughout reward processing. Bilateral ventral striatum, pallidum, insula, thalamus, hippocampus, cingulate cortex, midbrain, motor area, and occipital areas were reliably activated during reward anticipation. Bilateral ventromedial prefrontal cortex and bilateral thalamus exhibited positive and negative activation, respectively, during reward receipt. Bilateral ventral striatum was reliably active following prediction errors. Previously, individual differences in the personality trait of sensation seeking were shown to be related to individual differences in sensitivity to reward outcome. Here, we found that sensation seeking scores were negatively correlated with right inferior frontal gyrus activity following reward prediction errors estimated using a computational model. Psychophysiological interactions demonstrated widespread cortical and subcortical connectivity during reward processing, including connectivity between reward-related regions with motor areas and the salience network. Males had more activation in left putamen, right precuneus, and middle temporal gyrus during reward anticipation. In summary, we found that, in adolescents, different reward processing stages during the MID task were robustly associated with distinctive patterns of activation and of connectivity.
| S-EPMC6865381 | biostudies-literature
Project description:When deciding, we aim to choose the "best" possible outcome. This is not just selection of the option that is the most numerous or physically largest, as options are translated from objective value (count) to subjective value (worth or utility). We localized the neural instantiation of the value-to-utility transformation to the dorsal anterior midcingulate cortex (daMCC), with independent replication. The daMCC encodes the context-specific information necessary to convert from count to worth. This encoding is not simply a representation of utility or preference, but the interaction of the two. Specifically, the relationship of brain activation to value is dependent on individual preference, with both positive and negative slopes across the population depending on whether each individual's preference results in enhancement or diminishment of the valuation. For a given value, across participants, enhanced daMCC activation corresponds to diminished subjective valuation, deactivation to enhanced subjective valuation, and non-modulated activation with non-modulated subjective valuation. Further, functional connectivity analyses identified brain regions (positive connectivity with the inferior frontal gyrus and negative connectivity with the nucleus accumbens) through which contextual information may be integrated into the daMCC and allow for outputs to modulate valuation signals. All analyses were replicated through an independent within-study replication, with initial testing in the gains domain and replication in the intermixed and mirrored losses trials. We also present and discuss an ancillary finding: we were unable to identify parametric value signals for losses through whole-brain analyses, and ROI analyses of the vmPFC presented non-modulation across loss value levels. These results identify the neural locus of the value-to-utility transformation, and provide a specific computational function for the daMCC in the production of subjective valuation through the integration of value, context, and preferences.
| S-EPMC5101215 | biostudies-literature
Project description:Adolescence is associated with quickly changing environmental demands which require excellent adaptive skills and high cognitive flexibility. Feedback-guided adaptive learning and cognitive flexibility are driven by reward prediction error (RPE) signals, which indicate the accuracy of expectations and can be estimated using computational models. Despite the importance of cognitive flexibility during adolescence, only little is known about how RPE processing in cognitive flexibility deviates between adolescence and adulthood. In this study, we investigated the developmental aspects of cognitive flexibility by means of computational models and functional magnetic resonance imaging (fMRI). We compared the neural and behavioral correlates of cognitive flexibility in healthy adolescents (12-16years) to adults performing a probabilistic reversal learning task. Using a modified risk-sensitive reinforcement learning model, we found that adolescents learned faster from negative RPEs than adults. The fMRI analysis revealed that within the RPE network, the adolescents had a significantly altered RPE-response in the anterior insula. This effect seemed to be mainly driven by increased responses to negative prediction errors. In summary, our findings indicate that decision making in adolescence goes beyond merely increased reward-seeking behavior and provides a developmental perspective to the behavioral and neural mechanisms underlying cognitive flexibility in the context of reinforcement learning.
| S-EPMC4330550 | biostudies-literature
Project description:The role of the orbitofrontal cortex (OFC) in value processing is a focus of research. Conventional imaging analysis, where smoothing and averaging are employed, may not be sufficiently sensitive in studying the OFC, which has heterogeneous anatomical structures and functions. In this study, we employed representational similarity analysis (RSA) to reveal the multi-voxel fMRI patterns in the OFC associated with value processing during the anticipatory and the consummatory phases. We found that multi-voxel activation patterns in the OFC encoded magnitude and partial valence information (win vs. loss) but not outcome (favourable vs. unfavourable) during reward consummation. Furthermore, the lateral OFC rather than the medial OFC encoded loss information. Also, we found that OFC encoded values in a similar way to the ventral striatum (VS) or the anterior insula (AI) during reward anticipation regardless of motivated response and to the medial prefrontal cortex (MPFC) and the VS in reward consummation. In contrast, univariate analysis did not show changes of activation in the OFC. These findings suggest an important role of the OFC in value processing during reward anticipation and consummation.
| S-EPMC4932626 | biostudies-literature
Project description:An extensive literature from cognitive neuroscience examines the neural representation of value, but interpretations of these existing results are often complicated by the potential confound of saliency. At the same time, recent attempts to dissociate neural signals of value and saliency have not addressed their relationship with category information. Using a multi-category valuation task that incorporates rewards and punishments of different nature, we identify distributed neural representation of value, saliency, and category during outcome anticipation. Moreover, we reveal category encoding in multi-voxel blood-oxygen-level-dependent activity patterns of the vmPFC and the striatum that coexist with value signals. These results help clarify ambiguities regarding value and saliency encoding in the human brain and their category independence, lending strong support to the neural "common currency" hypothesis. Our results also point to potential novel mechanisms of integrating multiple aspects of decision-related information.
| S-EPMC5714958 | biostudies-literature