Response-based outcome predictions and confidence regulate feedback processing and learning.
Ontology highlight
ABSTRACT:
SUBMITTER: Fromer R
PROVIDER: S-EPMC8121545 | biostudies-literature |
REPOSITORIES: biostudies-literature
Similar Datasets
Project description:Brain-Machine Interfaces (BMIs) can be used to restore function in people living with paralysis. Current BMIs require extensive calibration that increase the set-up times and external inputs for decoder training that may be difficult to produce in paralyzed individuals. Both these factors have presented challenges in transitioning the technology from research environments to activities of daily living (ADL). For BMIs to be seamlessly used in ADL, these issues should be handled with minimal external input thus reducing the need for a technician/caregiver to calibrate the system. Reinforcement Learning (RL) based BMIs are a good tool to be used when there is no external training signal and can provide an adaptive modality to train BMI decoders. However, RL based BMIs are sensitive to the feedback provided to adapt the BMI. In actor-critic BMIs, this feedback is provided by the critic and the overall system performance is limited by the critic accuracy. In this work, we developed an adaptive BMI that could handle inaccuracies in the critic feedback in an effort to produce more accurate RL based BMIs. We developed a confidence measure, which indicated how appropriate the feedback is for updating the decoding parameters of the actor. The results show that with the new update formulation, the critic accuracy is no longer a limiting factor for the overall performance. We tested and validated the system onthree different data sets: synthetic data generated by an Izhikevich neural spiking model, synthetic data with a Gaussian noise distribution, and data collected from a non-human primate engaged in a reaching task. All results indicated that the system with the critic confidence built in always outperformed the system without the critic confidence. Results of this study suggest the potential application of the technique in developing an autonomous BMI that does not need an external signal for training or extensive calibration.
| S-EPMC4033619 | biostudies-literature
Project description:It is well established that learning can occur without external feedback, yet normative reinforcement learning theories have difficulties explaining such instances of learning. Here, we propose that human observers are capable of generating their own feedback signals by monitoring internal decision variables. We investigated this hypothesis in a visual perceptual learning task using fMRI and confidence reports as a measure for this monitoring process. Employing a novel computational model in which learning is guided by confidence-based reinforcement signals, we found that mesolimbic brain areas encoded both anticipation and prediction error of confidence-in remarkable similarity to previous findings for external reward-based feedback. We demonstrate that the model accounts for choice and confidence reports and show that the mesolimbic confidence prediction error modulation derived through the model predicts individual learning success. These results provide a mechanistic neurobiological explanation for learning without external feedback by augmenting reinforcement models with confidence-based feedback.
| S-EPMC4821804 | biostudies-literature
Project description:BackgroundPredicting a system's behavior based on a mathematical model is a primary task in Systems Biology. If the model parameters are estimated from experimental data, the parameter uncertainty has to be translated into confidence intervals for model predictions. For dynamic models of biochemical networks, the nonlinearity in combination with the large number of parameters hampers the calculation of prediction confidence intervals and renders classical approaches as hardly feasible.ResultsIn this article reliable confidence intervals are calculated based on the prediction profile likelihood. Such prediction confidence intervals of the dynamic states can be utilized for a data-based observability analysis. The method is also applicable if there are non-identifiable parameters yielding to some insufficiently specified model predictions that can be interpreted as non-observability. Moreover, a validation profile likelihood is introduced that should be applied when noisy validation experiments are to be interpreted.ConclusionsThe presented methodology allows the propagation of uncertainty from experimental to model predictions. Although presented in the context of ordinary differential equations, the concept is general and also applicable to other types of models. Matlab code which can be used as a template to implement the method is provided at http://www.fdmold.uni-freiburg.de/?ckreutz/PPL.
| S-EPMC3490710 | biostudies-literature
Project description:Empathy allows us to respond to the emotional state of another person. Considering that an empathic interaction may last beyond the initial response, learning mechanisms may be involved in dynamic adaptation of the reaction to the changing emotional state of the other person. However, traditionally, empathy is assessed through sets of isolated reactions to another's distress. Here we address this gap by focusing on adaptive empathy, defined as the ability to learn and adjust one's empathic responses based on feedback. For this purpose, we designed a novel paradigm of associative learning in which participants chose one of two empathic strategies (reappraisal or distraction) to attenuate the distress of a target person, where one strategy had a higher probability of relieving distress. After each choice, participants received feedback about the success of their chosen strategy in relieving the target person's distress, which they could use to inform their future decisions. The results show that the participants made more accurate choices in the adaptive empathy condition than in a non-social control condition, pointing to an advantage for learning from social feedback. We found a correlation between adaptive empathy and a trait measure of cognitive empathy. These findings indicate that the ability to learn about the effectiveness of empathic responses may benefit from incorporating mentalizing abilities. Our findings provide a lab-based model for studying adaptive empathy and point to the potential contribution of learning theory to enhancing our understanding of the dynamic nature of empathy.
| S-EPMC8339423 | biostudies-literature
Project description:Decisions are usually accompanied by a feeling of being wrong or right - a subjective confidence estimate. But what information is this confidence estimate based on, and what is confidence used for? To answer these questions, research has largely focused on confidence regarding current or past decisions, for example identifying how characteristics of the stimulus affect confidence, how confidence can be used as an internally generated feedback signal, and how communicating confidence can affect group decisions. Here, we report two studies which implemented a novel metacognitive measure: predictions of confidence for future perceptual decisions. Using computational modeling of behaviour and EEG, we established that experience-based confidence predictions are one source of information that affects how confident we are in future decision-making, and that learned confidence-expectations affect neural preparation for future decisions. Results from both studies show that participants develop precise confidence predictions informed by past confidence experience. Notably, our results also show that confidence predictions affect performance confidence rated after a decision is made; this finding supports the proposal that confidence judgments are based on multiple sources of information, including expectations. We found strong support for this link in neural correlates of stimulus preparation and processing. EEG measures of preparatory neural activity (contingent negative variation; CNV) and evidence accumulation (centro-parietal positivity; CPP) show that predicted confidence affects neural preparation for stimulus processing, supporting the proposal that one purpose of confidence judgments may be to learn about performance for future encounters and prepare accordingly.
| S-EPMC6411854 | biostudies-literature
Project description:Reliable prediction of orthology is central to comparative genomics. Approaches based on phylogenetic analyses closely resemble the original definition of orthology and paralogy and are known to be highly accurate. However, the large computational cost associated to these analyses is a limiting factor that often prevents its use at genomic scales. Recently, several projects have addressed the reconstruction of large collections of high-quality phylogenetic trees from which orthology and paralogy relationships can be inferred. This provides us with the opportunity to infer the evolutionary relationships of genes from multiple, independent, phylogenetic trees. Using such strategy, we combine phylogenetic information derived from different databases, to predict orthology and paralogy relationships for 4.1 million proteins in 829 fully sequenced genomes. We show that the number of independent sources from which a prediction is made, as well as the level of consistency across predictions, can be used as reliable confidence scores. A webserver has been developed to easily access these data (http://orthology.phylomedb.org), which provides users with a global repository of phylogeny-based orthology and paralogy predictions.
| S-EPMC3061081 | biostudies-other
Project description:Successful learning hinges on the evaluation of positive and negative feedback. We assessed differential learning from reward and punishment in a monetary reinforcement learning paradigm, together with cardiac concomitants of positive and negative feedback processing. On the behavioral level, learning from reward resulted in more advantageous behavior than learning from punishment, suggesting a differential impact of reward and punishment on successful feedback-based learning. On the autonomic level, learning and feedback processing were closely mirrored by phasic cardiac responses on a trial-by-trial basis: (1) Negative feedback was accompanied by faster and prolonged heart rate deceleration compared to positive feedback. (2) Cardiac responses shifted from feedback presentation at the beginning of learning to stimulus presentation later on. (3) Most importantly, the strength of phasic cardiac responses to the presentation of feedback correlated with the strength of prediction error signals that alert the learner to the necessity for behavioral adaptation. Considering participants' weight status and gender revealed obesity-related deficits in learning to avoid negative consequences and less consistent behavioral adaptation in women compared to men. In sum, our results provide strong new evidence for the notion that during learning phasic cardiac responses reflect an internal value and feedback monitoring system that is sensitive to the violation of performance-based expectations. Moreover, inter-individual differences in weight status and gender may affect both behavioral and autonomic responses in reinforcement-based learning.
| S-EPMC5670147 | biostudies-literature
Project description:Previous studies suggest that factual learning, that is, learning from obtained outcomes, is biased, such that participants preferentially take into account positive, as compared to negative, prediction errors. However, whether or not the prediction error valence also affects counterfactual learning, that is, learning from forgone outcomes, is unknown. To address this question, we analysed the performance of two groups of participants on reinforcement learning tasks using a computational model that was adapted to test if prediction error valence influences learning. We carried out two experiments: in the factual learning experiment, participants learned from partial feedback (i.e., the outcome of the chosen option only); in the counterfactual learning experiment, participants learned from complete feedback information (i.e., the outcomes of both the chosen and unchosen option were displayed). In the factual learning experiment, we replicated previous findings of a valence-induced bias, whereby participants learned preferentially from positive, relative to negative, prediction errors. In contrast, for counterfactual learning, we found the opposite valence-induced bias: negative prediction errors were preferentially taken into account, relative to positive ones. When considering valence-induced bias in the context of both factual and counterfactual learning, it appears that people tend to preferentially take into account information that confirms their current choice.
| S-EPMC5568446 | biostudies-literature
Project description:When encountering a visual error during a reaching movement, the motor system improves the motor command for the subsequent trial. This improvement is impaired by visual error uncertainty, which is considered evidence that the motor system optimally estimates the error. However, how such statistical computation is accomplished remains unclear. Here, we propose an alternative scheme implemented with a divisive normalization (DN): the responses of neuronal elements are normalized by the summed activity of the population. This scheme assumes that when an uncertain visual error is provided by multiple cursors, the motor system processes the error conveyed by each cursor and integrates the information using DN. The DN model reproduced the patterns of learning response to 1-3 cursor errors and the impairment of learning response with visual error uncertainty. This study provides a new perspective on how the motor system updates motor commands according to uncertain visual error information.
| S-EPMC10733368 | biostudies-literature
Project description:Uncertainty is ubiquitous in cognitive processing. In this study, we aim to investigate the ability agents possess to track and report the noise inherent in their mental operations, often in the form of confidence judgments. Here, we argue that humans can use uncertainty inherent in their representations of value beliefs to arbitrate between exploration and exploitation. Such uncertainty is reflected in explicit confidence judgments. Using a novel variant of a multi-armed bandit paradigm, we studied how beliefs were formed and how uncertainty in the encoding of these value beliefs (belief confidence) evolved over time. We found that people used uncertainty to arbitrate between exploration and exploitation, reflected in a higher tendency toward exploration when their confidence in their value representations was low. We furthermore found that value uncertainty can be linked to frameworks of metacognition in decision making in two ways. First, belief confidence drives decision confidence, i.e. people's evaluation of their own choices. Second, individuals with higher metacognitive insight into their choices were also better at tracing the uncertainty in their environment. Together, these findings argue that such uncertainty representations play a key role in the context of cognitive control.
| S-EPMC6505439 | biostudies-literature