Computational medication regimen for Parkinson's disease using reinforcement learning.
Ontology highlight
ABSTRACT:
SUBMITTER: Kim Y
PROVIDER: S-EPMC8085228 | biostudies-literature |
REPOSITORIES: biostudies-literature
Similar Datasets
Project description:Reinforcement learning is a computational framework for an active agent to learn behaviors on the basis of a scalar reward signal. The agent can be an animal, a human, or an artificial system such as a robot or a computer program. The reward can be food, water, money, or whatever measure of the performance of the agent. The theory of reinforcement learning, which was developed in an artificial intelligence community with intuitions from animal learning theory, is now giving a coherent account on the function of the basal ganglia. It now serves as the "common language" in which biologists, engineers, and social scientists can exchange their problems and findings. This article reviews the basic theoretical framework of reinforcement learning and discusses its recent and future contributions toward the understanding of animal behaviors and human decision making.
| S-EPMC2645553 | biostudies-other
Project description:Emerging evidence suggests that dopamine may modulate learning and memory with important implications for understanding the neurobiology of memory and future therapeutic targeting. An influential hypothesis posits that dopamine biases reinforcement learning. More recent data also suggest an influence during both consolidation and retrieval. Eighteen Parkinson's disease patients learned through feedback ON or OFF medication, with memory tested 24 hr later ON or OFF medication (4 conditions, within-subjects design with matched healthy control group). Patients OFF medication during learning decreased in memory accuracy over the following 24 hr. In contrast to previous studies, however, dopaminergic medication during learning and testing did not affect expression of positive or negative reinforcement. Two further experiments were run without the 24 hr delay, but they too failed to reproduce effects of dopaminergic medication on reinforcement learning. While supportive of a dopaminergic role in consolidation, this study failed to replicate previous findings on reinforcement learning.
| S-EPMC5531832 | biostudies-literature
Project description:Empirical evidence suggests that levodopa medication used to treat the motor symptoms of Parkinson's disease (PD) may either improve, impair or not affect specific cognitive processes. This evidence led to the 'dopamine overdose' hypothesis that levodopa medication impairs performance on cognitive tasks if they recruit fronto-striatal circuits which are not yet dopamine-depleted in early PD and as a result the medication leads to an excess of dopamine. This hypothesis has been supported for various learning tasks including conditional associative learning, reversal learning, classification learning and intentional deterministic sequence learning, on all of which PD patients demonstrated significantly worse performance when tested on relative to off dopamine medication. Incidental sequence learning is impaired in PD, but how such learning is affected by dopaminergic therapy remains undetermined. The aim of the current study was to investigate the effect of dopaminergic medication on incidental sequence learning in PD. We used a probabilistic serial reaction time task (SRTT), a sequence learning paradigm considered to make the sequence less apparent and more likely to be learned incidentally rather than intentionally. We compared learning by the same group of PD patients (n=15) on two separate occasions following oral administration of levodopa medication (on state) and after overnight withdrawal of medication (off state). Our results demonstrate for the first time that levodopa medication enhances incidental learning of a probabilistic sequence on the serial reaction time task in PD. However, neither group significantly differed from performance of a control group without a neurological disease, which indicates the importance of within group comparisons for identifying deficits. Levodopa medication enhanced incidental learning by patients with PD on a probabilistic sequence learning paradigm even though the patients were not aware of the existence of the sequence or their acquired knowledge. The results suggest a role in acquiring incidental motor sequence learning for dorsal striatal areas strongly affected by dopamine depletion in early PD.
| S-EPMC5155668 | biostudies-literature
Project description:The role dopamine plays in decision-making has important theoretical, empirical and clinical implications. Here, we examined its precise contribution by exploiting the lesion deficit model afforded by Parkinson's disease. We studied patients in a two-stage reinforcement learning task, while they were ON and OFF dopamine replacement medication. Contrary to expectation, we found that dopaminergic drug state (ON or OFF) did not impact learning. Instead, the critical factor was drug state during the performance phase, with patients ON medication choosing correctly significantly more frequently than those OFF medication. This effect was independent of drug state during initial learning and appears to reflect a facilitation of generalization for learnt information. This inference is bolstered by our observation that neural activity in nucleus accumbens and ventromedial prefrontal cortex, measured during simultaneously acquired functional magnetic resonance imaging, represented learnt stimulus values during performance. This effect was expressed solely during the ON state with activity in these regions correlating with better performance. Our data indicate that dopamine modulation of nucleus accumbens and ventromedial prefrontal cortex exerts a specific effect on choice behaviour distinct from pure learning. The findings are in keeping with the substantial other evidence that certain aspects of learning are unaffected by dopamine lesions or depletion, and that dopamine plays a key role in performance that may be distinct from its role in learning.
| S-EPMC3359751 | biostudies-literature
Project description:Parkinson's disease is clinically defined by bradykinesia, along with rigidity and tremor. However, the severity of these motor signs is greatly variable between individuals, particularly the presence or absence of tremor. This variability in tremor relates to variation in cognitive/motivational impairment, as well as the spatial distribution of neurodegeneration in the midbrain and dopamine depletion in the striatum. Here we ask whether interindividual heterogeneity in tremor symptoms could account for the puzzlingly large variability in the effects of dopaminergic medication on reinforcement learning, a fundamental cognitive function known to rely on dopamine. Given that tremor-dominant and non-tremor Parkinson's disease patients have different dopaminergic phenotypes, we hypothesized that effects of dopaminergic medication on reinforcement learning differ between tremor-dominant and non-tremor patients. Forty-three tremor-dominant and 20 non-tremor patients with Parkinson's disease were recruited to be tested both OFF and ON dopaminergic medication (200/50 mg levodopa-benserazide), while 22 age-matched control subjects were recruited to be tested twice OFF medication. Participants performed a reinforcement learning task designed to dissociate effects on learning rate from effects on motivational choice (i.e. the tendency to 'Go/NoGo' in the face of reward/threat of punishment). In non-tremor patients, dopaminergic medication improved reward-based choice, replicating previous studies. In contrast, in tremor-dominant patients, dopaminergic medication improved learning from punishment. Formal modelling showed divergent computational effects of dopaminergic medication as a function of Parkinson's disease motor phenotype, with a modulation of motivational choice bias and learning rate in non-tremor and tremor patients, respectively. This finding establishes a novel cognitive/motivational difference between tremor and non-tremor Parkinson's disease patients, and highlights the importance of considering motor phenotype in future work.
| S-EPMC7719026 | biostudies-literature
Project description:Adolescence is a period of life characterised by changes in learning and decision-making. Learning and decision-making do not rely on a unitary system, but instead require the coordination of different cognitive processes that can be mathematically formalised as dissociable computational modules. Here, we aimed to trace the developmental time-course of the computational modules responsible for learning from reward or punishment, and learning from counterfactual feedback. Adolescents and adults carried out a novel reinforcement learning paradigm in which participants learned the association between cues and probabilistic outcomes, where the outcomes differed in valence (reward versus punishment) and feedback was either partial or complete (either the outcome of the chosen option only, or the outcomes of both the chosen and unchosen option, were displayed). Computational strategies changed during development: whereas adolescents' behaviour was better explained by a basic reinforcement learning algorithm, adults' behaviour integrated increasingly complex computational features, namely a counterfactual learning module (enabling enhanced performance in the presence of complete feedback) and a value contextualisation module (enabling symmetrical reward and punishment learning). Unlike adults, adolescent performance did not benefit from counterfactual (complete) feedback. In addition, while adults learned symmetrically from both reward and punishment, adolescents learned from reward but were less likely to learn from punishment. This tendency to rely on rewards and not to consider alternative consequences of actions might contribute to our understanding of decision-making in adolescence.
| S-EPMC4920542 | biostudies-literature
Project description:Many studies have explored factors which influence gaze-following behavior of young infants. However, the results of empirical studies were inconsistent, and the mechanism underlying the contextual modulation of gaze following remains unclear. In order to provide valuable insight into the mechanisms underlying gaze following, we conducted computational modeling using Q-learning algorithm and simulated the learning process of infant gaze following to suggest a feasible model. In Experiment 1, we simulated how communicative cues and infant internal states affect the learning process of gaze following. The simulation indicated that the model in which communicative cues enhance infant internal states is the most feasible to explain the infant learning process. In Experiment 2, we simulated how individual differences in motivation for communication affect the learning process. The results showed that low motivation for communication can delay the learning process and decrease the frequency of gaze following. These simulations suggest that communicative cues may enhance infants' internal states and promote the development of gaze following. Also, initial social motivation may affect the learning process of social behaviors in the long term.
| S-EPMC7063100 | biostudies-literature
Project description:Instrumental learning involves corticostriatal circuitry and the dopaminergic system. This system is typically modeled in the reinforcement learning (RL) framework by incrementally accumulating reward values of states and actions. However, human learning also implicates prefrontal cortical mechanisms involved in higher level cognitive functions. The interaction of these systems remains poorly understood, and models of human behavior often ignore working memory (WM) and therefore incorrectly assign behavioral variance to the RL system. Here we designed a task that highlights the profound entanglement of these two processes, even in simple learning problems. By systematically varying the size of the learning problem and delay between stimulus repetitions, we separately extracted WM-specific effects of load and delay on learning. We propose a new computational model that accounts for the dynamic integration of RL and WM processes observed in subjects' behavior. Incorporating capacity-limited WM into the model allowed us to capture behavioral variance that could not be captured in a pure RL framework even if we (implausibly) allowed separate RL systems for each set size. The WM component also allowed for a more reasonable estimation of a single RL process. Finally, we report effects of two genetic polymorphisms having relative specificity for prefrontal and basal ganglia functions. Whereas the COMT gene coding for catechol-O-methyl transferase selectively influenced model estimates of WM capacity, the GPR6 gene coding for G-protein-coupled receptor 6 influenced the RL learning rate. Thus, this study allowed us to specify distinct influences of the high-level and low-level cognitive functions on instrumental learning, beyond the possibilities offered by simple RL models.
| S-EPMC3390186 | biostudies-literature
Project description:The robotics field has been deeply influenced by the advent of deep learning. In recent years, this trend has been characterized by the adoption of large, pretrained models for robotic use cases, which are not compatible with the computational hardware available in robotic systems. Moreover, such large, computationally intensive models impede the low-latency execution which is required for many closed-loop control systems. In this work, we propose different strategies for improving the computational efficiency of the deep-learning models adopted in reinforcement-learning (RL) scenarios. As a use-case project, we consider an image-based RL method on the synergy between push-and-grasp actions. As a first optimization step, we reduce the model architecture in complexity, by decreasing the number of layers and by altering the architecture structure. Second, we consider downscaling the input resolution to reduce the computational load. Finally, we perform weight quantization, where we compare post-training quantization and quantized-aware training. We benchmark the improvements introduced in each optimization by running a standard testing routine. We show that the optimization strategies introduced can improve the computational efficiency by around 300 times, while also slightly improving the functional performance of the system. In addition, we demonstrate closed-loop control behaviour on a real-world robot, while processing everything on a Jetson Xavier NX edge device.
| S-EPMC9571553 | biostudies-literature
Project description:Humans have the fascinating ability to achieve goals in a complex and constantly changing world, still surpassing modern machine-learning algorithms in terms of flexibility and learning speed. It is generally accepted that a crucial factor for this ability is the use of abstract, hierarchical representations, which employ structure in the environment to guide learning and decision making. Nevertheless, how we create and use these hierarchical representations is poorly understood. This study presents evidence that human behavior can be characterized as hierarchical reinforcement learning (RL). We designed an experiment to test specific predictions of hierarchical RL using a series of subtasks in the realm of context-based learning and observed several behavioral markers of hierarchical RL, such as asymmetric switch costs between changes in higher-level versus lower-level features, faster learning in higher-valued compared to lower-valued contexts, and preference for higher-valued compared to lower-valued contexts. We replicated these results across three independent samples. We simulated three models-a classic RL, a hierarchical RL, and a hierarchical Bayesian model-and compared their behavior to human results. While the flat RL model captured some aspects of participants' sensitivity to outcome values, and the hierarchical Bayesian model captured some markers of transfer, only hierarchical RL accounted for all patterns observed in human behavior. This work shows that hierarchical RL, a biologically inspired and computationally simple algorithm, can capture human behavior in complex, hierarchical environments and opens the avenue for future research in this field.
| S-EPMC7703642 | biostudies-literature