Post-Contextual-Bandit Inference.
Ontology highlight
ABSTRACT:
SUBMITTER: Bibaut A
PROVIDER: S-EPMC9249103 | biostudies-literature |
REPOSITORIES: biostudies-literature
Similar Datasets
Project description:Online decision-making problem requires us to make a sequence of decisions based on incremental information. Common solutions often need to learn a reward model of different actions given the contextual information and then maximize the long-term reward. It is meaningful to know if the posited model is reasonable and how the model performs in the asymptotic sense. We study this problem under the setup of the contextual bandit framework with a linear reward model. The ε-greedy policy is adopted to address the classic exploration-and-exploitation dilemma. Using the martingale central limit theorem, we show that the online ordinary least squares estimator of model parameters is asymptotically normal. When the linear model is misspecified, we propose the online weighted least squares estimator using the inverse propensity score weighting and also establish its asymptotic normality. Based on the properties of the parameter estimators, we further show that the in-sample inverse propensity weighted value estimator is asymptotically normal. We illustrate our results using simulations and an application to a news article recommendation dataset from Yahoo!.
| S-EPMC7962379 | biostudies-literature
Project description:AsbtractHumans spend a lifetime learning, storing and refining a repertoire of motor memories. For example, through experience, we become proficient at manipulating a large range of objects with distinct dynamical properties. However, it is unknown what principle underlies how our continuous stream of sensorimotor experience is segmented into separate memories and how we adapt and use this growing repertoire. Here we develop a theory of motor learning based on the key principle that memory creation, updating and expression are all controlled by a single computation-contextual inference. Our theory reveals that adaptation can arise both by creating and updating memories (proper learning) and by changing how existing memories are differentially expressed (apparent learning). This insight enables us to account for key features of motor learning that had no unified explanation: spontaneous recovery1, savings2, anterograde interference3, how environmental consistency affects learning rate4,5 and the distinction between explicit and implicit learning6. Critically, our theory also predicts new phenomena-evoked recovery and context-dependent single-trial learning-which we confirm experimentally. These results suggest that contextual inference, rather than classical single-context mechanisms1,4,7-9, is the key principle underlying how a diverse set of experiences is reflected in our motor behaviour.
| S-EPMC8809113 | biostudies-literature
Project description:When probed with complex stimuli that extend beyond their classical receptive field, neurons in primary visual cortex display complex and non-linear response characteristics. Sparse coding models reproduce some of the observed contextual effects, but still fail to provide a satisfactory explanation in terms of realistic neural structures and cortical mechanisms, since the connection scheme they propose consists only of interactions among neurons with overlapping input fields. Here we propose an extended generative model for visual scenes that includes spatial dependencies among different features. We derive a neurophysiologically realistic inference scheme under the constraint that neurons have direct access only to local image information. The scheme can be interpreted as a network in primary visual cortex where two neural populations are organized in different layers within orientation hypercolumns that are connected by local, short-range and long-range recurrent interactions. When trained with natural images, the model predicts a connectivity structure linking neurons with similar orientation preferences matching the typical patterns found for long-ranging horizontal axons and feedback projections in visual cortex. Subjected to contextual stimuli typically used in empirical studies, our model replicates several hallmark effects of contextual processing and predicts characteristic differences for surround modulation between the two model populations. In summary, our model provides a novel framework for contextual processing in the visual system proposing a well-defined functional role for horizontal axons and feedback projections.
| S-EPMC6793885 | biostudies-literature
Project description:The identification of human violence determinants has sparked multiple questions from different academic fields. Innovative methodological assessments of the weight and interaction of multiple determinants are still required. Here, we examine multiple features potentially associated with confessed acts of violence in ex-members of illegal armed groups in Colombia (N = 26,349) through deep learning and feature-derived machine learning. We assessed 162 social-contextual and individual mental health potential predictors of historical data regarding consequentialist, appetitive, retaliative, and reactive domains of violence. Deep learning yields high accuracy using the full set of determinants. Progressive feature elimination revealed that contextual factors were more important than individual factors. Combined social network adversities, membership identification, and normalization of violence were among the more accurate social-contextual factors. To a lesser extent the best individual factors were personality traits (borderline, paranoid, and antisocial) and psychiatric symptoms. The results provide a population-based computational classification regarding historical assessments of violence in vulnerable populations.
| S-EPMC7892360 | biostudies-literature
Project description:Advantageous decision-making is an adaptive trade-off between exploring alternatives and exploiting the most rewarding option. This trade-off may be related to maladaptive decision-making associated with nicotine dependence; however, explore/exploit behavior has not been previously investigated in the context of addiction. The explore/exploit trade-off is captured by the multiarmed bandit task, in which different arms of a slot machine are chosen to discover the relative payoffs. The goal of this study was to preliminarily investigate whether smokers differ from nonsmokers in their degree of exploratory behavior. Smokers (n = 18) and nonsmokers (n = 17) completed a 6-armed bandit task as well as self-report measures of behavior and personality traits. Smokers were found to exhibit less exploratory behavior (i.e., made fewer switches between slot machine arms) than nonsmokers within the first 300 trials of the bandit task. The overall proportion of exploratory choices negatively correlated with self-reported measures of delay aversion and nonplanning impulsivity. These preliminary results suggest that smokers make fewer initial exploratory choices on the bandit task. The bandit task is a promising measure that could provide valuable insights into how nicotine use and dependence is associated with explore/exploit decision-making.
| S-EPMC4028629 | biostudies-literature
Project description:Nonpoint source water quality management is challenged with allocating uncertain management actions and monitoring their performance in the absence of state-dependent decision making. This adaptive management context can be expressed as a multiarmed bandit problem. Multiarmed bandit strategies attempt to balance the exploitation of actions that appear to maximize performance with the exploration of uncertain, but potentially better, actions. We performed a test of multiarmed bandit strategies to inform adaptive water quality management in Massachusetts, USA. Conservation and restoration practitioners were tasked with allocating household wastewater treatments to minimize N inputs to impaired waters. We obtained time series of N monitoring data from 3 wastewater treatment types and organized them chronologically and randomly. The chronological data set represented nonstationary performance based on recent monitoring data, whereas the random data set represented stationary performance. We tested 2 multiarmed bandit strategies in hypothetical experiments to sample from the treatment data through 20 sequential decisions. A deterministic probability-matching strategy allocated treatments with the highest probability of success regarding their performance at each decision. A randomized probability-matching strategy randomly allocated treatments according to their probability of success at each decision. The strategies were compared with a nonadaptive strategy that equally allocated treatments at each decision. Results indicated that equal allocation is useful for learning in nonstationary situations but tended to overexplore inferior treatments and thus did not maximize performance when compared with the other strategies. Deterministic probability matching maximized performance in many stationary situations, but the strategy did not adequately explore treatments and converged on inferior treatments in nonstationary situations. Randomized probability matching balanced performance and learning in stationary situations, but the strategy could converge on inferior treatments in nonstationary situations. These findings provide evidence that probability-matching strategies are useful for adaptive management. Integr Environ Assess Manag 2020;16:841-852. © 2020 The Authors. Integrated Environmental Assessment and Management published by Wiley Periodicals LLC on behalf of Society of Environmental Toxicology & Chemistry (SETAC).
| S-EPMC7689691 | biostudies-literature
Project description:The present study aimed to investigate the influence of housing conditions on contextual fear memory malleability. Male Wistar rats were housed in enriched, standard, or impoverished conditions after weaning and remained in these conditions throughout the entire experiment. After six weeks into those housing conditions, all animals underwent a 3-day protocol including contextual fear conditioning (day 1), memory reactivation followed by systemic administration of midazolam or vehicle (day 2), and a retention test (day 3). Percentage freezing was used as a behavioral measure of contextual fear. There was no evidence for an effect of housing conditions on the sensitivity of contextual fear memory to amnestic effects of post-reactivation midazolam administration, and no indication for amnestic effects of post-reactivation midazolam overall (including in the standard group). The inability to replicate previous demonstrations of post-reactivation amnesia using the same protocol underscores the subtle nature of post-reactivation pharmacological memory interference. Notably, impoverished housing resulted in a decrease in contextual freezing during contextual fear conditioning, reactivation and retention testing, compared to enriched and standard housing conditions. This observation warrants caution when interpreting the results from experiments regarding effects of housing on fear memory processes, particularly when freezing is used as a measure of fear.
| S-EPMC6314464 | biostudies-literature
Project description:Post-traumatic stress disorder (PTSD) is a common anxiety mental disorder and can be manifested after exposure to a real or perceived life-threatening event. Increased noradrenaline and adrenaline in plasma and urine have been documented in PTSD. Dopamine-β-hydroxylase (DBH) catalyzes the conversion of dopamine to noradrenaline and consequently, DBH inhibition reduces catecholamines. Our aim was to evaluate if nepicastat treatment decreases PTSD signs in an animal model. Wild-type (129x1/SvJ) female mice were submitted to PTSD induction protocol. DBH-inhibitor nepicastat (30 mg/kg) or vehicle (0.2% HPMC) were administered once daily since day 0 until day 7 or 12. The percentage of freezing was calculated on days 0, 1, 2, and 7, and behavioral tests were performed. Quantification of nepicastat in plasma and DBH activity in the adrenal gland was evaluated. Catecholamines were quantified by HPLC with electrochemical detection. mRNA expression of Npas4 and Bdnf in hippocampus was evaluated by qPCR.Mice in the PTSD-group and treated with nepicastat showed a decrease in freezing, and an increase in the time spent and entries in open arms in elevated plus maze test. In mice treated with nepicastat, adrenal gland DBH activity was decreased, and catecholamines were also decreased in plasma and tissues. On day 7, in mice treated with nepicastat, there was an increase of Npas4 and Bdnf mRNA expression in the hippocampus.In conclusion, DBH inhibitor nepicastat has an effect consistent with a decrease in the persistence of traumatic memories and anxiety-like behavior in this PTSD mice model. The disruption of traumatic memories through interference with the formation, consolidation, retrieval, and/or expression processes may be important to decrease PTSD symptoms and signs. The increase in Npas4 and Bdnf mRNA expression in the hippocampus may be important to develop a weaker traumatic contextual memory after nepicastat treatment.
| S-EPMC8498196 | biostudies-literature
Project description:Sequential portfolio selection has attracted increasing interest in the machine learning and quantitative finance communities in recent years. As a mathematical framework for reinforcement learning policies, the stochastic multi-armed bandit problem addresses the primary difficulty in sequential decision-making under uncertainty, namely the exploration versus exploitation dilemma, and therefore provides a natural connection to portfolio selection. In this paper, we incorporate risk awareness into the classic multi-armed bandit setting and introduce an algorithm to construct portfolio. Through filtering assets based on the topological structure of the financial market and combining the optimal multi-armed bandit policy with the minimization of a coherent risk measure, we achieve a balance between risk and return.
| S-EPMC5717697 | biostudies-literature
Project description:Because of the use of radiation in cancer therapy, the risk of nuclear contamination from power plants, military conflicts, and terrorism, there is a compelling scientific and public health interest in the effects of environmental radiation exposure on brain function, in particular hippocampal function and learning and memory. Previous studies have emphasized changes in learning and memory following radiation exposure. These approaches have ignored the question of how radiation exposure might impact recently acquired memories, which might be acquired under traumatic circumstances (cancer treatment, nuclear disaster, etc.). To address the question of how radiation exposure might affect the processing and recall of recently acquired memories, we employed a fear conditioning paradigm wherein animals were trained, and subsequently irradiated (whole-body X-ray irradiation) 24 h later. Animals were given 2 weeks to recover, and were tested for retention and extinction of hippocampus-dependent contextual fear conditioning or hippocampus-independent cued fear conditioning. Exposure to irradiation following training was associated with reduced daily increases in body weights over the 22-days of the study and resulted in greater freezing levels and aberrant extinction 2 weeks later. This was also observed when the intensity of the training protocol was increased. Cued freezing levels and measures of anxiety 2 weeks after training were also higher in irradiated than sham-irradiated mice. In contrast to contextual freezing levels, cued freezing levels were even higher in irradiated mice receiving 5 shocks during training than sham-irradiated mice receiving 10 shocks during training. In addition, the effects of radiation on extinction of contextual fear were more profound than those on the extinction of cued fear. Thus, whole-body irradiation elevates contextual and cued fear memory recall.
| S-EPMC4078460 | biostudies-literature