Browse
Submit Data
Databases
API
Help

Dataset Information

11 Views

0 Connections

0 Citations

0 Reanalyses

0 Downloads

Omics score: 0

An Analysis of the Value of Information When Exploring Stochastic, Discrete Multi-Armed Bandits.

ABSTRACT: In this paper, we propose an information-theoretic exploration strategy for stochastic, discrete multi-armed bandits that achieves optimal regret. Our strategy is based on the value of information criterion. This criterion measures the trade-off between policy information and obtainable rewards. High amounts of policy information are associated with exploration-dominant searches of the space and yield high rewards. Low amounts of policy information favor the exploitation of existing knowledge. Information, in this criterion, is quantified by a parameter that can be varied during search. We demonstrate that a simulated-annealing-like update of this parameter, with a sufficiently fast cooling schedule, leads to a regret that is logarithmic with respect to the number of arm pulls.

SUBMITTER: Sledge IJ

PROVIDER: S-EPMC7512671 | biostudies-literature | 2018 Feb

REPOSITORIES: biostudies-literature

ACCESS DATA

Json Xml

Publications

An Analysis of the Value of Information When Exploring Stochastic, Discrete Multi-Armed Bandits.

Sledge Isaac J IJ Príncipe José C JC

Entropy (Basel, Switzerland) 20180228 3

In this paper, we propose an information-theoretic exploration strategy for stochastic, discrete multi-armed bandits that achieves optimal regret. Our strategy is based on the value of information criterion. This criterion measures the trade-off between policy information and obtainable rewards. High amounts of policy information are associated with exploration-dominant searches of the space and yield high rewards. Low amounts of policy information favor the exploitation of existing knowledge. I ...[more]

PMID: 33265246

Similar Datasets

Anytime Exploration for Multi-armed Bandits using Confidence Information.

Project description:We introduce anytime Explore-m, a pure exploration problem for multi-armed bandits (MAB) that requires making a prediction of the top-m arms at every time step. Anytime Explore-m is more practical than fixed budget or fixed confidence formulations of the top-m problem, since many applications involve a finite, but unpredictable, budget. However, the development and analysis of anytime algorithms present many challenges. We propose AT-LUCB (AnyTime Lower and Upper Confidence Bound), the first nontrivial algorithm that provably solves anytime Explore-m. Our analysis shows that the sample complexity of AT-LUCB is competitive to anytime variants of existing algorithms. Moreover, our empirical evaluation on AT-LUCB shows that AT-LUCB performs as well as or better than state-of-the-art baseline methods for anytime Explore-m.

| S-EPMC5846129 | biostudies-literature

Optimal Query Selection Using Multi-Armed Bandits.

Project description:Query selection for latent variable estimation is conventionally performed by opting for observations with low noise or optimizing information theoretic objectives related to reducing the level of estimated uncertainty based on the current best estimate. In these approaches, typically the system makes a decision by leveraging the current available information about the state. However, trusting the current best estimate results in poor query selection when truth is far from the current estimate, and this negatively impacts the speed and accuracy of the latent variable estimation procedure. We introduce a novel sequential adaptive action value function for query selection using the multi-armed bandit (MAB) framework which allows us to find a tractable solution. For this adaptive-sequential query selection method, we analytically show: (i) performance improvement in the query selection for a dynamical system, (ii) the conditions where the model outperforms competitors. We also present favorable empirical assessments of the performance for this method, compared to alternative methods, both using Monte Carlo simulations and human-in-the-loop experiments with a brain computer interface (BCI) typing system where the language model provides the prior information.

| S-EPMC6777547 | biostudies-literature

An Efficient Algorithm for Deep Stochastic Contextual Bandits.

Project description:In stochastic contextual bandit (SCB) problems, an agent selects an action based on certain observed context to maximize the cumulative reward over iterations. Recently there have been a few studies using a deep neural network (DNN) to predict the expected reward for an action, and the DNN is trained by a stochastic gradient based method. However, convergence analysis has been greatly ignored to examine whether and where these methods converge. In this work, we formulate the SCB that uses a DNN reward function as a non-convex stochastic optimization problem, and design a stage-wise stochastic gradient descent algorithm to optimize the problem and determine the action policy. We prove that with high probability, the action sequence chosen by this algorithm converges to a greedy action policy respecting a local optimal reward function. Extensive experiments have been performed to demonstrate the effectiveness and efficiency of the proposed algorithm on multiple real-world datasets.

| S-EPMC8567845 | biostudies-literature

Exploring the most stable aptamer/target molecule complex by the stochastic tunnelling-basin hopping-discrete molecular dynamics method.

Project description:The stochastic tunnelling-basin hopping-discrete molecular dynamics (STUN-BH-DMD) method was applied to the search for the most stable biomolecular complexes in water by using the MARTINI coarse-grained (CG) model. The epithelial cell adhesion molecule (EpCAM, PDB code: 4MZV) was used as an EpCAM adaptor for an EpA (AptEpA) benchmark target molecule. The effects of two adsorption positions on the EpCAM were analysed, and it is found that the AptEpA adsorption configuration located within the EpCAM pocket-like structure is more stable and the energy barrier is lower due to the interaction with water. By the root mean square deviation (RMSD), the configuration of EpCAM in water is more conservative when the AptEpA binds to EpCAM by attaching to the pocket space of the EpCAM dimer. For AptEpA, the root mean square fluctuation (RMSF) analysis result indicates Nucleobase 1 and Nucleobase 2 display higher flexibility during the CGMD simulation. Finally, from the binding energy contour maps and histogram plots of EpCAM and each AptEpA nucleobase, it is clear that the binding energy adsorbed to the pocket-like structure is more continuous than that energy not adsorbed to the pocket-like structure. This study has proposed a new numerical process for applying the STUN-BH-DMD with the CG model, which can reduce computational details and directly find a more stable AptEpA/EpCAM complex in water.

| S-EPMC8169667 | biostudies-literature

Signals of value drive engagement with multi-round information interventions

Project description:For information interventions to be effective, recipients must first engage with them. We show that engagement with repeated digital information interventions is shaped by subtle and strategically controllable signals of the information’s value. In particular, recipients’ expectations are shaped by signals from the “envelope” that surrounds a message in an information intervention. The envelope conveys clues about the message but does not reveal the message itself. When people expect the message to be valuable, delivering it in a consistent and recognizable envelope over time increases engagement relative to varying the envelope. Conversely, when people expect the message to be of little value, delivering it in a consistent and recognizable envelope decreases engagement relative to varying the envelope. We show this with two field experiments involving massive open online courses and one online survey experiment (all pre-registered, N = 439,150).

| S-EPMC9595537 | biostudies-literature

Exploring Careers in Medicine: Implementation and Perceived Value of a Multi-Specialty Elective.

Project description:IntroductionChoosing a medical specialty is one of the most crucial and difficult decisions made during medical school. Given that specialty exposure is among the most important factors in decision-making, the Careers in Medicine (CiM) multi-specialty elective was designed to provide clerkship students an avenue to explore three or more specialties of interest during a single elective.MethodsA cross-sectional study was conducted at Vanderbilt University School of Medicine using anonymous surveys and de-identified written reflections submitted by students enrolled in the CiM course between August 2015 and June 2018. Data were analyzed using a mixed-methods approach.ResultsThe majority of students reported the elective guided them in ruling out (80%) and ruling in (65%) specialties. About half (51%) of students decided between the procedural versus critical-thinking dichotomy. Finally, 80% of students reported that they would take the course again rather than a focused elective. Major themes identified from student reflections included course attributes, specialty impacts, and student values.DiscussionImplementation of a multi-specialty elective during the clerkship year was an effective way to help students understand their career values, gain early exposure to specialties not featured in core clinical curriculums, and determine future fields of interest.Supplementary informationThe online version contains supplementary material available at 10.1007/s40670-021-01311-0.

| S-EPMC8368157 | biostudies-literature

Patterns of mesenchymal condensation in a multiscale, discrete stochastic model.

Project description:Cells of the embryonic vertebrate limb in high-density culture undergo chondrogenic pattern formation, which results in the production of regularly spaced "islands" of cartilage similar to the cartilage primordia of the developing limb skeleton. The first step in this process, in vitro and in vivo, is the generation of "cell condensations," in which the precartilage cells become more tightly packed at the sites at which cartilage will form. In this paper we describe a discrete, stochastic model for the behavior of limb bud precartilage mesenchymal cells in vitro. The model uses a biologically motivated reaction-diffusion process and cell-matrix adhesion (haptotaxis) as the bases of chondrogenic pattern formation, whereby the biochemically distinct condensing cells, as well as the size, number, and arrangement of the multicellular condensations, are generated in a self-organizing fashion. Improving on an earlier lattice-gas representation of the same process, it is multiscale (i.e., cell and molecular dynamics occur on distinct scales), and the cells are represented as spatially extended objects that can change their shape. The authors calibrate the model using experimental data and study sensitivity to changes in key parameters. The simulations have disclosed two distinct dynamic regimes for pattern self-organization involving transient or stationary inductive patterns of morphogens. The authors discuss these modes of pattern formation in relation to available experimental evidence for the in vitro system, as well as their implications for understanding limb skeletal patterning during embryonic development.

| S-EPMC1857812 | biostudies-literature

Oscillatory regulation of Hes1: Discrete stochastic delay modelling and simulation.

Project description:Discrete stochastic simulations are a powerful tool for understanding the dynamics of chemical kinetics when there are small-to-moderate numbers of certain molecular species. In this paper we introduce delays into the stochastic simulation algorithm, thus mimicking delays associated with transcription and translation. We then show that this process may well explain more faithfully than continuous deterministic models the observed sustained oscillations in expression levels of hes1 mRNA and Hes1 protein.

| S-EPMC1560403 | biostudies-literature

The Information Content of Discrete Functions and Their Application in Genetic Data Analysis.

Project description:The complex of central problems in data analysis consists of three components: (1) detecting the dependence of variables using quantitative measures, (2) defining the significance of these dependence measures, and (3) inferring the functional relationships among dependent variables. We have argued previously that an information theory approach allows separation of the detection problem from the inference of functional form problem. We approach here the third component of inferring functional forms based on information encoded in the functions. We present here a direct method for classifying the functional forms of discrete functions of three variables represented in data sets. Discrete variables are frequently encountered in data analysis, both as the result of inherently categorical variables and from the binning of continuous numerical variables into discrete alphabets of values. The fundamental question of how much information is contained in a given function is answered for these discrete functions, and their surprisingly complex relationships are illustrated. The all-important effect of noise on the inference of function classes is found to be highly heterogeneous and reveals some unexpected patterns. We apply this classification approach to an important area of biological data analysis-that of inference of genetic interactions. Genetic analysis provides a rich source of real and complex biological data analysis problems, and our general methods provide an analytical basis and tools for characterizing genetic problems and for analyzing genetic data. We illustrate the functional description and the classes of a number of common genetic interaction modes and also show how different modes vary widely in their sensitivity to noise.

| S-EPMC5729883 | biostudies-other

Signal detection models as contextual bandits.

Project description:Signal detection theory (SDT) has been widely applied to identify the optimal discriminative decisions of receivers under uncertainty. However, the approach assumes that decision-makers immediately adopt the appropriate acceptance threshold, even though the optimal response must often be learned. Here we recast the classical normal-normal (and power-law) signal detection model as a contextual multi-armed bandit (CMAB). Thus, rather than starting with complete information, decision-makers must infer how the magnitude of a continuous cue is related to the probability that a signaller is desirable, while simultaneously seeking to exploit the information they acquire. We explain how various CMAB heuristics resolve the trade-off between better estimating the underlying relationship and exploiting it. Next, we determined how naive human volunteers resolve signal detection problems with a continuous cue. As anticipated, a model of choice (accept/reject) that assumed volunteers immediately adopted the SDT-predicted acceptance threshold did not predict volunteer behaviour well. The Softmax rule for solving CMABs, with choices based on a logistic function of the expected payoffs, best explained the decisions of our volunteers but a simple midpoint algorithm also predicted decisions well under some conditions. CMABs offer principled parametric solutions to solving many classical SDT problems when decision-makers start with incomplete information.

| S-EPMC10282591 | biostudies-literature

OmicsDI is part of the ELIXIR infrastructure

OmicsDI is an Elixir interoperability service. Learn more ›

Tweets

OmicsDI Databases

PRIDE
PeptideAtlas
MassIVE
JPOST Repository
Physiome Model Repository

EGA
EVA
ENA
LINCS
PAXDB
Cell Collective

MetaboLights
Metabolomics Workbench
MetabolomeExpress
GNPS
BioModels
FAIRDOMHub

ArrayExpress
dbGaP
ExpressionAtlas
GEO
NODE

Information

Databases
Help
API
Contact us
Code on GitHub
Terms of use
Submit Data