Browse
Submit Data
Databases
API
Help

Dataset Information

11 Views

0 Connections

0 Citations

0 Reanalyses

0 Downloads

Omics score: 0

An Analysis of the Value of Information When Exploring Stochastic, Discrete Multi-Armed Bandits.

ABSTRACT: In this paper, we propose an information-theoretic exploration strategy for stochastic, discrete multi-armed bandits that achieves optimal regret. Our strategy is based on the value of information criterion. This criterion measures the trade-off between policy information and obtainable rewards. High amounts of policy information are associated with exploration-dominant searches of the space and yield high rewards. Low amounts of policy information favor the exploitation of existing knowledge. Information, in this criterion, is quantified by a parameter that can be varied during search. We demonstrate that a simulated-annealing-like update of this parameter, with a sufficiently fast cooling schedule, leads to a regret that is logarithmic with respect to the number of arm pulls.

SUBMITTER: Sledge IJ

PROVIDER: S-EPMC7512671 | biostudies-literature | 2018 Feb

REPOSITORIES: biostudies-literature

ACCESS DATA

Json Xml

Publications

An Analysis of the Value of Information When Exploring Stochastic, Discrete Multi-Armed Bandits.

Sledge Isaac J IJ Príncipe José C JC

Entropy (Basel, Switzerland) 20180228 3

In this paper, we propose an information-theoretic exploration strategy for stochastic, discrete multi-armed bandits that achieves optimal regret. Our strategy is based on the value of information criterion. This criterion measures the trade-off between policy information and obtainable rewards. High amounts of policy information are associated with exploration-dominant searches of the space and yield high rewards. Low amounts of policy information favor the exploitation of existing knowledge. I ...[more]

PMID: 33265246

Similar Datasets

Anytime Exploration for Multi-armed Bandits using Confidence Information.

Project description:We introduce anytime Explore-m, a pure exploration problem for multi-armed bandits (MAB) that requires making a prediction of the top-m arms at every time step. Anytime Explore-m is more practical than fixed budget or fixed confidence formulations of the top-m problem, since many applications involve a finite, but unpredictable, budget. However, the development and analysis of anytime algorithms present many challenges. We propose AT-LUCB (AnyTime Lower and Upper Confidence Bound), the first nontrivial algorithm that provably solves anytime Explore-m. Our analysis shows that the sample complexity of AT-LUCB is competitive to anytime variants of existing algorithms. Moreover, our empirical evaluation on AT-LUCB shows that AT-LUCB performs as well as or better than state-of-the-art baseline methods for anytime Explore-m.

| S-EPMC5846129 | biostudies-literature

Optimal Query Selection Using Multi-Armed Bandits.

Project description:Query selection for latent variable estimation is conventionally performed by opting for observations with low noise or optimizing information theoretic objectives related to reducing the level of estimated uncertainty based on the current best estimate. In these approaches, typically the system makes a decision by leveraging the current available information about the state. However, trusting the current best estimate results in poor query selection when truth is far from the current estimate, and this negatively impacts the speed and accuracy of the latent variable estimation procedure. We introduce a novel sequential adaptive action value function for query selection using the multi-armed bandit (MAB) framework which allows us to find a tractable solution. For this adaptive-sequential query selection method, we analytically show: (i) performance improvement in the query selection for a dynamical system, (ii) the conditions where the model outperforms competitors. We also present favorable empirical assessments of the performance for this method, compared to alternative methods, both using Monte Carlo simulations and human-in-the-loop experiments with a brain computer interface (BCI) typing system where the language model provides the prior information.

| S-EPMC6777547 | biostudies-literature

An Efficient Algorithm for Deep Stochastic Contextual Bandits.

Project description:In stochastic contextual bandit (SCB) problems, an agent selects an action based on certain observed context to maximize the cumulative reward over iterations. Recently there have been a few studies using a deep neural network (DNN) to predict the expected reward for an action, and the DNN is trained by a stochastic gradient based method. However, convergence analysis has been greatly ignored to examine whether and where these methods converge. In this work, we formulate the SCB that uses a DNN reward function as a non-convex stochastic optimization problem, and design a stage-wise stochastic gradient descent algorithm to optimize the problem and determine the action policy. We prove that with high probability, the action sequence chosen by this algorithm converges to a greedy action policy respecting a local optimal reward function. Extensive experiments have been performed to demonstrate the effectiveness and efficiency of the proposed algorithm on multiple real-world datasets.

| S-EPMC8567845 | biostudies-literature

Exploring the most stable aptamer/target molecule complex by the stochastic tunnelling-basin hopping-discrete molecular dynamics method.

Project description:The stochastic tunnelling-basin hopping-discrete molecular dynamics (STUN-BH-DMD) method was applied to the search for the most stable biomolecular complexes in water by using the MARTINI coarse-grained (CG) model. The epithelial cell adhesion molecule (EpCAM, PDB code: 4MZV) was used as an EpCAM adaptor for an EpA (AptEpA) benchmark target molecule. The effects of two adsorption positions on the EpCAM were analysed, and it is found that the AptEpA adsorption configuration located within the EpCAM pocket-like structure is more stable and the energy barrier is lower due to the interaction with water. By the root mean square deviation (RMSD), the configuration of EpCAM in water is more conservative when the AptEpA binds to EpCAM by attaching to the pocket space of the EpCAM dimer. For AptEpA, the root mean square fluctuation (RMSF) analysis result indicates Nucleobase 1 and Nucleobase 2 display higher flexibility during the CGMD simulation. Finally, from the binding energy contour maps and histogram plots of EpCAM and each AptEpA nucleobase, it is clear that the binding energy adsorbed to the pocket-like structure is more continuous than that energy not adsorbed to the pocket-like structure. This study has proposed a new numerical process for applying the STUN-BH-DMD with the CG model, which can reduce computational details and directly find a more stable AptEpA/EpCAM complex in water.

| S-EPMC8169667 | biostudies-literature

Signals of value drive engagement with multi-round information interventions

Project description:For information interventions to be effective, recipients must first engage with them. We show that engagement with repeated digital information interventions is shaped by subtle and strategically controllable signals of the information’s value. In particular, recipients’ expectations are shaped by signals from the “envelope” that surrounds a message in an information intervention. The envelope conveys clues about the message but does not reveal the message itself. When people expect the message to be valuable, delivering it in a consistent and recognizable envelope over time increases engagement relative to varying the envelope. Conversely, when people expect the message to be of little value, delivering it in a consistent and recognizable envelope decreases engagement relative to varying the envelope. We show this with two field experiments involving massive open online courses and one online survey experiment (all pre-registered, N = 439,150).

| S-EPMC9595537 | biostudies-literature

Exploring Careers in Medicine: Implementation and Perceived Value of a Multi-Specialty Elective.

Project description:IntroductionChoosing a medical specialty is one of the most crucial and difficult decisions made during medical school. Given that specialty exposure is among the most important factors in decision-making, the Careers in Medicine (CiM) multi-specialty elective was designed to provide clerkship students an avenue to explore three or more specialties of interest during a single elective.MethodsA cross-sectional study was conducted at Vanderbilt University School of Medicine using anonymous surveys and de-identified written reflections submitted by students enrolled in the CiM course between August 2015 and June 2018. Data were analyzed using a mixed-methods approach.ResultsThe majority of students reported the elective guided them in ruling out (80%) and ruling in (65%) specialties. About half (51%) of students decided between the procedural versus critical-thinking dichotomy. Finally, 80% of students reported that they would take the course again rather than a focused elective. Major themes identified from student reflections included course attributes, specialty impacts, and student values.DiscussionImplementation of a multi-specialty elective during the clerkship year was an effective way to help students understand their career values, gain early exposure to specialties not featured in core clinical curriculums, and determine future fields of interest.Supplementary informationThe online version contains supplementary material available at 10.1007/s40670-021-01311-0.

| S-EPMC8368157 | biostudies-literature

Patterns of mesenchymal condensation in a multiscale, discrete stochastic model.

Project description:Cells of the embryonic vertebrate limb in high-density culture undergo chondrogenic pattern formation, which results in the production of regularly spaced "islands" of cartilage similar to the cartilage primordia of the developing limb skeleton. The first step in this process, in vitro and in vivo, is the generation of "cell condensations," in which the precartilage cells become more tightly packed at the sites at which cartilage will form. In this paper we describe a discrete, stochastic model for the behavior of limb bud precartilage mesenchymal cells in vitro. The model uses a biologically motivated reaction-diffusion process and cell-matrix adhesion (haptotaxis) as the bases of chondrogenic pattern formation, whereby the biochemically distinct condensing cells, as well as the size, number, and arrangement of the multicellular condensations, are generated in a self-organizing fashion. Improving on an earlier lattice-gas representation of the same process, it is multiscale (i.e., cell and molecular dynamics occur on distinct scales), and the cells are represented as spatially extended objects that can change their shape. The authors calibrate the model using experimental data and study sensitivity to changes in key parameters. The simulations have disclosed two distinct dynamic regimes for pattern self-organization involving transient or stationary inductive patterns of morphogens. The authors discuss these modes of pattern formation in relation to available experimental evidence for the in vitro system, as well as their implications for understanding limb skeletal patterning during embryonic development.

| S-EPMC1857812 | biostudies-literature

Oscillatory regulation of Hes1: Discrete stochastic delay modelling and simulation.

Project description:Discrete stochastic simulations are a powerful tool for understanding the dynamics of chemical kinetics when there are small-to-moderate numbers of certain molecular species. In this paper we introduce delays into the stochastic simulation algorithm, thus mimicking delays associated with transcription and translation. We then show that this process may well explain more faithfully than continuous deterministic models the observed sustained oscillations in expression levels of hes1 mRNA and Hes1 protein.

| S-EPMC1560403 | biostudies-literature

Signal detection models as contextual bandits.

Project description:Signal detection theory (SDT) has been widely applied to identify the optimal discriminative decisions of receivers under uncertainty. However, the approach assumes that decision-makers immediately adopt the appropriate acceptance threshold, even though the optimal response must often be learned. Here we recast the classical normal-normal (and power-law) signal detection model as a contextual multi-armed bandit (CMAB). Thus, rather than starting with complete information, decision-makers must infer how the magnitude of a continuous cue is related to the probability that a signaller is desirable, while simultaneously seeking to exploit the information they acquire. We explain how various CMAB heuristics resolve the trade-off between better estimating the underlying relationship and exploiting it. Next, we determined how naive human volunteers resolve signal detection problems with a continuous cue. As anticipated, a model of choice (accept/reject) that assumed volunteers immediately adopted the SDT-predicted acceptance threshold did not predict volunteer behaviour well. The Softmax rule for solving CMABs, with choices based on a logistic function of the expected payoffs, best explained the decisions of our volunteers but a simple midpoint algorithm also predicted decisions well under some conditions. CMABs offer principled parametric solutions to solving many classical SDT problems when decision-makers start with incomplete information.

| S-EPMC10282591 | biostudies-literature

Value of Information Analytical Methods: Report 2 of the ISPOR Value of Information Analysis Emerging Good Practices Task Force.

Project description:The allocation of healthcare resources among competing priorities requires an assessment of the expected costs and health effects of investing resources in the activities and of the opportunity cost of the expenditure. To date, much effort has been devoted to assessing the expected costs and health effects, but there remains an important need to also reflect the consequences of uncertainty in resource allocation decisions and the value of further research to reduce uncertainty. Decision making with uncertainty may turn out to be suboptimal, resulting in health loss. Consequently, there may be value in reducing uncertainty, through the collection of new evidence, to better inform resource decisions. This value can be quantified using value of information (VOI) analysis. This report from the ISPOR VOI Task Force describes methods for computing 4 VOI measures: the expected value of perfect information, expected value of partial perfect information (EVPPI), expected value of sample information (EVSI), and expected net benefit of sampling (ENBS). Several methods exist for computing EVPPI and EVSI, and this report provides guidance on selecting the most appropriate method based on the features of the decision problem. The report provides a number of recommendations for good practice when planning, undertaking, or reviewing VOI analyses. The software needed to compute VOI is discussed, and areas for future research are highlighted.

| S-EPMC7373630 | biostudies-literature

OmicsDI is part of the ELIXIR infrastructure

OmicsDI is an Elixir interoperability service. Learn more ›

Tweets

OmicsDI Databases

PRIDE
PeptideAtlas
MassIVE
JPOST Repository
Physiome Model Repository

EGA
EVA
ENA
LINCS
PAXDB
Cell Collective

MetaboLights
Metabolomics Workbench
MetabolomeExpress
GNPS
BioModels
FAIRDOMHub

ArrayExpress
dbGaP
ExpressionAtlas
GEO
NODE

Information

Databases
Help
API
Contact us
Code on GitHub
Terms of use
Submit Data