Linear Fitted-Q Iteration with Multiple Reward Functions.
Ontology highlight
ABSTRACT:
SUBMITTER: Lizotte DJ
PROVIDER: S-EPMC3670261 | biostudies-literature | 2012 Nov
REPOSITORIES: biostudies-literature
Publications
Linear Fitted-Q Iteration with Multiple Reward Functions.
Lizotte Daniel J DJ Bowling Michael M Murphy Susan A SA
Journal of machine learning research : JMLR 20121101 Nov
We present a general and detailed development of an algorithm for finite-horizon fitted-Q iteration with an arbitrary number of reward signals and linear value function approximation using an arbitrary number of state features. This includes a detailed treatment of the 3-reward function case using triangulation primitives from computational geometry and a method for identifying globally dominated actions. We also present an example of how our methods can be used to construct a real-world decisio ...[more]
Similar Datasets
Project description:Data on the description of growth of female Boer goats from the Mexican national breeding flock are presented. Goat meat is highly appreciated for the preparation of traditional dishes of Mexican cuisine, and its demand is on the rise. Boer goats are of relatively recent arrival in Mexico and the size of the performance-recorded flock has been increasing steadily in the last ten years. Repeated measures of body weight at different ages from birth to adulthood of Boer goats are scarce. When available, such data can be used to describe the growth pattern and the meat production potential of goat meat breeds such as the Boer. This paper presents data on estimators of growth curve parameters, plots of average predicted growth curves, plots of residuals on age, and data on goodness of fit statistics of ten non-linear functions fitted to describe the growth curve of Boer goats.
| S-EPMC6373209 | biostudies-literature
Project description:BackgroundTransmission-blocking interventions (TBIs) aim to eliminate malaria by reducing transmission of the parasite between the host and the invertebrate vector. TBIs include transmission-blocking drugs and vaccines that, when given to humans, are taken up by mosquitoes and inhibit parasitic development within the vector. Accurate methodologies are key to assess TBI efficacy to ensure that only the most potent candidates progress to expensive and time-consuming clinical trials. Measuring intervention efficacy can be problematic because there is substantial variation in the number of parasites in both the host and vector populations, which can impact transmission even in laboratory settings.MethodsA statistically robust empirical method is introduced for estimating intervention efficacy from standardised population assay experiments. This method will be more reliable than simple summary statistics as it captures changes in parasite density in different life-stages. It also allows efficacy estimates at a finer resolution than previous methods enabling the impact of the intervention over successive generations to be tracked. A major advantage of the new methodology is that it makes no assumptions on the population dynamics of infection. This enables both host-to-vector and vector-to-host transmission to be density-dependent (or other) processes and generates easy-to-understand estimates of intervention efficacy.ResultsThis method increases the precision of intervention efficacy estimates and demonstrates that relying on changes in infection prevalence (the proportion of infected hosts) alone may be insufficient to capture the impact of TBIs, which also suppress parasite density in secondarily infected hosts.ConclusionsThe method indicates that potentially useful, partially effective TBIs may require multiple infection cycles before substantial reductions in prevalence are observed, despite more rapidly suppressing parasite density. Accurate models to quantify efficacy will have important implications for understanding how TBI candidates might perform in field situations and how they should be evaluated in clinical trials.
| S-EPMC5379616 | biostudies-literature
Project description:State-of-the-art protein-ligand docking methods are generally limited by the traditionally low accuracy of their scoring functions, which are used to predict binding affinity and thus vital for discriminating between active and inactive compounds. Despite intensive research over the years, classical scoring functions have reached a plateau in their predictive performance. These assume a predetermined additive functional form for some sophisticated numerical features, and use standard multivariate linear regression (MLR) on experimental data to derive the coefficients.In this study we show that such a simple functional form is detrimental for the prediction performance of a scoring function, and replacing linear regression by machine learning techniques like random forest (RF) can improve prediction performance. We investigate the conditions of applying RF under various contexts and find that given sufficient training samples RF manages to comprehensively capture the non-linearity between structural features and measured binding affinities. Incorporating more structural features and training with more samples can both boost RF performance. In addition, we analyze the importance of structural features to binding affinity prediction using the RF variable importance tool. Lastly, we use Cyscore, a top performing empirical scoring function, as a baseline for comparison study.Machine-learning scoring functions are fundamentally different from classical scoring functions because the former circumvents the fixed functional form relating structural features with binding affinities. RF, but not MLR, can effectively exploit more structural features and more training samples, leading to higher prediction performance. The future availability of more X-ray crystal structures will further widen the performance gap between RF-based and MLR-based scoring functions. This further stresses the importance of substituting RF for MLR in scoring function development.
| S-EPMC4153907 | biostudies-literature
Project description:This study investigated how the experience of different reward distributions would shape the utility functions that can be inferred from economic choice. Despite the generally accepted notion that utility functions are not insensitive to external references, the exact way in which such changes take place remains largely unknown. Here we benefitted from the capacity to engage in thorough and prolonged empirical tests of economic choice by one of our evolutionary cousins, the rhesus macaque. We analyzed data from thousands of binary choices and found that the animals' preferences changed depending on the statistics of rewards experienced in the past (up to weeks) and that these changes could reflect monkeys' adapting their expectations of reward. The utility functions we elicited from their choices stretched and shifted over several months of sequential changes in the mean and range of rewards that the macaques experienced. However, this adaptation was usually incomplete, suggesting that - even after months - past experiences held weight when monkeys' assigned value to future rewards. Rather than having stable and fixed preferences assumed by normative economic models, our results demonstrate that rhesus macaques flexibly shape their preferences around the past and present statistics of their environment. That is, rather than relying on a singular reference-point, reference-dependent preferences are likely to capture a monkey's range of expectations.
| S-EPMC8346953 | biostudies-literature
Project description:Approximately 1% of the open reading frames in the human genome encode proteins that function as DNA or RNA helicases. These enzymes act in all aspects of nucleic acid metabolism where the complementary strands of DNA:DNA or DNA:RNA duplexes require to be transiently opened. However, they perform wider roles in nucleic acid metabolism due to their ability to couple the energy derived from hydrolysis of ATP to their unidirectional translocation along strands of DNARNA. In this way, helicases can displace proteins from DNARNA, drive the migration of DNA junctions (such as the Holliday junction recombination intermediate), or generate superhelical tension in nucleic acid duplexes. Here, we review a subgroup of DNA helicase enzymes, the RecQ family, that has attracted considerable interest in recent years due to their role not only in suppression of genome instability, but also in the avoidance of human disease. We focus particularly on the protein structural motifs and the multiple assembly states that characterize RecQ helicases and discuss novel biophysical techniques to study the different RecQ structures present in solution. We also speculate on the roles of the different domains and oligomeric forms in defining which DNA structures will represent substrates for RecQ helicase-mediated transactions.
| S-EPMC2714954 | biostudies-literature
Project description:BACKGROUND:Multiple hypothesis testing is a major issue in genome-wide association studies (GWAS), which often analyze millions of markers. The permutation test is considered to be the gold standard in multiple testing correction as it accurately takes into account the correlation structure of the genome. Recently, the linear mixed model (LMM) has become the standard practice in GWAS, addressing issues of population structure and insufficient power. However, none of the current multiple testing approaches are applicable to LMM. RESULTS:We were able to estimate per-marker thresholds as accurately as the gold standard approach in real and simulated datasets, while reducing the time required from months to hours. We applied our approach to mouse, yeast, and human datasets to demonstrate the accuracy and efficiency of our approach. CONCLUSIONS:We provide an efficient and accurate multiple testing correction approach for linear mixed models. We further provide an intuition about the relationships between per-marker threshold, genetic relatedness, and heritability, based on our observations in real data.
| S-EPMC4818520 | biostudies-literature
Project description:Since the discovery of the ubiquitin system and the description of its important role in the degradation of proteins, many studies have shown the importance of ubiquitin-specific peptidases (USPs). One special member of this family is the USP18 protein (formerly UBP43). In the past two decades, several functions of USP18 have been discovered: this protein is not only an isopeptidase but also a potent inhibitor of interferon signaling. Therefore, USP18 functions as 'a' maestro of many biological pathways in various cell types. This review outlines multiple functions of USP18 in the regulation of various immunological processes, including pathogen control, cancer development, and autoimmune diseases.
| S-EPMC5260889 | biostudies-literature
Project description:Learning and adaptive behaviour are fundamental biological processes. A key goal in the field of bioengineering is to develop biochemical circuit architectures with the ability to adapt to dynamic chemical environments. Here, we present a novel design for a biomolecular circuit capable of supervised learning of linear functions, using a model based on chemical reactions catalysed by DNAzymes. To achieve this, we propose a novel mechanism of maintaining and modifying internal state in biochemical systems, thereby advancing the state of the art in biomolecular circuit architecture. We use simulations to demonstrate that the circuit is capable of learning behaviour and assess its asymptotic learning performance, scalability and robustness to noise. Such circuits show great potential for building autonomous in vivo nanomedical devices. While such a biochemical system can tell us a great deal about the fundamentals of learning in living systems and may have broad applications in biomedicine (e.g. autonomous and adaptive drugs), it also offers some intriguing challenges and surprising behaviours from a machine learning perspective.
| S-EPMC4223911 | biostudies-literature
Project description:Key messageA new R-software procedure for fixed/random Diallel models was developed. We eased the diallel schemes approach by considering them as specific cases with different parameterisations of a general linear model. Diallel experiments are based on a set of possible crosses between some homozygous (inbred) lines. For these experiments, six main diallel models are available in literature, to quantify genetic effects, such as general combining ability (GCA), specific combining ability (SCA), reciprocal (maternal) effects and heterosis. Those models tend to be presented as separate entities, to be fitted by using specialised software. In this manuscript, we reinforce the idea that diallel models should be better regarded as specific cases (different parameterisations) of a general linear model and might be fitted with general purpose software facilities, as used for all other types of linear models. We start from the estimation of fixed genetical effects within the R environment and try to bridge the gap between diallel models, linear models and ordinary least squares estimation (OLS). First, we review the main diallel models in literature. Second, we build a set of tools to enable geneticists, plant/animal breeders and students to fit diallel models by using the most widely known R functions for OLS fitting, i.e. the 'lm()' function and related methods. Here, we give three examples to show how diallel models can be built by using the typical process of GLMs and fitted, inspected and processed as all other types of linear models in R. Finally, we give a fourth example to show how our tools can be also used to fit random/mixed effect diallel models in the Bayesian framework.
| S-EPMC7843492 | biostudies-literature
Project description:Honey bees (genus Apis) can communicate the approximate location of a resource to their nestmates via the waggle dance. The distance to a goal is encoded by the duration of the waggle phase of the dance, but the precise shape of this distance-duration relationship is ambiguous: earlier studies (before the 1990s) proposed that it is non-linear, with the increase in waggle duration flattening with distance, while more recent studies suggested that it follows a simple linear function (i.e. a straight line). Strikingly, authors of earlier studies trained bees to much longer distances than authors of more recent studies, but unfortunately they usually measured the duration of dance circuits (waggle phase plus return phase of the dance), which is only a correlate of the bees' distance signal. We trained honey bees (A. mellifera carnica) to visit sugar feeders over a relatively long array of distances between 0.1 and 1.7 km from the hive and measured the duration of both the waggle phase and the return phase of their dances from video recordings. The distance-related increase in waggle duration was better described by a non-linear model with a decreasing slope than by a simple linear model. The relationship was equally well captured by a model with two linear segments separated at a "break-point" at 1 km distance. In turn, the relationship between return phase duration and distance was sufficiently well described by a simple linear model. The data suggest that honey bees process flight distance differently before and beyond a certain threshold distance. While the physiological and evolutionary causes of this behavior remain to be explored, our results can be applied to improve the estimation of honey bee foraging distances based on the decoding of waggle dances.
| S-EPMC8029670 | biostudies-literature