Solving the RNA design problem with reinforcement learning.
Ontology highlight
ABSTRACT:
SUBMITTER: Eastman P
PROVIDER: S-EPMC6029810 | biostudies-literature | 2018 Jun
REPOSITORIES: biostudies-literature
Publications
Solving the RNA design problem with reinforcement learning.
Eastman Peter P Shi Jade J Ramsundar Bharath B Pande Vijay S VS
PLoS computational biology 20180621 6
We use reinforcement learning to train an agent for computational RNA design: given a target secondary structure, design a sequence that folds to that structure in silico. Our agent uses a novel graph convolutional architecture allowing a single model to be applied to arbitrary target structures of any length. After training it on randomly generated targets, we test it on the Eterna100 benchmark and find it outperforms all previous algorithms. Analysis of its solutions shows it has successfully ...[more]
Similar Datasets
Project description:Reinforcement learning is generally accepted to be an appropriate and successful method to learn robot control. Symbolic action planning is useful to resolve causal dependencies and to break a causally complex problem down into a sequence of simpler high-level actions. A problem with the integration of both approaches is that action planning is based on discrete high-level action- and state spaces, whereas reinforcement learning is usually driven by a continuous reward function. Recent advances in model-free reinforcement learning, specifically, universal value function approximators and hindsight experience replay, have focused on goal-independent methods based on sparse rewards that are only given at the end of a rollout, and only if the goal has been fully achieved. In this article, we build on these novel methods to facilitate the integration of action planning with model-free reinforcement learning. Specifically, the paper demonstrates how the reward-sparsity can serve as a bridge between the high-level and low-level state- and action spaces. As a result, we demonstrate that the integrated method is able to solve robotic tasks that involve non-trivial causal dependencies under noisy conditions, exploiting both data and knowledge.
| S-EPMC7805615 | biostudies-literature
Project description:Developing methods for improving student learning is a long-standing goal in undergraduate science education. However, the extent to which students working on problems in small groups versus individually results in improved learning among undergraduate science students has not been evaluated in a randomized controlled trial. We have performed such a trial with 80 students in an undergraduate biochemistry class, in which students were randomized to either learning in groups or learning individually. All students participated in the same class, which consisted of a lecture with periodic breaks for students to solve problems using an audience response system. Students in the individual learning condition answered these questions on their own, but students in the group-based learning condition answered these questions in an assigned group of four students. At the end of the class, all students then took the same exam as individuals. The exam had two types of questions-recall questions, in which students had to simply recall information provided to them, and predict questions, in which students had to apply their new knowledge to a new context. Students in the individual and group-based learning conditions performed similarly well on recall questions. However, students who had been in the group-based learning condition performed significantly better as individuals on the predict questions. This suggests that learning in groups may be more effective than individual learning for undergraduate science students, particularly for applying their knowledge to new contexts; this highlights the potential need for pedagogical approaches in undergraduate science courses that incorporate learning in groups.
| S-EPMC5492254 | biostudies-other
Project description:E-learning systems are capable of providing more adaptive and efficient learning experiences for learners than traditional classroom settings. A key component of such systems is the learning policy. The learning policy is an algorithm that designs the learning paths or rather it selects learning materials for learners based on information such as the learners' current progresses and skills, learning material contents. In this article, the authors address the problem of finding the optimal learning policy. To this end, a model for learners' hierarchical skills in the E-learning system is first developed. Based on the hierarchical skill model and the classical cognitive diagnosis model, a framework to model various mastery levels related to hierarchical skills is further developed. The optimal learning path in consideration of the hierarchical structure of skills is found by applying a model-free reinforcement learning method, which does not require any assumption about learners' learning transition processes. The effectiveness of the proposed framework is demonstrated via simulation studies.
| S-EPMC7711250 | biostudies-literature
Project description:Engineered microbial cells present a sustainable alternative to fossil-based synthesis of chemicals and fuels. Cellular synthesis routes are readily assembled and introduced into microbial strains using state-of-the-art synthetic biology tools. However, the optimization of the strains required to reach industrially feasible production levels is far less efficient. It typically relies on trial-and-error leading into high uncertainty in total duration and cost. New techniques that can cope with the complexity and limited mechanistic knowledge of the cellular regulation are called for guiding the strain optimization. In this paper, we put forward a multi-agent reinforcement learning (MARL) approach that learns from experiments to tune the metabolic enzyme levels so that the production is improved. Our method is model-free and does not assume prior knowledge of the microbe's metabolic network or its regulation. The multi-agent approach is well-suited to make use of parallel experiments such as multi-well plates commonly used for screening microbial strains. We demonstrate the method's capabilities using the genome-scale kinetic model of Escherichia coli, k-ecoli457, as a surrogate for an in vivo cell behaviour in cultivation experiments. We investigate the method's performance relevant for practical applicability in strain engineering i.e. the speed of convergence towards the optimum response, noise tolerance, and the statistical stability of the solutions found. We further evaluate the proposed MARL approach in improving L-tryptophan production by yeast Saccharomyces cerevisiae, using publicly available experimental data on the performance of a combinatorial strain library. Overall, our results show that multi-agent reinforcement learning is a promising approach for guiding the strain optimization beyond mechanistic knowledge, with the goal of faster and more reliably obtaining industrially attractive production levels.
| S-EPMC9200333 | biostudies-literature
Project description:Small group work embraces independent study and interactive learning, which enhance knowledge acquisition and skills. Self-directed learning (SDL) and problem-solving (PS) are essential skills in the development of the nursing profession. During the coronavirus pandemic, virtual learning was indispensable. However, little is known about how students develop SDL and PS abilities through online learning through group work. To evaluate the effects of the online intervention on SDL and PS abilities through interactive group work. A randomised waitlist-control trial was carried out. A structured intervention using problem-based learning (PBL) as a guideline was used to direct student learning in small group work. Assessments were scheduled at Time 0 (baseline), Time 1 (8th week), Time 2 (16th week), and Time 3 (28th week). The mean student age was 21.45 (SD = 0.86). About 78% of students were female. There was no significant difference in demographic characteristics and analysis at the baseline. Students in the intervention group reported greater improvement in the SDLRS and PSI at the 8th week, whereas those in the waitlist control group reported greater improvement in the SDLRS and PSI at the 16th week. Sustained effects in the SDLRS and PSI were observed in both the intervention and waitlist control groups at the 16th and 28th weeks, respectively. A repeated-measure analysis was performed to compare the SDLRS and PSI in different periods and revealed statistically significant results (p < 0.001) in all subscales of SDLRS and PSI in the four study periods. The guidelines appear to be an effective treatment for SDL and PS ability enhancement with sustainable effects through interactive group work. The guidelines with explicit instructions and learning objectives provide directions and guidance to students to learn more effectively. The educator plays a vital role in facilitating the students' SDL and PS ability improvement.
| S-EPMC8775567 | biostudies-literature
Project description:ObjectivesNear misses happen more frequently than actual errors, and highlight system vulnerabilities without causing any harm, thus provide a safe space for organizational learning. Second-order problem solving behavior offers a new perspective to better understand how nurses promote learning from near misses to improve organizational outcomes. This study aimed to explore frontline nurses' perspectives on using second-order problem solving behavior in learning from near misses to improve patient safety.MethodsA qualitative exploratory study design was employed. This study was conducted in three tertiary hospitals in east China from June to November 2015. Purposive sampling was used to recruit 19 frontline nurses. Semi-structured interviews and a qualitative directed content analysis was undertaken using Crossan's 4I Framework of Organizational Learning as a coding framework.ResultsSecond-order problem solving behavior, based on the 4I Framework of Organizational Learning, was referred to as being a leader in exposing near misses, pushing forward the cause analysis within limited capacity, balancing the active and passive role during improvement project, and promoting the continuous improvement with passion while feeling low-powered.Conclusions4I Framework of Organizational Learning can be an underlying guide to enrich frontline nurses' role in promoting organizations to learn from near misses. In this study, nurses displayed their pivotal role in organizational learning from near misses by using second-order problem solving. However, additional knowledge, skills, and support are needed to maximize the application of second-order problem solving behavior when near misses are recognized.
| S-EPMC8488812 | biostudies-literature
Project description:Diet induces parallel changes to the gut microbiota and problem solving performance in a wild bird
| PRJEB40949 | ENA
Project description:In this paper we argue that a synthesis of findings across the various sub-areas of research in complex problem solving and consequently progress in theory building is hampered by an insufficient differentiation of complexity and difficulty. In the proposed framework of person, task, and situation (PTS), complexity is conceptualized as a quality that is determined by the cognitive demands that the characteristics of the task and the situation impose. Difficulty represents the quantifiable level of a person's success in dealing with such demands. We use the well-documented "semantic effect" as an exemplar for testing some of the conceptual assumptions derived from the PTS framework. We demonstrate how a differentiation between complexity and difficulty can help take beyond a potentially too narrowly defined psychometric perspective and subsequently gain a better understanding of the cognitive mechanisms behind this effect. In an empirical study a total of 240 university students were randomly allocated to one of four conditions. The four conditions resulted from contrasting the semanticity level of the variable labels used in the CPS system (high vs. low) and two instruction conditions for how to explore the CPS system's causal structure (starting with the assumption that all relationships between variables existed vs. starting with the assumption that none of the relationships existed). The variation in the instruction aimed at inducing knowledge acquisition processes of either (1) systematic elimination of presumptions, or (2) systematic compilation of a mental representation of the causal structure underpinning the system. Results indicate that (a) it is more complex to adopt a "blank slate" perspective under high semanticity as it requires processes of inhibiting prior assumptions, and (b) it seems more difficult to employ a systematic heuristic when testing against presumptions. In combination, situational characteristics, such as the semanticity of variable labels, have the potential to trigger qualitatively different tasks. Failing to differentiate between 'task' and 'situation' as independent sources of complexity and treating complexity and difficulty synonymously threaten the validity of performance scores obtained in CPS research.
| S-EPMC5641385 | biostudies-other
Project description:Reinforcement learning (RL) is a powerful and flexible paradigm for searching for solutions in high-dimensional action spaces. However, bridging the gap between playing computer games with thousands of simulated episodes and solving real scientific problems with complex and involved environments (up to actual laboratory experiments) requires improvements in terms of sample efficiency to make the most of expensive information. The discovery of new drugs is a major commercial application of RL, motivated by the very large nature of the chemical space and the need to perform multiparameter optimization (MPO) across different properties. In silico methods, such as virtual library screening (VS) and de novo molecular generation with RL, show great promise in accelerating this search. However, incorporation of increasingly complex computational models in these workflows requires increasing sample efficiency. Here, we introduce an active learning system linked with an RL model (RL-AL) for molecular design, which aims to improve the sample-efficiency of the optimization process. We identity and characterize unique challenges combining RL and AL, investigate the interplay between the systems, and develop a novel AL approach to solve the MPO problem. Our approach greatly expedites the search for novel solutions relative to baseline-RL for simple ligand- and structure-based oracle functions, with a 5-66-fold increase in hits generated for a fixed oracle budget and a 4-64-fold reduction in computational time to find a specific number of hits. Furthermore, compounds discovered through RL-AL display substantial enrichment of a multi-parameter scoring objective, indicating superior efficacy in curating high-scoring compounds, without a reduction in output diversity. This significant acceleration improves the feasibility of oracle functions that have largely been overlooked in RL due to high computational costs, for example free energy perturbation methods, and in principle is applicable to any RL domain.
| S-EPMC10935729 | biostudies-literature
Project description:According to the restructuring hypothesis, insight problem solving typically progresses through consecutive stages of search, impasse, insight, and search again for someone, who solves the task. The order of these stages was determined through self-reports of problem solvers and has never been verified behaviorally. We asked whether individual analysis of problem solving attempts of participants revealed the same order of problem solving stages as defined by the theory and whether their subjective feelings corresponded to the problem solving stages they were in. Our participants tried to solve the Five-Square problem in an online task, while we recorded the time and trajectory of their stick movements. After the task they were asked about their feelings related to insight and some of them also had the possibility of reporting impasse while working on the task. We found that the majority of participants did not follow the classic four-stage model of insight, but had more complex sequences of problem solving stages, with search and impasse recurring several times. This means that the classic four-stage model is not sufficient to describe variability on the individual level. We revised the classic model and we provide a new model that can generate all sequences found. Solvers reported insight more often than non-solvers and non-solvers reported impasse more often than solvers, as expected; but participants did not report impasse more often during behaviorally defined impasse stages than during other stages. This shows that impasse reports might be unreliable indicators of impasse. Our study highlights the importance of individual analysis of problem solving behavior to verify insight theory.
| S-EPMC4523725 | biostudies-literature