Dataset Information

Curriculum Learning Strategies for IR : An Empirical Study on Conversation Response Ranking

ABSTRACT: Neural ranking models are traditionally trained on a series of random batches, sampled uniformly from the entire training set. Curriculum learning has recently been shown to improve neural models’ effectiveness by sampling batches non-uniformly, going from easy to difficult instances during training. In the context of neural Information Retrieval (IR) curriculum learning has not been explored yet, and so it remains unclear (1) how to measure the difficulty of training instances and (2) how to transition from easy to difficult instances during training. To address both challenges and determine whether curriculum learning is beneficial for neural ranking models, we need large-scale datasets and a retrieval task that allows us to conduct a wide range of experiments. For this purpose, we resort to the task of conversation response ranking: ranking responses given the conversation history. In order to deal with challenge (1), we explore scoring functions to measure the difficulty of conversations based on different input spaces. To address challenge (2) we evaluate different pacing functions, which determine the velocity in which we go from easy to difficult instances. We find that, overall, by just intelligently sorting the training data (i.e., by performing curriculum learning) we can improve the retrieval effectiveness by up to 2% (The source code is available at https://github.com/Guzpenha/transformers_cl.).

SUBMITTER: Jose J

PROVIDER: S-EPMC7148246 | biostudies-literature | 2020 Mar

REPOSITORIES: biostudies-literature

ACCESS DATA

Similar Datasets

Project description:ObjectiveTo empirically explore the level of agreement of the treatment hierarchies from different ranking metrics in network meta-analysis (NMA) and to investigate how network characteristics influence the agreement.DesignEmpirical evaluation from re-analysis of NMA.Data232 networks of four or more interventions from randomised controlled trials, published between 1999 and 2015.MethodsWe calculated treatment hierarchies from several ranking metrics: relative treatment effects, probability of producing the best value [Formula: see text] and the surface under the cumulative ranking curve (SUCRA). We estimated the level of agreement between the treatment hierarchies using different measures: Kendall's τ and Spearman's ρ correlation; and the Yilmaz [Formula: see text] and Average Overlap, to give more weight to the top of the rankings. Finally, we assessed how the amount of the information present in a network affects the agreement between treatment hierarchies, using the average variance, the relative range of variance and the total sample size over the number of interventions of a network.ResultsOverall, the pairwise agreement was high for all treatment hierarchies obtained by the different ranking metrics. The highest agreement was observed between SUCRA and the relative treatment effect for both correlation and top-weighted measures whose medians were all equal to 1. The agreement between rankings decreased for networks with less precise estimates and the hierarchies obtained from [Formula: see text] appeared to be the most sensitive to large differences in the variance estimates. However, such large differences were rare.ConclusionsDifferent ranking metrics address different treatment hierarchy problems, however they produced similar rankings in the published networks. Researchers reporting NMA results can use the ranking metric they prefer, unless there are imprecise estimates or large imbalances in the variance estimates. In this case treatment hierarchies based on both probabilistic and non-probabilistic ranking metrics should be presented.

Project description:MotivationThe computer-aided design of metabolic intervention strategies has become a key component of an integrated metabolic engineering approach and a broad range of methods and algorithms has been developed for this task. Many of these algorithms enforce coupling of growth with product synthesis and may return thousands of possible intervention strategies from which the most suitable strategy must then be selected.ResultsThis work focuses on how to evaluate and rank, in a meaningful way, a given pool of computed metabolic engineering strategies for growth-coupled product synthesis. Apart from straightforward criteria, such as a preferably small number of necessary interventions, a reasonable growth rate and a high product yield, we present several new criteria useful to pick the most suitable intervention strategy. Among others, we investigate the robustness of the intervention strategies by searching for metabolites that may disrupt growth coupling when accumulated or secreted and by checking whether the interventions interrupt pathways at their origin (preferable) or at downstream steps. We also assess thermodynamic properties of the pathway(s) favored by the intervention strategy. Furthermore, strategies that have a significant overlap with alternative solutions are ranked higher because they provide flexibility in implementation. We also introduce the notion of equivalence classes for grouping intervention strategies with identical solution spaces. Our ranking procedure involves in total ten criteria and we demonstrate its applicability by assessing knockout-based intervention strategies computed in a genome-scale model of E.coli for the growth-coupled synthesis of l-methionine and of the heterologous product 1,4-butanediol.Availability and implementationThe MATLAB scripts that were used to characterize and rank the example intervention strategies are available at http://www2.mpi-magdeburg.mpg.de/projects/cna/etcdownloads.html.Supplementary informationSupplementary data are available at Bioinformatics online.

Dataset Information

Curriculum Learning Strategies for IR : An Empirical Study on Conversation Response Ranking

Similar Datasets

OmicsDI is part of the ELIXIR infrastructure

Tweets