Dataset Information

Using prediction markets to predict the outcomes in the Defense Advanced Research Projects Agency's next-generation social science programme.

ABSTRACT: There is evidence that prediction markets are useful tools to aggregate information on researchers' beliefs about scientific results including the outcome of replications. In this study, we use prediction markets to forecast the results of novel experimental designs that test established theories. We set up prediction markets for hypotheses tested in the Defense Advanced Research Projects Agency's (DARPA) Next Generation Social Science (NGS2) programme. Researchers were invited to bet on whether 22 hypotheses would be supported or not. We define support as a test result in the same direction as hypothesized, with a Bayes factor of at least 10 (i.e. a likelihood of the observed data being consistent with the tested hypothesis that is at least 10 times greater compared with the null hypothesis). In addition to betting on this binary outcome, we asked participants to bet on the expected effect size (in Cohen's d) for each hypothesis. Our goal was to recruit at least 50 participants that signed up to participate in these markets. While this was the case, only 39 participants ended up actually trading. Participants also completed a survey on both the binary result and the effect size. We find that neither prediction markets nor surveys performed well in predicting outcomes for NGS2.

SUBMITTER: Viganola D

PROVIDER: S-EPMC8278038 | biostudies-literature |

REPOSITORIES: biostudies-literature

ACCESS DATA

Similar Datasets

Project description:BackgroundParents are increasingly searching online for information supported by research but can find it difficult to identify results relevant to their own experiences. More troublingly, a number of studies indicate that parenting information found online often can be misleading or wrong. The goal of the Parenting Science Gang (PSG) project was to use the power of the Internet to help parents ask questions they wanted to have answered by scientific research and to feel confident in assessing research evidence.MethodsBy using Facebook to recruit groups and facilitate interactions, PSG was able to engage fully the target public of parents of young children in the radical co-production of scientific studies, while not creating an undue burden on time or restricting participants due to disability, financial status or location. By giving parents true partnership and control of creation of projects, PSG ensured that the chosen questions were ones that were of most relevance and interest to them.ResultsThis paper presents a summary of eight projects, with three in more detail, designed and implemented by PSG Facebook groups in collaboration with experts. Most projects had health related themes, often prompted by dissatisfaction with treatment of parents by health professionals or by feelings of being marginalised by pregnancy and motherhood, as well as by the lack of evidence for their questions and concerns. The PSG approach meant that these frustrations were channelled into actions. All eight of the PSG groups engaged in meaningful interactions with experts and co-produced studies with the groups defining the questions of interest.ConclusionsThis radically user-led design meant that the PSG staff and the collaborating experts had to live with a high degree of uncertainty. Nevertheless, PSG achieved its goal of academically productive, truly co-produced projects, but as important were the positive effects it had on many of the participants, both parents and experts. At the point of writing this paper, PSG projects have led to outputs including at least eight papers published, in press or in preparation, seven conference presentations, testimony to the Infant Feeding All-Party Parliamentary Group, and with more to come.

Project description:ObjectivesHealthcare budgets are limited, and therefore, research funds should be wisely allocated to ensure high-quality, useful and cost-effective research. We aimed to critically review the criteria considered by major Australian organisations in prioritising and selecting health research projects for funding.MethodsWe reviewed all grant schemes listed on the Australian Competitive Grants Register that were health-related, active in 2017 and with publicly available selection criteria on the funders' websites. Data extracted included scheme name, funding organisation, selection criteria and the relative weight assigned to each criterion. Selection criteria were grouped into five representative domains: relevance, appropriateness, significance, feasibility (including team quality) and cost-effectiveness (ie, value for money).ResultsThirty-six schemes were included from 158 identified. One-half of the schemes were under the National Health and Medical Research Council. The most commonly used criteria were research team quality and capability (94%), research plan clarity (94%), scientific quality (92%) and research impact (92%). Criteria considered less commonly were existing knowledge (22%), fostering collaboration (22%), research environment (19%), value for money (14%), disease burden (8%) and ethical/moral considerations (3%). In terms of representative domains, relevance was considered in 72% of the schemes, appropriateness in 92%, significance in 94%, feasibility in 100% and cost-effectiveness in 17%. The relative weights for the selection criteria varied across schemes with 5%-30% for relevance, 20%-60% for each appropriateness and significance, 20%-75% for feasibility and 15%-33% for cost-effectiveness.ConclusionsIn selecting research projects for funding, Australian research organisations focus largely on research appropriateness, significance and feasibility; however, value for money is most often overlooked. Research funding decisions should include an assessment of value for money in order to maximise return on research investment.

Dataset Information

Using prediction markets to predict the outcomes in the Defense Advanced Research Projects Agency's next-generation social science programme.

Similar Datasets

OmicsDI is part of the ELIXIR infrastructure

Tweets