Dataset Information

Using prediction markets to estimate the reproducibility of scientific research.

ABSTRACT: Concerns about a lack of reproducibility of statistically significant results have recently been raised in many fields, and it has been argued that this lack comes at substantial economic costs. We here report the results from prediction markets set up to quantify the reproducibility of 44 studies published in prominent psychology journals and replicated in the Reproducibility Project: Psychology. The prediction markets predict the outcomes of the replications well and outperform a survey of market participants' individual forecasts. This shows that prediction markets are a promising tool for assessing the reproducibility of published scientific results. The prediction markets also allow us to estimate probabilities for the hypotheses being true at different testing stages, which provides valuable information regarding the temporal dynamics of scientific discovery. We find that the hypotheses being tested in psychology typically have low prior probabilities of being true (median, 9%) and that a "statistically significant" finding needs to be confirmed in a well-powered replication to have a high probability of being true. We argue that prediction markets could be used to obtain speedy information about reproducibility at low cost and could potentially even be used to determine which studies to replicate to optimally allocate limited resources into replications.

SUBMITTER: Dreber A

PROVIDER: S-EPMC4687569 | biostudies-other | 2015 Dec

REPOSITORIES: biostudies-other

ACCESS DATA

Similar Datasets

Project description:To evaluate and determine the value of monitoring models developed by the Mahidol Oxford Tropical Research Unit and the East African Consortium for Clinical Research, consider how this can be measured and explore monitors' and investigators' experiences of and views about the nature, purpose and practice of monitoring.A case study approach was used within the context of participatory action research because one of the aims was to guide and improve practice. 34 interviews, five focus groups and observations of monitoring practice were conducted.Fieldwork occurred in the places where the monitoring models are coordinated and applied in Thailand, Cambodia, Uganda and Kenya. Participants included those coordinating the monitoring schemes, monitors, senior investigators and research staff.Transcribed textual data from field notes, interviews and focus groups was imported into a qualitative data software program (NVIVO V. 10) and analysed inductively and thematically by a qualitative researcher. The initial coding framework was reviewed internally and two main categories emerged from the subsequent interrogation of the data.The categories that were identified related to the conceptual framing and nature of monitoring, and the practice of monitoring, including relational factors. Particular emphasis was given to the value of a scientific and cooperative style of monitoring as a means of enhancing data quality, trust and transparency. In terms of practice the primary purpose of monitoring was defined as improving the conduct of health research and increasing the capacity of researchers and trial sites.The models studied utilise internal and network wide expertise to improve the ethics and quality of clinical research. They demonstrate how monitoring can be a scientific and constructive exercise rather than a threatening process. The value of cooperative relations needs to be given more emphasis in monitoring activities, which seek to ensure that research protects human rights and produces reliable data.

Project description:BackgroundArtificial intelligence (AI) has many applications in various aspects of our daily life, including health, criminal, education, civil, business, and liability law. One aspect of AI that has gained significant attention is natural language processing (NLP), which refers to the ability of computers to understand and generate human language.ObjectiveThis study aims to examine the potential for, and concerns of, using AI in scientific research. For this purpose, high-impact research articles were generated by analyzing the quality of reports generated by ChatGPT and assessing the application's impact on the research framework, data analysis, and the literature review. The study also explored concerns around ownership and the integrity of research when using AI-generated text.MethodsA total of 4 articles were generated using ChatGPT, and thereafter evaluated by 23 reviewers. The researchers developed an evaluation form to assess the quality of the articles generated. Additionally, 50 abstracts were generated using ChatGPT and their quality was evaluated. The data were subjected to ANOVA and thematic analysis to analyze the qualitative data provided by the reviewers.ResultsWhen using detailed prompts and providing the context of the study, ChatGPT would generate high-quality research that could be published in high-impact journals. However, ChatGPT had a minor impact on developing the research framework and data analysis. The primary area needing improvement was the development of the literature review. Moreover, reviewers expressed concerns around ownership and the integrity of the research when using AI-generated text. Nonetheless, ChatGPT has a strong potential to increase human productivity in research and can be used in academic writing.ConclusionsAI-generated text has the potential to improve the quality of high-impact research articles. The findings of this study suggest that decision makers and researchers should focus more on the methodology part of the research, which includes research design, developing research tools, and analyzing data in depth, to draw strong theoretical and practical implications, thereby establishing a revolution in scientific research in the era of AI. The practical implications of this study can be used in different fields such as medical education to deliver materials to develop the basic competencies for both medicine students and faculty members.

Dataset Information

Using prediction markets to estimate the reproducibility of scientific research.

Similar Datasets

OmicsDI is part of the ELIXIR infrastructure

Tweets