Dataset Information

Problems in using p-curve analysis and text-mining to detect rate of p-hacking and evidential value.

ABSTRACT: Background. The p-curve is a plot of the distribution of p-values reported in a set of scientific studies. Comparisons between ranges of p-values have been used to evaluate fields of research in terms of the extent to which studies have genuine evidential value, and the extent to which they suffer from bias in the selection of variables and analyses for publication, p-hacking. Methods. p-hacking can take various forms. Here we used R code to simulate the use of ghost variables, where an experimenter gathers data on several dependent variables but reports only those with statistically significant effects. We also examined a text-mined dataset used by Head et al. (2015) and assessed its suitability for investigating p-hacking. Results. We show that when there is ghost p-hacking, the shape of the p-curve depends on whether dependent variables are intercorrelated. For uncorrelated variables, simulated p-hacked data do not give the "p-hacking bump" just below .05 that is regarded as evidence of p-hacking, though there is a negative skew when simulated variables are inter-correlated. The way p-curves vary according to features of underlying data poses problems when automated text mining is used to detect p-values in heterogeneous sets of published papers. Conclusions. The absence of a bump in the p-curve is not indicative of lack of p-hacking. Furthermore, while studies with evidential value will usually generate a right-skewed p-curve, we cannot treat a right-skewed p-curve as an indicator of the extent of evidential value, unless we have a model specific to the type of p-values entered into the analysis. We conclude that it is not feasible to use the p-curve to estimate the extent of p-hacking and evidential value unless there is considerable control over the type of data entered into the analysis. In particular, p-hacking with ghost variables is likely to be missed.

SUBMITTER: Bishop DV

PROVIDER: S-EPMC4768688 | biostudies-literature | 2016

REPOSITORIES: biostudies-literature

ACCESS DATA

Publications

Problems in using p-curve analysis and text-mining to detect rate of p-hacking and evidential value.

Bishop Dorothy V M DV Thompson Paul A PA

PeerJ 20160218

Background. The p-curve is a plot of the distribution of p-values reported in a set of scientific studies. Comparisons between ranges of p-values have been used to evaluate fields of research in terms of the extent to which studies have genuine evidential value, and the extent to which they suffer from bias in the selection of variables and analyses for publication, p-hacking. Methods. p-hacking can take various forms. Here we used R code to simulate the use of ghost variables, where an experime ...[more]

PMID: 26925335

Similar Datasets

Project description:BackgroundToxicogenomics studies often profile gene expression from assays involving multiple doses and time points. The dose- and time-dependent pattern is of great importance to assess toxicity but computational approaches are lacking to effectively utilize this characteristic in toxicity assessment. Topic modeling is a text mining approach, but may be used analogously in toxicogenomics due to the similar data structures between text and gene dysregulation.ResultsTopic modeling was applied to a very large toxicogenomics dataset containing microarray gene expression data from >15,000 samples associated with 131 drugs tested in three different assay platforms (i.e., in vitro assay, in vivo repeated dose study and in vivo single dose experiment) with a design including multiple doses and time points. A set of "topics" which each consist of a set of genes was determined, by which the varying sensitivity of three assay systems was observed. We found that the drug-dependent effect was more pronounced in the two in vivo systems than the in vitro system, while the time-dependent effect was most strongly reflected in the in vitro system followed by the single dose study and lastly the repeated dose experiment. The dose-dependent effect was similar across three assay systems. Although the results indicated a challenge to extrapolate the in vitro results to the in vivo situation, we did notice that, for some drugs but not for all the drugs, the similarity in gene expression patterns was observed across all three assay systems, indicating a possibility of using in vitro systems with careful designs (such as the choice of dose and time point), to replace the in vivo testing strategy. Nonetheless, a potential to replace the repeated dose study by the single-dose short-term methodology was strongly implied.ConclusionsThe study demonstrated that text mining methodologies such as topic modeling provide an alternative method compared to traditional means for data reduction in toxicogenomics, enhancing researchers' capabilities to interpret biological information.

Project description:BACKGROUND:Some medications carry increased risk of patient harm when they are given in error. In incident reports, names of the medications that are involved in errors could be found written both in a specific medication field and/or within the free text description of the incident. Analysing only the names of the medications implicated in a specific unstructured medication field does not give information of the associated factors and risk areas, but when analysing unstructured free text descriptions, the information about the medication involved and associated risk factors may be buried within other non-relevant text. Thus, the aim of this study was to extract medication names most commonly used in free text descriptions of medication administration incident reports to identify terms most frequently associated with risk for each of these medications using text mining. METHOD:Free text descriptions of medication administration incidents (n = 72,390) reported in 2016 to the National Reporting and Learning System for England and Wales were analysed using SAS® Text miner. Analysis included text parsing and filtering free text to identify most commonly mentioned medications, followed by concept linking, and clustering to identify terms associated with commonly mentioned medications and the associated risk areas. RESULTS:The following risk areas related to medications were identified: 1. Allergic reactions to antibacterial drugs, 2. Intravenous administration of antibacterial drugs, 3. Fentanyl patches, 4. Checking and documenting of analgesic doses, 5. Checking doses of anticoagulants, 6. Insulin doses and blood glucose, 7. Administration of intravenous infusions. CONCLUSIONS:Interventions to increase medication administration safety should focus on checking patient allergies and medication doses, especially for intravenous and transdermal medications. High-risk medications include insulin, analgesics, antibacterial drugs, anticoagulants, and potassium chloride. Text mining may be useful for analysing large free text datasets and should be developed further.

Dataset Information

Problems in using p-curve analysis and text-mining to detect rate of p-hacking and evidential value.

Publications

Problems in using p-curve analysis and text-mining to detect rate of p-hacking and evidential value.

Similar Datasets

OmicsDI is part of the ELIXIR infrastructure

Tweets