Dataset Information

Difficulty in detecting discrepancies in a clinical trial report: 260-reader evaluation.

ABSTRACT: Scientific literature can contain errors. Discrepancies, defined as two or more statements or results that cannot both be true, may be a signal of problems with a trial report. In this study, we report how many discrepancies are detected by a large panel of readers examining a trial report containing a large number of discrepancies.We approached a convenience sample of 343 journal readers in seven countries, and invited them in person to participate in a study. They were asked to examine the tables and figures of one published article for discrepancies. 260 participants agreed, ranging from medical students to professors. The discrepancies they identified were tabulated and counted. There were 39 different discrepancies identified. We evaluated the probability of discrepancy identification, and whether more time spent or greater participant experience as academic authors improved the ability to detect discrepancies.Overall, 95.3% of discrepancies were missed. Most participants (62%) were unable to find any discrepancies. Only 11.5% noticed more than 10% of the discrepancies. More discrepancies were noted by participants who spent more time on the task (Spearman's ??=?0.22, P?

SUBMITTER: Cole GD

PROVIDER: S-EPMC4521134 | biostudies-literature | 2015 Jun

REPOSITORIES: biostudies-literature

ACCESS DATA

Publications

Difficulty in detecting discrepancies in a clinical trial report: 260-reader evaluation.

Cole Graham D GD Shun-Shin Matthew J MJ Nowbar Alexandra N AN Buell Kevin G KG Al-Mayahi Faisal F Zargaran David D Mahmood Saliha S Singh Bharpoor B Mielewczik Michael M Francis Darrel P DP

International journal of epidemiology 20150601 3

<h4>Background</h4>Scientific literature can contain errors. Discrepancies, defined as two or more statements or results that cannot both be true, may be a signal of problems with a trial report. In this study, we report how many discrepancies are detected by a large panel of readers examining a trial report containing a large number of discrepancies.<h4>Methods</h4>We approached a convenience sample of 343 journal readers in seven countries, and invited them in person to participate in a study. ...[more]

PMID: 26174517

Similar Datasets

Project description:Mixed methods are commonly used in health services research; however, data are not often integrated to explore complementarity of findings. A triangulation protocol is one approach to integrating such data. A retrospective triangulation protocol was carried out on mixed methods data collected as part of a process evaluation of a trial. The multi-country randomised controlled trial found that a web-based training in communication skills (including use of a patient booklet) and the use of a C-reactive protein (CRP) point-of-care test decreased antibiotic prescribing by general practitioners (GPs) for acute cough. The process evaluation investigated GPs' and patients' experiences of taking part in the trial.Three analysts independently compared findings across four data sets: qualitative data collected view semi-structured interviews with (1) 62 patients and (2) 66 GPs and quantitative data collected via questionnaires with (3) 2886 patients and (4) 346 GPs. Pairwise comparisons were made between data sets and were categorised as agreement, partial agreement, dissonance or silence.Three instances of dissonance occurred in 39 independent findings. GPs and patients reported different views on the use of a CRP test. GPs felt that the test was useful in convincing patients to accept a no-antibiotic decision, but patient data suggested that this was unnecessary if a full explanation was given. Whilst qualitative data indicated all patients were generally satisfied with their consultation, quantitative data indicated highest levels of satisfaction for those receiving a detailed explanation from their GP with a booklet giving advice on self-care. Both qualitative and quantitative data sets indicated higher patient enablement for those in the communication groups who had received a booklet.Use of CRP tests does not appear to engage patients or influence illness perceptions and its effect is more centred on changing clinician behaviour. Communication skills and the patient booklet were relevant and useful for all patients and associated with increased patient satisfaction. A triangulation protocol to integrate qualitative and quantitative data can reveal findings that need further interpretation and also highlight areas of dissonance that lead to a deeper insight than separate analyses.

Project description:ObjectivesTo compare the frequency of discrepancies in retracted reports of clinical trials with those in adjacent unretracted reports in the same journal.DesignBlinded case-control study.SettingJournals in PubMed.Population50 manuscripts, classified on PubMed as retracted clinical trials, paired with 50 adjacent unretracted manuscripts from the same journals. Reports were randomly selected from PubMed in December 2012, with no restriction on publication date. Controls were the preceding unretracted clinical trial published in the same journal. All traces of retraction were removed. Three scientists, blinded to the retraction status of individual reports, reviewed all 100 trial reports for discrepancies. Discrepancies were pooled and cross checked before being counted into prespecified categories. Only then was the retraction status unblinded for analysis.Main outcome measureTotal number of discrepancies (defined as mathematically or logically contradictory statements) in each clinical trial report.ResultsOf 479 discrepancies found in the 100 trial reports, 348 were in the 50 retracted reports and 131 in the 50 unretracted reports. On average, individual retracted reports had a greater number of discrepancies than unretracted reports (median 4 (interquartile range 2-8.75) v 0 (0-5); P<0.001). Papers with a discrepancy were significantly more likely to be retracted than those without a discrepancy (odds ratio 5.7 (95% confidence interval 2.2 to 14.5); P<0.001). In particular, three types of discrepancy arose significantly more frequently in retracted than unretracted reports: factual discrepancies (P=0.002), arithmetical errors (P=0.01), and missed P values (P=0.02). Results from a retrospective analysis indicated that citations and journal impact factor were unlikely to affect the result.ConclusionsDiscrepancies in published trial reports should no longer be assumed to be unimportant. Scientists, blinded to retraction status and with no specialist skill in the field, identify significantly more discrepancies in retracted than unretracted reports of clinical trials. Discrepancies could be an early and accessible signal of unreliability in clinical trial reports.

Dataset Information

Difficulty in detecting discrepancies in a clinical trial report: 260-reader evaluation.

Publications

Difficulty in detecting discrepancies in a clinical trial report: 260-reader evaluation.

Similar Datasets

OmicsDI is part of the ELIXIR infrastructure

Tweets