Dataset Information

The ability of different peer review procedures to flag problematic publications.

ABSTRACT: There is a mounting worry about erroneous and outright fraudulent research that gets published in the scientific literature. Although peer review's ability to filter out such publications is contentious, several peer review innovations attempt to do just that. However, there is very little systematic evidence documenting the ability of different review procedures to flag problematic publications. In this article, we use survey data on peer review in a wide range of journals to compare the retraction rates of specific review procedures, using the Retraction Watch database. We were able to identify which peer review procedures were used since 2000 for 361 journals, publishing a total of 833,172 articles, of which 670 were retracted. After addressing the dual character of retractions, signalling both a failure to identify problems prior to publication, but also the willingness to correct mistakes, we empirically assess review procedures. With considerable conceptual caveats, we were able to identify peer review procedures that seem able to detect problematic research better than others. Results were verified for disciplinary differences and variation between reasons for retraction. This leads to informed recommendations for journal editors about strengths and weaknesses of specific peer review procedures, allowing them to select review procedures that address issues most relevant to their field.

SUBMITTER: Horbach SPJM

PROVIDER: S-EPMC6404393 | biostudies-literature | 2019

REPOSITORIES: biostudies-literature

ACCESS DATA

Publications

The ability of different peer review procedures to flag problematic publications.

Horbach S P J M SPJM Halffman W W

Scientometrics 20181129 1

There is a mounting worry about erroneous and outright fraudulent research that gets published in the scientific literature. Although peer review's ability to filter out such publications is contentious, several peer review innovations attempt to do just that. However, there is very little systematic evidence documenting the ability of different review procedures to flag problematic publications. In this article, we use survey data on peer review in a wide range of journals to compare the retrac ...[more]

PMID: 30930504

Similar Datasets

Project description:ObjectiveChildren with attention deficit hyperactivity disorder (ADHD) experience many peer interaction problems and are at risk of peer rejection and victimisation. Although many studies have investigated problematic peer functioning in children with ADHD, this research has predominantly focused on boys and studies investigating girls are scant. Those studies that did examine girls, often used a male comparison sample, disregarding the inherent gender differences between girls and boys. Previous studies have highlighted this limitation and recommended the need for comparisons between ADHD females and typical females, in order to elucidate the picture of female ADHD with regards to problematic peer functioning. The aim of this literature review was to gain insight into peer functioning difficulties in school-aged girls with ADHD.MethodsPsychINFO, PubMed, and Web of Knowledge were searched for relevant literature comparing school-aged girls with ADHD to typically developing girls (TDs) in relation to peer functioning. The peer relationship domains were grouped into 'friendship', 'peer status', 'social skills/competence', and 'peer victimisation and bullying'. In total, thirteen studies were included in the review.ResultsAll of the thirteen studies included reported that girls with ADHD, compared to TD girls, demonstrated increased difficulties in the domains of friendship, peer interaction, social skills and functioning, peer victimization and externalising behaviour. Studies consistently showed small to medium effects for lower rates of friendship participation and stability in girls with ADHD relative to TD girls. Higher levels of peer rejection with small to large effect sizes were reported in all studies, which were predicted by girls' conduct problems. Peer rejection in turn predicted poor social adjustment and a host of problem behaviours. Very high levels of peer victimisation were present in girls with ADHD with large effect sizes. Further, very high levels of social impairment and social skills deficits, with large effect sizes, were found across all studies. Levels of pro-social behaviour varied across studies, but were mostly lower in girls with ADHD, with small to large effect sizes. Overall, social disability was significantly higher among girls with ADHD than among TD girls.ConclusionCongruous evidence was found for peer functioning difficulties in the peer relationship domains of friendship, peer status, social skills/competence, and peer victimisation and bullying in girls with ADHD.

Project description:BackgroundThere is significant variability in the performance and outcomes of invasive medical procedures such as percutaneous coronary intervention, endoscopy, and bronchoscopy. Peer evaluation is a common mechanism for assessment of clinician performance and care quality, and may be ideally suited for the evaluation of medical procedures. We therefore sought to perform a systematic review to identify and characterize peer evaluation tools for practicing clinicians, assess evidence supporting the validity of peer evaluation, and describe best practices of peer evaluation programs across multiple invasive medical procedures.MethodsA systematic search of Medline and Embase (through September 7, 2021) was conducted to identify studies of peer evaluation and feedback relating to procedures in the field of internal medicine and related subspecialties. The methodological quality of the studies was assessed. Data were extracted on peer evaluation methods, feedback structures, and the validity and reproducibility of peer evaluations, including inter-observer agreement and associations with other quality measures when available.ResultsOf 2,135 retrieved references, 32 studies met inclusion criteria. Of these, 21 were from the field of gastroenterology, 5 from cardiology, 3 from pulmonology, and 3 from interventional radiology. Overall, 22 studies described the development or testing of peer scoring systems and 18 reported inter-observer agreement, which was good or excellent in all but 2 studies. Only 4 studies, all from gastroenterology, tested the association of scoring systems with other quality measures, and no studies tested the impact of peer evaluation on patient outcomes. Best practices included standardized scoring systems, prospective criteria for case selection, and collaborative and non-judgmental review.ConclusionsPeer evaluation of invasive medical procedures is feasible and generally demonstrates good or excellent inter-observer agreement when performed with structured tools. Our review identifies common elements of successful interventions across specialties. However, there is limited evidence that peer-evaluated performance is linked to other quality measures or that feedback to clinicians improves patient care or outcomes. Additional research is needed to develop and test peer evaluation and feedback interventions.

Project description:Study Design: Scoping review Objective: To study the design, clinical setting and outcome measures used in spinal cord injury rehabilitation publications. Methods: A literature search on PubMed and Medline was conducted focusing on articles published between 1990-2016 and using "traumatic SCI", "functional outcomes", "rehabilitation", "work" and "return to work" as outcomes. Studies were categorized based on design (intervention, including RCTs vs. non-intervention studies), settings (inpatient vs. outpatient vs. transition), and outcome measures used (impairment vs. function vs. participation/integration vs. quality of life vs. symptoms). Work-related studies were categorized independently. Results: Five hundred forty-four articles met the inclusion criteria. Of these, 234 were interventional studies, including 23 RCTs. Studies were evenly divided among inpatient, outpatient and transition settings. Of the 234 interventional studies, 143 used functional evaluations. Sixty-one different functional instruments were used, with a predominant use of the Functional Independence Measure (61 times) and an additional use of SCI-specific measures, i.e. Spinal Cord Independence Measure and Craig Handicap Assessment and Reporting Technique (13 times each). Fifty-one studies measured mobility, while only three measured hand functions. The work-related sub-analysis revealed 32 intervention studies (no RCTs), of which 15 used functional evaluations and only three focused on tetraplegia. Conclusion: Our study revealed a paucity of intervention trials and RCTs, indicating a dearth of knowledge that would be needed to establish evidence-based practice guidelines. This is particularly true for tetraplegia. While standard measures of function were frequently used, providing valuable data, there is no consensus about what exact outcome measure to use. Using newer measurement techniques, for instance based on the application of item response theory, should be considered to enhance uniformity.

Project description:ImportanceClinical trial registries are intended to increase clinical research transparency by nonselectively identifying and documenting clinical trial designs and outcomes. Inconsistencies in reported data undermine the utility of such registries and have previously been noted in general medical literature.ObjectiveTo assess whether inconsistencies in reported data exist between ophthalmic literature and clinical trial registries.Design, setting, and participantsIn this retrospective, cross-sectional study, interventional clinical trials published from January 1, 2014, to December 31, 2014, in the American Journal of Ophthalmology, JAMA Ophthalmology, and Ophthalmology were reviewed. Observational, retrospective, uncontrolled, and post hoc reports were excluded, yielding a sample size of 106 articles. Data collection was performed from January through September 2016. Data review and adjudication continued through January 2017.Main outcomes and measuresIf possible, articles were matched to registry entries listed in the ClinicalTrials.gov database or in 1 of 16 international registries indexed by the World Health Organization International Clinical Trials Registry Platform version 3.2 search engine. Each article-registry pair was assessed for inconsistencies in design, results, and funding (each of which was further divided into subcategories) by 2 reviewers and adjudicated by a third.ResultsOf 106 trials that met the study criteria, matching registry entries were found for 68 (64.2%), whereas no matching registry entries were found for 38 (35.8%). Inconsistencies were identified in study design, study results, and funding sources, including specific interventions in 8 (11.8%), primary outcome measure (POM) designs in 32 (47.1%), and POM results in 48 (70.6%). In addition, numerous data pieces were unreported, including analysis methods in 52 (76.5%) and POM results in 38 (55.9%).Conclusions and relevanceClinical trial registries were underused in this sample of ophthalmology clinical trials. For studies with registry data, inconsistency rates between published and registered data were similar to those previously reported for general medical literature. In most cases, inconsistencies involved missing data, but explicit discrepancies in methods and/or data were also found. Transparency and credibility of published trials may be improved by closer attention to their registration and reporting.

Project description:BackgroundThere are increasing calls for patient involvement in sharing health research results, but no evidence-based recommendations to guide such involvement. Our objectives were to: (1) conduct a systematic review of the evidence on patient involvement in results sharing, (2) propose evidence-based recommendations to help maximize benefits and minimize risks of such involvement and (3) conduct this project with patient authors.MethodsTo avoid research waste, we verified that no systematic reviews were registered or published on this topic. We co-created, with patients, a PRISMA-P-compliant protocol. We included peer-reviewed publications reporting the effects of patient involvement in preparing peer-reviewed publications or results summaries from health research studies. We searched (9/10/2017) MEDLINE, EMBASE and the Cochrane Database of Systematic Reviews, and secondary information sources (until 11/06/2018). We assessed the risk of bias in eligible publications and extracted data using standardized processes. To evaluate patient involvement in this project, we co-created a Patient Authorship Experience Tool.ResultsAll nine eligible publications reported on patient involvement in preparing publications; none on preparing results summaries. Evidence quality was moderate. A qualitative synthesis of evidence indicated the benefits of patient involvement may outweigh the risks. We have proposed 21 evidence-based recommendations to help maximize the benefits and minimize the risks when involving patients as authors of peer-reviewed publications. The recommendations focus on practical actions patient and non-patient authors can take before (10 recommendations), during (7 recommendations) and after (4 recommendations) manuscript development. Using the Patient Authorship Experience Tool, both patient and non-patient authors rated their experience highly.ConclusionsBased on a systematic review, we have proposed 21 evidence-based recommendations to help maximize the benefits and minimize the risks of involving patients as authors of peer-reviewed publications.

Dataset Information

The ability of different peer review procedures to flag problematic publications.

Publications

The ability of different peer review procedures to flag problematic publications.

Similar Datasets

OmicsDI is part of the ELIXIR infrastructure

Tweets