Dataset Information

Characteristics of stakeholder involvement in systematic and rapid reviews: a methodological review in the area of health services research.

ABSTRACT: OBJECTIVE:Engaging stakeholders in reviews is considered to generate more relevant evidence and to facilitate dissemination and use. As little is known about stakeholder involvement, we assessed the characteristics of their engagement in systematic and rapid reviews and the methodological quality of included studies. Stakeholders were people with a particular interest in the research topic. DESIGN:Methodological review. SEARCH STRATEGY:Four databases (Medline, Embase, Cochrane database of systematic reviews, databases of the University of York, Center for Reviews and Dissemination (CRD)) were searched based on an a priori protocol. Four types of reviews (Cochrane and non-Cochrane systematic reviews, rapid and CRD rapid reviews) were retrieved between January 2011 and October 2015, pooled by potential review type and duplicates excluded. Articles were randomly ordered and screened for inclusion and exclusion criteria until 30 reviews per group were reached. Their methodological quality was assessed using AMSTAR and stakeholder characteristics were collected. RESULTS:In total, 57 822 deduplicated citations were detected with potential non-Cochrane systematic reviews being the biggest group (56 986 records). We found stakeholder involvement in 13% (4/30) of Cochrane, 20% (6/30) of non-Cochrane, 43% (13/30) of rapid and 93% (28/30) of CRD reviews. Overall, 33% (17/51) of the responding contact authors mentioned positive effects of stakeholder involvement. A conflict of interest statement remained unmentioned in 40% (12/30) of non-Cochrane and in 27% (8/30) of rapid reviews, but not in Cochrane or CRD reviews. At most, half of non-Cochrane and rapid reviews mentioned an a priori study protocol in contrast to all Cochrane reviews. CONCLUSION:Stakeholder engagement was not general practice, except for CRD reviews, although it was more common in rapid reviews. Reporting factors, such as including an a priori study protocol and a conflict of interest statement should be considered in conjunction with involving stakeholders.

SUBMITTER: Feldmann J

PROVIDER: S-EPMC6701675 | biostudies-literature | 2019 Aug

REPOSITORIES: biostudies-literature

ACCESS DATA

Publications

Characteristics of stakeholder involvement in systematic and rapid reviews: a methodological review in the area of health services research.

Feldmann Jonas J Puhan Milo Alan MA Mütsch Margot M

BMJ open 20190815 8

<h4>Objective</h4>Engaging stakeholders in reviews is considered to generate more relevant evidence and to facilitate dissemination and use. As little is known about stakeholder involvement, we assessed the characteristics of their engagement in systematic and rapid reviews and the methodological quality of included studies. Stakeholders were people with a particular interest in the research topic.<h4>Design</h4>Methodological review.<h4>Search strategy</h4>Four databases (Medline, Embase, Cochr ...[more]

PMID: 31420378

Similar Datasets

Project description:BACKGROUND:There is increasing recognition that it is good practice to involve stakeholders (meaning patients, the public, health professionals and others) in systematic reviews, but limited evidence about how best to do this. We aimed to document the evidence-base relating to stakeholder involvement in systematic reviews and to use this evidence to describe how stakeholders have been involved in systematic reviews. METHODS:We carried out a scoping review, following a published protocol. We searched multiple electronic databases (2010-2016), using a stepwise searching approach, supplemented with hand searching. Two authors independently screened and discussed the first 500 abstracts and, after clarifying selection criteria, screened a further 500. Agreement on screening decisions was 97%, so screening was done by one reviewer only. Pre-planned data extraction was completed, and the comprehensiveness of the description of methods of involvement judged. Additional data extraction was completed for papers judged to have most comprehensive descriptions. Three stakeholder representatives were co-authors for this systematic review. RESULTS:We included 291 papers in which stakeholders were involved in a systematic review. Thirty percent involved patients and/or carers. Thirty-two percent were from the USA, 26% from the UK and 10% from Canada. Ten percent (32 reviews) were judged to provide a comprehensive description of methods of involving stakeholders. Sixty-nine percent (22/32) personally invited people to be involved; 22% (7/32) advertised opportunities to the general population. Eighty-one percent (26/32) had between 1 and 20 face-to-face meetings, with 83% of these holding ??4 meetings. Meetings lasted 1 h to ½ day. Nineteen percent (6/32) used a Delphi method, most often involving three electronic rounds. Details of ethical approval were reported by 10/32. Expenses were reported to be paid to people involved in 8/32 systematic reviews. DISCUSSION/CONCLUSION:We identified a relatively large number (291) of papers reporting stakeholder involvement in systematic reviews, but the quality of reporting was generally very poor. Information from a subset of papers judged to provide the best descriptions of stakeholder involvement in systematic reviews provide examples of different ways in which stakeholders have been involved in systematic reviews. These examples arguably currently provide the best available information to inform and guide decisions around the planning of stakeholder involvement within future systematic reviews. This evidence has been used to develop online learning resources. SYSTEMATIC REVIEW REGISTRATION:The protocol for this systematic review was published on 21 April 2017. Publication reference: Pollock A, Campbell P, Struthers C, Synnot A, Nunn J, Hill S, Goodare H, Watts C, Morley R: Stakeholder involvement in systematic reviews: a protocol for a systematic review of methods, outcomes and effects. Research Involvement and Engagement 2017, 3:9. https://doi.org/10.1186/s40900-017-0060-4 .

Project description:Qualitative systematic reviews or qualitative evidence syntheses (QES) are increasingly recognised as a way to enhance the value of systematic reviews (SRs) of clinical trials. They can explain the mechanisms by which interventions, evaluated within trials, might achieve their effect. They can investigate differences in effects between different population groups. They can identify which outcomes are most important to patients, carers, health professionals and other stakeholders. QES can explore the impact of acceptance, feasibility, meaningfulness and implementation-related factors within a real world setting and thus contribute to the design and further refinement of future interventions. To produce valid, reliable and meaningful QES requires systematic identification of relevant qualitative evidence. Although the methodologies of QES, including methods for information retrieval, are well-documented, little empirical evidence exists to inform their conduct and reporting.This structured methodological overview examines papers on searching for qualitative research identified from the Cochrane Qualitative and Implementation Methods Group Methodology Register and from citation searches of 15 key papers.A single reviewer reviewed 1299 references. Papers reporting methodological guidance, use of innovative methodologies or empirical studies of retrieval methods were categorised under eight topical headings: overviews and methodological guidance, sampling, sources, structured questions, search procedures, search strategies and filters, supplementary strategies and standards.This structured overview presents a contemporaneous view of information retrieval for qualitative research and identifies a future research agenda. This review concludes that poor empirical evidence underpins current information practice in information retrieval of qualitative research. A trend towards improved transparency of search methods and further evaluation of key search procedures offers the prospect of rapid development of search methods.

Project description:BackgroundTelemental health care has been rapidly adopted for maintaining services during the COVID-19 pandemic, and a substantial interest is now being devoted in its future role. Service planning and policy making for recovery from the pandemic and beyond should draw on both COVID-19 experiences and the substantial research evidence accumulated before this pandemic.ObjectiveWe aim to conduct an umbrella review of systematic reviews available on the literature and evidence-based guidance on telemental health, including both qualitative and quantitative literature.MethodsThree databases were searched between January 2010 and August 2020 for systematic reviews meeting the predefined criteria. The retrieved reviews were independently screened, and those meeting the inclusion criteria were synthesized and assessed for risk of bias. Narrative synthesis was used to report these findings.ResultsIn total, 19 systematic reviews met the inclusion criteria. A total of 15 reviews examined clinical effectiveness, 8 reported on the aspects of telemental health implementation, 10 reported on acceptability to service users and clinicians, 2 reported on cost-effectiveness, and 1 reported on guidance. Most reviews were assessed to be of low quality. The findings suggested that video-based communication could be as effective and acceptable as face-to-face formats, at least in the short term. Evidence on the extent of digital exclusion and how it can be overcome and that on some significant contexts, such as children and young people's services and inpatient settings, was found to be lacking.ConclusionsThis umbrella review suggests that telemental health has the potential to be an effective and acceptable form of service delivery. However, we found limited evidence on the impact of its large-scale implementation across catchment areas. Combining previous evidence and COVID-19 experiences may allow realistic planning for the future implementation of telemental health.

Project description:BackgroundThe screening process for systematic reviews and meta-analyses in medical research is a labor-intensive and time-consuming task. While machine learning and deep learning have been applied to facilitate this process, these methods often require training data and user annotation. This study aims to assess the efficacy of ChatGPT, a large language model based on the Generative Pretrained Transformers (GPT) architecture, in automating the screening process for systematic reviews in radiology without the need for training data.MethodsA prospective simulation study was conducted between May 2nd and 24th, 2023, comparing ChatGPT's performance in screening abstracts against that of general physicians (GPs). A total of 1198 abstracts across three subfields of radiology were evaluated. Metrics such as sensitivity, specificity, positive and negative predictive values (PPV and NPV), workload saving, and others were employed. Statistical analyses included the Kappa coefficient for inter-rater agreement, ROC curve plotting, AUC calculation, and bootstrapping for p-values and confidence intervals.ResultsChatGPT completed the screening process within an hour, while GPs took an average of 7-10 days. The AI model achieved a sensitivity of 95% and an NPV of 99%, slightly outperforming the GPs' sensitive consensus (i.e., including records if at least one person includes them). It also exhibited remarkably low false negative counts and high workload savings, ranging from 40 to 83%. However, ChatGPT had lower specificity and PPV compared to human raters. The average Kappa agreement between ChatGPT and other raters was 0.27.ConclusionsChatGPT shows promise in automating the article screening phase of systematic reviews, achieving high sensitivity and workload savings. While not entirely replacing human expertise, it could serve as an efficient first-line screening tool, particularly in reducing the burden on human resources. Further studies are needed to fine-tune its capabilities and validate its utility across different medical subfields.

Project description:BACKGROUND: Systematic reviews (SRs) must be of high quality. The purpose of our research was to compare the methodological and reporting quality of original versus updated Cochrane SRs to determine whether updating had improved these two quality dimensions. METHODS: We identified updated Cochrane SRs published in issue 4, 2002 of the Cochrane Library. We assessed the updated and original versions of the SRs using two instruments: the 10 item enhanced Overview Quality Assessment Questionnaire (OQAQ), and an 18-item reporting quality checklist and flow chart based upon the Quality of Reporting of Meta-analyses (QUOROM) statement. At least two reviewers extracted data and assessed quality. We calculated the percentage (with a 95% confidence interval) of 'yes' answers to each question. We calculated mean differences in percentage, 95% confidence intervals and p-values for each of the individual items and the overall methodological quality score of the updated and pre-updated versions using OQAQ. RESULTS: We assessed 53 SRs. There was no significant improvement in the global quality score of the OQAQ (mean difference 0.11 (-0.28; 0.70 p = 0.52)). Updated reviews showed a significant improvement of 18.9 (7.2; 30.6 p < .01) on the OQAQ item assessing whether the conclusions drawn by the author(s) were supported by the data and/or analysis presented in the SR. The QUOROM statement showed that the quality of reporting of Cochrane reviews improved in some areas with updating. Improvements were seen on the items relating to data sources reported in the abstract, with a significant difference of 17.0 (9.8; 28.7 p = 0.01), review methods, reported in the abstract 35 (24.1; 49.1 p = 0.00), searching methods 18.9 (9.7; 31.6 p = 0.01), and data abstraction 18.9 (11.7; 30.9 p = 0.00). CONCLUSION: The overall quality of Cochrane SRs is fair-to-good. Although reporting quality improved on certain individual items there was no overall improvement seen with updating and methodological quality remained unchanged. Further improvement of quality of reporting is possible. There is room for improvement of methodological quality as well. Authors updating reviews should address identified methodological or reporting weaknesses. We recommend to give full attention to both quality domains when updating SRs.

Dataset Information

Characteristics of stakeholder involvement in systematic and rapid reviews: a methodological review in the area of health services research.

Publications

Characteristics of stakeholder involvement in systematic and rapid reviews: a methodological review in the area of health services research.

Similar Datasets

OmicsDI is part of the ELIXIR infrastructure

Tweets