Methodological challenges for living systematic reviews conducted during the COVID-19 pandemic: a concept paper.
Ontology highlight
ABSTRACT: Background
A living systematic review (LSR) is an emerging review type that makes use of continual updating. In the COVID-19 pandemic, we were confronted with a shifting epidemiological landscape, clinical uncertainties and evolving evidence. These unexpected challenges compelled us to amend standard LSR methodology. Objective and outline: Our primary objective is to discuss some challenges faced when conducting LSRs in the context of the COVID-19 pandemic, and to provide methodological guidance for others doing similar work. Based on our experience and lessons learned from two Cochrane LSRs and challenges identified in several non-Cochrane LSRs, we highlight methodological considerations, particularly with regards to the study design, interventions and comparators, changes in outcome measure, and the search strategy. We discuss when to update, or rather when not to update the review, and the importance of transparency when reporting changes. Lessons learned and conclusions: We learned that a LSR is a very suitable review type for the pandemic context, even in the face of new methodological and clinical challenges. Our experience showed that the decision for updating a LSR depends not only on the evolving disease or emerging evidence, but also on the individual review question and the review teams' resources.
SUBMITTER: Iannizzi C
PROVIDER: S-EPMC8435072 | biostudies-literature |
REPOSITORIES: biostudies-literature
Similar Datasets
Project description:BACKGROUND:There is a notable lack of methodological and reporting guidance for systematic reviews of prevalence data. This information void has the potential to result in reviews that are inconsistent and inadequate to inform healthcare policy and decision making. The aim of this meta-epidemiological study is to describe the methodology of recently published prevalence systematic reviews. METHODS:We searched MEDLINE (via PubMed) from February 2017 to February 2018 for systematic reviews of prevalence studies. We included systematic reviews assessing the prevalence of any clinical condition using patients as the unit of measurement and we summarized data related to reporting and methodology of the reviews. RESULTS:A total of 235 systematic reviews of prevalence were analyzed. The median number of authors was 5 (interquartile range [IQR] 4-7), the median number of databases searched was 4 (3-6) and the median number of studies included in each review was 24 (IQR 15-41.5). Search strategies were presented for 68% of reviews. Forty five percent of reviews received external funding, and 24% did not provide funding information. Twenty three percent of included reviews had published or registered the systematic review protocol. Reporting guidelines were used in 72% of reviews. The quality of included studies was assessed in 80% of reviews. Nine reviews assessed the overall quality of evidence (4 using GRADE). Meta-analysis was conducted in 65% of reviews; 1% used Bayesian methods. Random effect meta-analysis was used in 94% of reviews; among them, 75% did not report the variance estimator used. Among the reviews with meta-analysis, 70% did not report how data was transformed; 59% percent conducted subgroup analysis, 38% conducted meta-regression and 2% estimated prediction interval; I2 was estimated in 95% of analysis. Publication bias was examined in 48%. The most common software used was STATA (55%). CONCLUSIONS:Our results indicate that there are significant inconsistencies regarding how these reviews are conducted. Many of these differences arose in the assessment of methodological quality and the formal synthesis of comparable data. This variability indicates the need for clearer reporting standards and consensus on methodological guidance for systematic reviews of prevalence data.
| S-EPMC7184711 | biostudies-literature
Project description:Background: The living systematic review (LSR) is an emerging approach for improved evidence synthesis that uses continual updating to include relevant new evidence as soon as it is published. The objectives of this study are to: 1) assess the methods of conduct and reporting of living systematic reviews using a living study approach; and 2) describe the life cycle of living systematic reviews, i.e., describe the changes over time to their methods and findings. Methods: For objective 1, we will begin by conducting a cross-sectional survey and then update its findings every 6 months by including newly published LSRs. For objective 2, we will conduct a prospective longitudinal follow-up of the cohort of included LSRs. To identify LSRs, we will continually search the following electronic databases: Medline, EMBASE and the Cochrane library. We will also contact groups conducting LSRs to identify eligible studies that we might have missed. We will follow the standard systematic review methodology for study selection and data abstraction. For each LSR update, we will abstract information on the following: 1) general characteristics, 2) systematic review methodology, 3) living approach methodology, 4) results, and 5) editorial and publication processes. We will update the findings of both the surveys and the longitudinal follow-up of included LSRs every 6 months. In addition, we will identify articles addressing LSR methods to be included in an 'LSR methods repository'. Conclusion: The proposed living methodological survey will allow us to monitor how the methods of conduct, and reporting as well as the findings of LSRs change over time. Ultimately this should help with ensuring the quality and transparency of LSRs.
| S-EPMC6556985 | biostudies-literature
Project description:Moderate-to-severe psoriasis is associated with significant comorbidity, an impaired quality of life, and increased medical costs, including those associated with treatments. Systematic reviews (SRs) and meta-analyses (MAs) of randomized clinical trials are considered two of the best approaches to the summarization of high-quality evidence. However, methodological bias can reduce the validity of conclusions from these types of studies and subsequently impair the quality of decision making. As co-authorship is among the most well-documented forms of research collaboration, the present study aimed to explore whether authors' collaboration methods might influence the methodological quality of SRs and MAs of psoriasis. Methodological quality was assessed by two raters who extracted information from full articles. After calculating total and per-item Assessment of Multiple Systematic Reviews (AMSTAR) scores, reviews were classified as low (0-4), medium (5-8), or high (9-11) quality. Article metadata and journal-related bibliometric indices were also obtained. A total of 741 authors from 520 different institutions and 32 countries published 220 reviews that were classified as high (17.2%), moderate (55%), or low (27.7%) methodological quality. The high methodological quality subnetwork was larger but had a lower connection density than the low and moderate methodological quality subnetworks; specifically, the former contained relatively fewer nodes (authors and reviews), reviews by authors, and collaborators per author. Furthermore, the high methodological quality subnetwork was highly compartmentalized, with several modules representing few poorly interconnected communities. In conclusion, structural differences in author-paper affiliation network may influence the methodological quality of SRs and MAs on psoriasis. As the author-paper affiliation network structure affects study quality in this research field, authors who maintain an appropriate balance between scientific quality and productivity are more likely to develop higher quality reviews.
| S-EPMC5389828 | biostudies-literature
Project description:BackgroundTreatment burden can be defined as the self-care practices that patients with chronic illness must perform to respond to the requirements of their healthcare providers, as well as the impact that these practices have on patient functioning and well being. Increasing levels of treatment burden may lead to suboptimal adherence and negative outcomes. Systematic review of the qualitative literature is a useful method for exploring the patient experience of care, in this case the experience of treatment burden. There is no consensus on methods for qualitative systematic review. This paper describes the methodology used for qualitative systematic reviews of the treatment burdens identified in three different common chronic conditions, using stroke as our exemplar.MethodsQualitative studies in peer reviewed journals seeking to understand the patient experience of stroke management were sought. Limitations of English language and year of publication 2000 onwards were set. An exhaustive search strategy was employed, consisting of a scoping search, database searches (Scopus, CINAHL, Embase, Medline & PsycINFO) and reference, footnote and citation searching. Papers were screened, data extracted, quality appraised and analysed by two individuals, with a third party for disagreements. Data analysis was carried out using a coding framework underpinned by Normalization Process Theory (NPT).ResultsA total of 4364 papers were identified, 54 were included in the review. Of these, 51 (94%) were retrieved from our database search. Methodological issues included: creating an appropriate search strategy; investigating a topic not previously conceptualised; sorting through irrelevant data within papers; the quality appraisal of qualitative research; and the use of NPT as a novel method of data analysis, shown to be a useful method for the purposes of this review.ConclusionThe creation of our search strategy may be of particular interest to other researchers carrying out synthesis of qualitative studies. Importantly, the successful use of NPT to inform a coding frame for data analysis involving qualitative data that describes processes relating to self management highlights the potential of a new method for analyses of qualitative data within systematic reviews.
| S-EPMC3568050 | biostudies-literature
Project description:BackgroundPharmacy interventions are a subset of public health interventions and its research is usually performed within the scope of a trial. The economic evaluation of pharmacy interventions requires certain considerations which have some similarities to those of public health interventions and to economic evaluations alongside trials. The objective of this research is to perform an overview of systematic reviews of economic evaluations of pharmacy services and triangulate results with recommendations for economic evaluations of both public health interventions and alongside trials.Methods(1) Exploratory review of recommendations on the economic evaluation of public health interventions, (2) exploratory review of recommendations for conducting economic evaluations alongside trials, (3) overview of systematic reviews of economic evaluations of pharmacy interventions (protocol registered with PROSPERO 2016 outlining information sources, inclusion criteria, appraisal of reviews and synthesis methods).ResultsFourteen systematic reviews containing 75 index publications were included. Reviews reported favorable economic findings for 71% of studies with full economic evaluations. The types of economic analysis are diverse. Two critical quality domains are absent from most reviews. Key findings include the following: certain types of risk of bias, wider scope of study designs, and most economic quality criteria met but some issues unresolved or unclear. Triangulation revealed additional gaps. Limitations include choice of critical quality domains and potential biases in the overview process.ConclusionsEconomic evaluations of pharmacy-based public health interventions seem to follow most economic quality criteria, but there are still some issues in certain key areas to improve. These findings may assist in improving the design of pilot trials of economic evaluations in pharmacy, leading to robust evidence for payers. Based on the findings, we propose a methodological approach for the economic evaluation of pharmacy-based public health interventions.Systematic review registrationPROSPERO CRD42016032768.
| S-EPMC6844055 | biostudies-literature
Project description:Background: Systematic reviews (SR) can be classified by type depending on the research question they are based on. This work identifies and describes the most relevant methodological resources to conduct high-quality reviews that answer health care questions regarding prevalence, prognosis, diagnostic accuracy and effects of interventions. Methods: Methodological resources have been identified from literature searches and consulting guidelines from institutions that develop SRs. The selected resources are organized by type of SR, and stage of development of the review (formulation of the research question, development of the protocol, literature search, risk of bias assessment, synthesis of findings, assessment of the quality of evidence, and report of SR results and conclusions). Results: Although the different types of SRs are developed following the same steps, each SR type requires specific methods, differing in characteristics and complexity. The extent of methodological development varies by type of SR, with more solid guidelines available for diagnostic accuracy and effects of interventions SRs. This methodological toolkit describes the most up-to-date risk of bias instruments: Quality in Prognostic Studies (QUIPS) tool and Prediction model study Risk Of Bias Assessment Tool (PROBAST) for prognostic SRs, Quality assessment of diagnostic accuracy studies tool (QUADAS-2) for diagnostic accuracy SRs, Cochrane risk of bias tool (ROB-2) and Risk of bias in non-randomised studies of interventions studies tool (ROBINS-I) for effects of interventions SRs, as well as the latest developments on the Grading of Recommendations Assessment, Development and Evaluation (GRADE) system. Conclusions: This structured compilation of the best methodological resources for each type of SR may prove to be a very useful tool for those researchers that wish to develop SRs or conduct methodological research works on SRs.
| S-EPMC7542253 | biostudies-literature
Project description:BackgroundNavigating the rapidly growing body of scientific literature on the SARS-CoV-2 pandemic is challenging, and ongoing critical appraisal of this output is essential. We aimed to summarize and critically appraise systematic reviews of coronavirus disease (COVID-19) in humans that were available at the beginning of the pandemic.MethodsNine databases (Medline, EMBASE, Cochrane Library, CINAHL, Web of Sciences, PDQ-Evidence, WHO's Global Research, LILACS, and Epistemonikos) were searched from December 1, 2019, to March 24, 2020. Systematic reviews analyzing primary studies of COVID-19 were included. Two authors independently undertook screening, selection, extraction (data on clinical symptoms, prevalence, pharmacological and non-pharmacological interventions, diagnostic test assessment, laboratory, and radiological findings), and quality assessment (AMSTAR 2). A meta-analysis was performed of the prevalence of clinical outcomes.ResultsEighteen systematic reviews were included; one was empty (did not identify any relevant study). Using AMSTAR 2, confidence in the results of all 18 reviews was rated as "critically low". Identified symptoms of COVID-19 were (range values of point estimates): fever (82-95%), cough with or without sputum (58-72%), dyspnea (26-59%), myalgia or muscle fatigue (29-51%), sore throat (10-13%), headache (8-12%) and gastrointestinal complaints (5-9%). Severe symptoms were more common in men. Elevated C-reactive protein and lactate dehydrogenase, and slightly elevated aspartate and alanine aminotransferase, were commonly described. Thrombocytopenia and elevated levels of procalcitonin and cardiac troponin I were associated with severe disease. A frequent finding on chest imaging was uni- or bilateral multilobar ground-glass opacity. A single review investigated the impact of medication (chloroquine) but found no verifiable clinical data. All-cause mortality ranged from 0.3 to 13.9%.ConclusionsIn this overview of systematic reviews, we analyzed evidence from the first 18 systematic reviews that were published after the emergence of COVID-19. However, confidence in the results of all reviews was "critically low". Thus, systematic reviews that were published early on in the pandemic were of questionable usefulness. Even during public health emergencies, studies and systematic reviews should adhere to established methodological standards.
| S-EPMC8177249 | biostudies-literature
Project description:Rationale, aims, and objectivesCOVID-19 has caused an ongoing public health crisis. Many systematic reviews and meta-analyses have been performed to synthesize evidence for better understanding this new disease. However, some concerns have been raised about rapid COVID-19 research. This meta-epidemiological study aims to methodologically assess the current systematic reviews and meta-analyses on COVID-19.MethodsWe searched in various databases for systematic reviews with meta-analyses published between 1 January 2020 and 31 October 2020. We extracted their basic characteristics, data analyses, evidence appraisal, and assessment of publication bias and heterogeneity.ResultsWe identified 295 systematic reviews on COVID-19. The median time from submission to acceptance was 33 days. Among these systematic reviews, 73.9% evaluated clinical manifestations or comorbidities of COVID-19. Stata was the most used software programme (43.39%). The odds ratio was the most used effect measure (34.24%). Moreover, 28.14% of the systematic reviews did not present evidence appraisal. Among those reporting the risk of bias results, 14.64% of studies had a high risk of bias. Egger's test was the most used method for assessing publication bias (38.31%), while 38.66% of the systematic reviews did not assess publication bias. The I2 statistic was widely used for assessing heterogeneity (92.20%); many meta-analyses had high values of I2 . Among the meta-analyses using the random-effects model, 75.82% did not report the methods for model implementation; among those meta-analyses reporting implementation methods, the DerSimonian-Laird method was the most used one.ConclusionsThe current systematic reviews and meta-analyses on COVID-19 might suffer from low transparency, high heterogeneity, and suboptimal statistical methods. It is recommended that future systematic reviews on COVID-19 strictly follow well-developed guidelines. Sensitivity analyses may be performed to examine how the synthesized evidence might depend on different methods for appraising evidence, assessing publication bias, and implementing meta-analysis models.
| S-EPMC8242754 | biostudies-literature
Project description:BackgroundWhen conducting an Overviews of Reviews on health-related topics, it is unclear which combination of bibliographic databases authors should use for searching for SRs. Our goal was to determine which databases included the most systematic reviews and identify an optimal database combination for searching systematic reviews.MethodsA set of 86 Overviews of Reviews with 1219 included systematic reviews was extracted from a previous study. Inclusion of the systematic reviews was assessed in MEDLINE, CINAHL, Embase, Epistemonikos, PsycINFO, and TRIP. The mean inclusion rate (% of included systematic reviews) and corresponding 95% confidence interval were calculated for each database individually, as well as for combinations of MEDLINE with each other database and reference checking.ResultsInclusion of systematic reviews was higher in MEDLINE than in any other single database (mean inclusion rate 89.7%; 95% confidence interval [89.0-90.3%]). Combined with reference checking, this value increased to 93.7% [93.2-94.2%]. The best combination of two databases plus reference checking consisted of MEDLINE and Epistemonikos (99.2% [99.0-99.3%]). Stratification by Health Technology Assessment reports (97.7% [96.5-98.9%]) vs. Cochrane Overviews (100.0%) vs. non-Cochrane Overviews (99.3% [99.1-99.4%]) showed that inclusion was only slightly lower for Health Technology Assessment reports. However, MEDLINE, Epistemonikos, and reference checking remained the best combination. Among the 10/1219 systematic reviews not identified by this combination, five were published as websites rather than journals, two were included in CINAHL and Embase, and one was included in the database ERIC.ConclusionsMEDLINE and Epistemonikos, complemented by reference checking of included studies, is the best database combination to identify systematic reviews on health-related topics.
| S-EPMC7268249 | biostudies-literature
Project description:BackgroundThe screening process for systematic reviews and meta-analyses in medical research is a labor-intensive and time-consuming task. While machine learning and deep learning have been applied to facilitate this process, these methods often require training data and user annotation. This study aims to assess the efficacy of ChatGPT, a large language model based on the Generative Pretrained Transformers (GPT) architecture, in automating the screening process for systematic reviews in radiology without the need for training data.MethodsA prospective simulation study was conducted between May 2nd and 24th, 2023, comparing ChatGPT's performance in screening abstracts against that of general physicians (GPs). A total of 1198 abstracts across three subfields of radiology were evaluated. Metrics such as sensitivity, specificity, positive and negative predictive values (PPV and NPV), workload saving, and others were employed. Statistical analyses included the Kappa coefficient for inter-rater agreement, ROC curve plotting, AUC calculation, and bootstrapping for p-values and confidence intervals.ResultsChatGPT completed the screening process within an hour, while GPs took an average of 7-10 days. The AI model achieved a sensitivity of 95% and an NPV of 99%, slightly outperforming the GPs' sensitive consensus (i.e., including records if at least one person includes them). It also exhibited remarkably low false negative counts and high workload savings, ranging from 40 to 83%. However, ChatGPT had lower specificity and PPV compared to human raters. The average Kappa agreement between ChatGPT and other raters was 0.27.ConclusionsChatGPT shows promise in automating the article screening phase of systematic reviews, achieving high sensitivity and workload savings. While not entirely replacing human expertise, it could serve as an efficient first-line screening tool, particularly in reducing the burden on human resources. Further studies are needed to fine-tune its capabilities and validate its utility across different medical subfields.
| S-EPMC10976661 | biostudies-literature