Dataset Information

Assessment of a method to detect signals for updating systematic reviews.

ABSTRACT:

Background

Systematic reviews are a cornerstone of evidence-based medicine but are useful only if up-to-date. Methods for detecting signals of when a systematic review needs updating have face validity, but no proposed method has had an assessment of predictive validity performed.

Methods

The AHRQ Comparative Effectiveness Review program had produced 13 comparative effectiveness reviews (CERs), a subcategory of systematic reviews, by 2009, 11 of which were assessed in 2009 using a surveillance system to determine the degree to which individual conclusions were out of date and to assign a priority for updating each report. Four CERs were judged to be a high priority for updating, four CERs were judged to be medium priority for updating, and three CERs were judged to be low priority for updating. AHRQ then commissioned full update reviews for 9 of these 11 CERs. Where possible, we matched the original conclusions with their corresponding conclusions in the update reports, and compared the congruence between these pairs with our original predictions about which conclusions in each CER remained valid. We then classified the concordance of each pair as good, fair, or poor. We also made a summary determination of the priority for updating each CER based on the actual changes in conclusions in the updated report, and compared these determinations with the earlier assessments of priority.

Results

The 9 CERs included 149 individual conclusions, 84% with matches in the update reports. Across reports, 83% of matched conclusions had good concordance, and 99% had good or fair concordance. The one instance of poor concordance was partially attributable to the publication of new evidence after the surveillance signal searches had been done. Both CERs originally judged as being low priority for updating had no substantive changes to their conclusions in the actual updated report. The agreement on overall priority for updating between prediction and actual changes to conclusions was Kappa?=?0.74.

Conclusions

These results provide some support for the validity of a surveillance system for detecting signals indicating when a systematic review needs updating.

SUBMITTER: Shekelle PG

PROVIDER: S-EPMC3937021 | biostudies-literature | 2014 Feb

REPOSITORIES: biostudies-literature

ACCESS DATA

Publications

Assessment of a method to detect signals for updating systematic reviews.

Shekelle Paul G PG Motala Aneesa A Johnsen Breanne B Newberry Sydne J SJ

Systematic reviews 20140214

<h4>Background</h4>Systematic reviews are a cornerstone of evidence-based medicine but are useful only if up-to-date. Methods for detecting signals of when a systematic review needs updating have face validity, but no proposed method has had an assessment of predictive validity performed.<h4>Methods</h4>The AHRQ Comparative Effectiveness Review program had produced 13 comparative effectiveness reviews (CERs), a subcategory of systematic reviews, by 2009, 11 of which were assessed in 2009 using a ...[more]

PMID: 24529068

Similar Datasets

Project description:BACKGROUND: Systematic reviews (SRs) must be of high quality. The purpose of our research was to compare the methodological and reporting quality of original versus updated Cochrane SRs to determine whether updating had improved these two quality dimensions. METHODS: We identified updated Cochrane SRs published in issue 4, 2002 of the Cochrane Library. We assessed the updated and original versions of the SRs using two instruments: the 10 item enhanced Overview Quality Assessment Questionnaire (OQAQ), and an 18-item reporting quality checklist and flow chart based upon the Quality of Reporting of Meta-analyses (QUOROM) statement. At least two reviewers extracted data and assessed quality. We calculated the percentage (with a 95% confidence interval) of 'yes' answers to each question. We calculated mean differences in percentage, 95% confidence intervals and p-values for each of the individual items and the overall methodological quality score of the updated and pre-updated versions using OQAQ. RESULTS: We assessed 53 SRs. There was no significant improvement in the global quality score of the OQAQ (mean difference 0.11 (-0.28; 0.70 p = 0.52)). Updated reviews showed a significant improvement of 18.9 (7.2; 30.6 p < .01) on the OQAQ item assessing whether the conclusions drawn by the author(s) were supported by the data and/or analysis presented in the SR. The QUOROM statement showed that the quality of reporting of Cochrane reviews improved in some areas with updating. Improvements were seen on the items relating to data sources reported in the abstract, with a significant difference of 17.0 (9.8; 28.7 p = 0.01), review methods, reported in the abstract 35 (24.1; 49.1 p = 0.00), searching methods 18.9 (9.7; 31.6 p = 0.01), and data abstraction 18.9 (11.7; 30.9 p = 0.00). CONCLUSION: The overall quality of Cochrane SRs is fair-to-good. Although reporting quality improved on certain individual items there was no overall improvement seen with updating and methodological quality remained unchanged. Further improvement of quality of reporting is possible. There is room for improvement of methodological quality as well. Authors updating reviews should address identified methodological or reporting weaknesses. We recommend to give full attention to both quality domains when updating SRs.

Project description:BackgroundSystematic reviews (SRs) can become outdated as new evidence emerges over time. Organizations that produce SRs need a surveillance method to determine when reviews are likely to require updating. This report describes the development and initial results of a surveillance system to assess SRs produced by the Agency for Healthcare Research and Quality (AHRQ) Evidence-based Practice Center (EPC) Program.MethodsTwenty-four SRs were assessed using existing methods that incorporate limited literature searches, expert opinion, and quantitative methods for the presence of signals triggering the need for updating. The system was designed to begin surveillance six months after the release of the original review, and then ceforth every six months for any review not classified as being a high priority for updating. The outcome of each round of surveillance was a classification of the SR as being low, medium or high priority for updating.ResultsTwenty-four SRs underwent surveillance at least once, and ten underwent surveillance a second time during the 18 months of the program. Two SRs were classified as high, five as medium, and 17 as low priority for updating. The time lapse between the searches conducted for the original reports and the updated searches (search time lapse - STL) ranged from 11 months to 62 months: The STL for the high priority reports were 29 months and 54 months; those for medium priority reports ranged from 19 to 62 months; and those for low priority reports ranged from 11 to 33 months. Neither the STL nor the number of new relevant articles was perfectly associated with a signal for updating. Challenges of implementing the surveillance system included determining what constituted the actual conclusions of an SR that required assessing; and sometimes poor response rates of experts.ConclusionIn this system of regular surveillance of 24 systematic reviews on a variety of clinical interventions produced by a leading organization, about 70% of reviews were determined to have a low priority for updating. Evidence suggests that the time period for surveillance is yearly rather than the six months used in this project.

Project description:ObjectiveTo compare cancer-related systematic reviews (SRs) published in the Cochrane Database of SRs (CDSR) and high-impact journals, with respect to type, content, quality and citation rates.DesignMethodological SR with assessment and comparison of SRs and meta-analyses. Two authors independently assessed methodological quality using an Assessment of Multiple Systematic Reviews (AMSTAR)-based extraction form. Both authors independently screened search results, extracted content-relevant characteristics and retrieved citation numbers of the included reviews using the Clarivate Analytics Web of Science database.Data sourcesCancer-related SRs were retrieved from the CDSR, as well as from the 10 journals which publish oncological SRs and had the highest impact factors, using a comprehensive search in both the CDSR and MEDLINE.Eligibility criteria for selecting studiesWe included all cancer-related SRs and meta-analyses published from January 2011 to May 2016. Methodological SRs were excluded.ResultsWe included 346 applicable Cochrane reviews and 215 SRs from high-impact journals. Cochrane reviews consistently met more individual AMSTAR criteria, notably with regard to an a priori design (risk ratio (RR) 3.89; 95%?CI 3.10 to 4.88), inclusion of the grey literature and trial registries (RR 3.52; 95%?CI 2.84 to 4.37) in their searches, and the reporting of excluded studies (RR 8.80; 95%?CI 6.06 to 12.78). Cochrane reviews were less likely to address questions of prognosis (RR 0.04; 95%?CI 0.02 to 0.09), use individual patient data (RR 0.03; 95%?CI 0.01 to 0.09) or be based on non-randomised controlled trials (RR 0.04; 95%?CI 0.02 to 0.09). Citation rates of Cochrane reviews were notably lower than those for high-impact journals (Cochrane reviews: mean number of citations 6.52 (range 0-143); high-impact journal SRs: 74.45 (0-652)).ConclusionsWhen comparing cancer-related SRs published in the CDSR versus those published in high-impact medical journals, Cochrane reviews were consistently of higher methodological quality, but cited less frequently.

Project description:BackgroundThere is evidence of under-detection and poor management of pain in patients with dementia, in both long-term and acute care. Accurate assessment of pain in people with dementia is challenging and pain assessment tools have received considerable attention over the years, with an increasing number of tools made available. Systematic reviews on the evidence of their validity and utility mostly compare different sets of tools. This review of systematic reviews analyses and summarises evidence concerning the psychometric properties and clinical utility of pain assessment tools in adults with dementia or cognitive impairment.MethodsWe searched for systematic reviews of pain assessment tools providing evidence of reliability, validity and clinical utility. Two reviewers independently assessed each review and extracted data from them, with a third reviewer mediating when consensus was not reached. Analysis of the data was carried out collaboratively. The reviews were synthesised using a narrative synthesis approach.ResultsWe retrieved 441 potentially eligible reviews, 23 met the criteria for inclusion and 8 provided data for extraction. Each review evaluated between 8 and 13 tools, in aggregate providing evidence on a total of 28 tools. The quality of the reviews varied and the reporting often lacked sufficient methodological detail for quality assessment. The 28 tools appear to have been studied in a variety of settings and with varied types of patients. The reviews identified several methodological limitations across the original studies. The lack of a 'gold standard' significantly hinders the evaluation of tools' validity. Most importantly, the samples were small providing limited evidence for use of any of the tools across settings or populations.ConclusionsThere are a considerable number of pain assessment tools available for use with the elderly cognitive impaired population. However there is limited evidence about their reliability, validity and clinical utility. On the basis of this review no one tool can be recommended given the existing evidence.

Project description:BackgroundSystematic reviews of in-vitro studies, like any other study, can be of heterogeneous quality. The present study aimed to evaluate the methodological quality of systematic reviews of in-vitro dental studies.MethodsWe searched for systematic reviews of in-vitro dental studies in PubMed, Web of Science, and Scopus databases published up to January 2022. We assessed the methodological quality of the systematic reviews using a modified "A MeaSurement Tool to Assess systematic Reviews" (AMSTAR-2) instrument. The 16 items, in the form of questions, were answered with yes, no, or py (partial yes). Univariable and multivariable linear regression models were used to examine the association between systematic review characteristics and AMSTAR-2 percent score. Overall confidence in the results of the systematic reviews was rated, based on weaknesses identified in critical and non-critical AMSTAR-2 items.ResultsThe search retrieved 908 potential documents, and after following the eligibility criteria, 185 systematic reviews were included. The most researched topics were ceramics and dental bonding. The overall rating for the confidence in the results was critically low in 126 (68%) systematic reviews. There was high variability in the response among the AMSTAR-2 items (0% to 75% positively answered). The univariable analyses indicated dental specialty (p = 0.03), number of authors (coef: 1.87, 95% CI: 0.26, 3.47, p = 0.02), and year of publication (coef: 2.64, 95% CI: 1.90, 3.38, p < 0.01) were significantly associated with the AMSTAR-2 percent score. Whereas, in the multivariable analysis only specialty (p = 0.01) and year of publication (coef: 2.60, 95% CI: 1.84, 3.35, p < 0.001) remained significant. Among specialties, endodontics achieved the highest AMSTAR-2 percent score.ConclusionsThe methods of systematic reviews of in vitro dental studies were suboptimal. Year of publication and dental specialty were associated with AMSTAR-2 scores. The overall rating of the confidence in the results was low and critically low for most systematic reviews.

Project description:BACKGROUND:A recent study by Page et al. (PLoS Med. 2016;13(5):e1002028) claimed that increasing numbers of reviews are being published and many are poorly-conducted and reported. The aim of the present study was to assess how well reporting standards of systematic reviews produced in a Health Technology Assessment (HTA) context compare with reporting in Cochrane and other 'non-Cochrane' systematic reviews from the same years (2004 and 2014), as reported by Page et al. (PLoS Med. 2016;13(5):e1002028). METHODS:All relevant UK HTA programme systematic reviews published in 2004 and 2014 were identified. After piloting of the form, two reviewers each extracted relevant data on conduct and reporting from these reviews. These data were compared with data for Cochrane and "non-Cochrane" systematic reviews, as published by Page et al. (PLoS Med. 2016;13(5):e1002028). All data were tabulated and summarized. RESULTS:There were 30 UK HTA programme systematic reviews and 300 other systematic reviews, including Cochrane reviews (n?=?45). The percentage of HTA reviews with required elements of conduct and reporting was frequently very similar to Cochrane and much higher than all other systematic reviews, e.g. availability of protocols (90, 98 and 16% respectively); the specification of study design criteria (100, 100, 79%); the reporting of outcomes (100, 100, 78%), quality assessment (100, 100, 70%); the searching of trial registries for unpublished data (70, 62, 19%); reporting of reasons for excluding studies (91, 91 and 70%) and reporting of authors' conflicts of interests (100, 100, 87%). HTA reviews only compared less favourably with Cochrane and other reviews in assessments of publication bias. CONCLUSIONS:UK HTA systematic reviews are often produced within a specific policy-making context. This context has implications for timelines, tools and resources. However, UK HTA systematic reviews still tend to present standards of conduct and reporting equivalent to "gold standard" Cochrane reviews and superior to systematic reviews more generally.

Dataset Information

Assessment of a method to detect signals for updating systematic reviews.

Background

Methods

Results

Conclusions

Publications

Assessment of a method to detect signals for updating systematic reviews.

Similar Datasets

OmicsDI is part of the ELIXIR infrastructure

Tweets