Dataset Information

Development of AMSTAR: a measurement tool to assess the methodological quality of systematic reviews.

ABSTRACT:

Background

Our objective was to develop an instrument to assess the methodological quality of systematic reviews, building upon previous tools, empirical evidence and expert consensus.

Methods

A 37-item assessment tool was formed by combining 1) the enhanced Overview Quality Assessment Questionnaire (OQAQ), 2) a checklist created by Sacks, and 3) three additional items recently judged to be of methodological importance. This tool was applied to 99 paper-based and 52 electronic systematic reviews. Exploratory factor analysis was used to identify underlying components. The results were considered by methodological experts using a nominal group technique aimed at item reduction and design of an assessment tool with face and content validity.

Results

The factor analysis identified 11 components. From each component, one item was selected by the nominal group. The resulting instrument was judged to have face and content validity.

Conclusion

A measurement tool for the 'assessment of multiple systematic reviews' (AMSTAR) was developed. The tool consists of 11 items and has good face and content validity for measuring the methodological quality of systematic reviews. Additional studies are needed with a focus on the reproducibility and construct validity of AMSTAR, before strong recommendations can be made on its use.

SUBMITTER: Shea BJ

PROVIDER: S-EPMC1810543 | biostudies-literature | 2007 Feb

REPOSITORIES: biostudies-literature

ACCESS DATA

Publications

Development of AMSTAR: a measurement tool to assess the methodological quality of systematic reviews.

Shea Beverley J BJ Grimshaw Jeremy M JM Wells George A GA Boers Maarten M Andersson Neil N Hamel Candyce C Porter Ashley C AC Tugwell Peter P Moher David D Bouter Lex M LM

BMC medical research methodology 20070215

<h4>Background</h4>Our objective was to develop an instrument to assess the methodological quality of systematic reviews, building upon previous tools, empirical evidence and expert consensus.<h4>Methods</h4>A 37-item assessment tool was formed by combining 1) the enhanced Overview Quality Assessment Questionnaire (OQAQ), 2) a checklist created by Sacks, and 3) three additional items recently judged to be of methodological importance. This tool was applied to 99 paper-based and 52 electronic sys ...[more]

PMID: 17302989

Similar Datasets

Project description:BackgroundOverviews of reviews (overviews) compile information from multiple systematic reviews (SRs) to provide a single synthesis of relevant evidence for decision-making. It is recommended that authors assess and report the methodological quality of SRs in overviews-for example, using A MeaSurement Tool to Assess systematic Reviews (AMSTAR). Currently, there is variation in whether and how overview authors assess and report SR quality, and limited guidance is available. Our objectives were to: examine methodological considerations involved in using AMSTAR to assess the quality of Cochrane and non-Cochrane SRs in overviews of healthcare interventions; identify challenges (and develop potential decision rules) when using AMSTAR in overviews; and examine the potential impact of considering methodological quality when making inclusion decisions in overviews.MethodsWe selected seven overviews of healthcare interventions and included all SRs meeting each overview's inclusion criteria. For each SR, two reviewers independently conducted AMSTAR assessments with consensus and discussed challenges encountered. We also examined the correlation between AMSTAR assessments and SR results/conclusions.ResultsNinety-five SRs were included (30 Cochrane, 65 non-Cochrane). Mean AMSTAR assessments (9.6/11 vs. 5.5/11; p < 0.001) and inter-rater reliability (AC1 statistic: 0.84 vs. 0.69; "almost perfect" vs. "substantial" using the Landis & Koch criteria) were higher for Cochrane compared to non-Cochrane SRs. Four challenges were identified when applying AMSTAR in overviews: the scope of the SRs and overviews often differed; SRs examining similar topics sometimes made different methodological decisions; reporting of non-Cochrane SRs was sometimes poor; and some non-Cochrane SRs included other SRs as well as primary studies. Decision rules were developed to address each challenge. We found no evidence that AMSTAR assessments were correlated with SR results/conclusions.ConclusionsResults indicate that the AMSTAR tool can be used successfully in overviews that include Cochrane and non-Cochrane SRs, though decision rules may be useful to circumvent common challenges. Findings support existing recommendations that quality assessments of SRs in overviews be conducted independently, in duplicate, with a process for consensus. Results also suggest that using methodological quality to guide inclusion decisions (e.g., to exclude poorly conducted and reported SRs) may not introduce bias into the overview process.

Project description:BACKGROUND:Systematic reviews (SRs) in the field of neuropathic pain (NeuP) are increasingly important for decision-making. However, methodological flaws in SRs can reduce the validity of conclusions. Hence, it is important to assess the methodological quality of NeuP SRs critically. Additionally, it remains unclear which assessment tool should be used. We studied the methodological quality of SRs published in the field of NeuP and compared two assessment tools. METHODS:We systematically searched 5 electronic databases to identify SRs of randomized controlled trials of interventions for NeuP available up to March 2015. Two independent reviewers assessed the methodological quality of the studies using the Assessment of Multiple Systematic Reviews (AMSTAR) and the revised AMSTAR (R-AMSTAR) tools. The scores were converted to percentiles and ranked into 4 grades to allow comparison between the two checklists. Gwet's AC1 coefficient was used for interrater reliability assessment. RESULTS:The 97 included SRs had a wide range of methodological quality scores (AMSTAR median (IQR): 6 (5-8) vs. R-AMSTAR median (IQR): 30 (26-35)). The overall agreement score between the 2 raters was 0.62 (95% CI 0.39-0.86) for AMSTAR and 0.62 (95% CI 0.53-0.70) for R-AMSTAR. The 31 Cochrane systematic reviews (CSRs) were consistently ranked higher than the 66 non-Cochrane systematic reviews (NCSRs). The analysis of individual domains showed the best compliance in a comprehensive literature search (item 3) on both checklists. The results for the domain that was the least compliant differed: conflict of interest (item 11) was the item most poorly reported on AMSTAR vs. publication bias assessment (item 10) on R-AMSTAR. A high positive correlation between the total AMSTAR and R-AMSTAR scores for all SRs, as well as for CSRs and NCSRs, was observed. CONCLUSIONS:The methodological quality of analyzed SRs in the field of NeuP was not optimal, and CSRs had a higher quality than NCSRs. Both AMSTAR and R-AMSTAR tools produced comparable quality ratings. Our results point out to weaknesses in the methodology of existing SRs on interventions for the management NeuP and call for future improvement by better adherence to analyzed quality checklists, either AMSTAR or R-AMSTAR.

Project description:BackgroundSystematic reviews (SRs) with or without meta-analyses (MAs) are widely used in resolving questions in various healthcare areas (such as, traditional Chinese medicine, public health and surgery), and they are the cornerstone of evidence-based healthcare. However, the reliability of SRs is typically influenced by their methodological quality. AMSTAR (A Measurement Tool to Assess Systematic Reviews) and AMSTAR-2 tools can assess the methodological quality of SRs, and the use of AMSTAR has been investigated. However, AMSTAR-2 is now widely used to evaluate the methodological quality of SRs, but the use of AMSTAR-2 for determining the methodological quality of SRs has not yet been investigated and assessed thoroughly. Thus, we designed the present study to investigate the use of AMSTAR-2 in studies that assessed the methodological quality of a sample of SRs with the AMSTAR-2 and provide references to potential users of AMSTAR-2 tool.MethodsFour commonly used electronic databases including PubMed, EmBase, the Cochrane Library, and Web of Science will be searched following a comprehensive search strategy to identify and retrieve studies that have used AMSTAR-2 tool for evaluating the methodological quality of SRs. Two independent authors will retrieve bibliometric information and methodological data, including all author names, time of publication, and journal names, whether a specific score value was given for each item, and whether overall quality assessment was performed. Descriptive statistical analyses will be used to present the study results, e.g., frequencies and percentages, mean and standard deviation (SD) or median and interquartile range (IQR). In addition, subgroup analyses will be performed to identify the methodological differences (e.g., the reporting of study designs included in SRs) between overviews and methodological studies. The risk ratio (RR) with 95% confidence interval (95% CI) will be calculated to measure the methodological differences. Cytoscape 3.7.1 software tool will be used to construct collaboration network maps. Further, Microsoft Office Excel 2016 and Stata 12.0 will be used to manage and analyze data.DiscussionThe results of this study will identify any gaps in the use of AMSTAR-2 and important bibliometric features, such as active researchers and journals, provide guidance to researchers in various healthcare areas (such as, traditional Chinese medicine and public health) for using AMSTAR-2 tool and help them in developing cooperation and submitting their manuscripts.

Project description:BackgroundIn 2007, AMSTAR (A MeaSurement Tool to Assess systematic Reviews), a critical appraisal tool for systematic reviews (SRs), was published, and it has since become one of the most widely used instruments for SR appraisal. In September 2017, AMSTAR 2 was published as an updated version of the tool. This mixed-methods study aimed to analyze the extent of the AMSTAR 2 uptake and explore potential barriers to its uptake.MethodsWe analyzed the frequency of AMSTAR or AMSTAR 2 use in articles published in 2018, 2019 and 2020. We surveyed authors who have used AMSTAR but not AMSTAR 2 in the analyzed time frame to identify their reasons and barriers. The inclusion criterion for those authors was that the month of manuscript submission was after September 2017, i.e. after AMSTAR 2 was published.ResultsWe included 871 studies. The majority (N = 451; 52%) used AMSTAR 2, while 44% (N = 382) used AMSTAR, 4% (N = 31) used R-AMSTAR and others used a combination of tools. In 2018, 81% of the analyzed studies used AMSTAR, while 16% used AMSTAR 2. In 2019, 52% used AMSTAR, while 44% used AMSTAR 2. Among articles published in 2020, 28% used AMSTAR, while AMSTAR 2 was used by 69%. An author survey indicated that the authors did not use AMSTAR 2 mostly because they were not aware of it, their protocol was already established, or data collection completed at the time when the new tool was published. Barriers towards AMSTAR 2 use were lack of quantitative assessment, insufficient awareness, length, difficulties with a specific item.ConclusionIn articles published in 2018-2020, that were submitted to a journal after AMSTAR 2 tool was published, almost half of the authors (44%) still used AMSTAR, the old version of the tool. However, the use of AMSTAR has been declining in each subsequent year. Our survey indicated that editors and peer-reviewers did not ask the authors to use the new version of the tool. Few barriers towards using AMSTAR 2 were identified, and thus it is anticipated that the use of the old version of AMSTAR will continue to decline.

Dataset Information

Development of AMSTAR: a measurement tool to assess the methodological quality of systematic reviews.

Background

Methods

Results

Conclusion

Publications

Development of AMSTAR: a measurement tool to assess the methodological quality of systematic reviews.

Similar Datasets

OmicsDI is part of the ELIXIR infrastructure

Tweets