Quality assessment with diverse studies (QuADS): an appraisal tool for methodological and reporting quality in systematic reviews of mixed- or multi-method studies.
Ontology highlight
ABSTRACT: Background
In the context of the volume of mixed- and multi-methods studies in health services research, the present study sought to develop an appraisal tool to determine the methodological and reporting quality of such studies when included in systematic reviews. Evaluative evidence regarding the design and use of our existing Quality Assessment Tool for Studies with Diverse Designs (QATSDD) was synthesised to enhance and refine it for application across health services research.Methods
Secondary data were collected through a literature review of all articles identified using Google Scholar that had cited the QATSDD tool from its inception in 2012 to December 2019. First authors of all papers that had cited the QATSDD (n=197) were also invited to provide further evaluative data via a qualitative online survey. Evaluative findings from the survey and literature review were synthesised narratively and these data used to identify areas requiring refinement. The refined tool was subject to inter-rater reliability, face and content validity analyses.Results
Key limitations of the QATSDD tool identified related to a lack of clarity regarding scope of use of the tool and in the ease of application of criteria beyond experimental psychological research. The Quality Appraisal for Diverse Studies (QuADS) tool emerged as a revised tool to address the limitations of the QATSDD. The QuADS tool demonstrated substantial inter-rater reliability (k=0.66), face and content validity for application in systematic reviews with mixed, or multi-methods health services research.Conclusion
Our findings highlight the perceived value of appraisal tools to determine the methodological and reporting quality of studies in reviews that include heterogeneous studies. The QuADS tool demonstrates strong reliability and ease of use for application to multi or mixed-methods health services research.
SUBMITTER: Harrison R
PROVIDER: S-EPMC7885606 | biostudies-literature |
REPOSITORIES: biostudies-literature
Similar Datasets
Project description:BACKGROUND: Systematic reviews (SRs) must be of high quality. The purpose of our research was to compare the methodological and reporting quality of original versus updated Cochrane SRs to determine whether updating had improved these two quality dimensions. METHODS: We identified updated Cochrane SRs published in issue 4, 2002 of the Cochrane Library. We assessed the updated and original versions of the SRs using two instruments: the 10 item enhanced Overview Quality Assessment Questionnaire (OQAQ), and an 18-item reporting quality checklist and flow chart based upon the Quality of Reporting of Meta-analyses (QUOROM) statement. At least two reviewers extracted data and assessed quality. We calculated the percentage (with a 95% confidence interval) of 'yes' answers to each question. We calculated mean differences in percentage, 95% confidence intervals and p-values for each of the individual items and the overall methodological quality score of the updated and pre-updated versions using OQAQ. RESULTS: We assessed 53 SRs. There was no significant improvement in the global quality score of the OQAQ (mean difference 0.11 (-0.28; 0.70 p = 0.52)). Updated reviews showed a significant improvement of 18.9 (7.2; 30.6 p < .01) on the OQAQ item assessing whether the conclusions drawn by the author(s) were supported by the data and/or analysis presented in the SR. The QUOROM statement showed that the quality of reporting of Cochrane reviews improved in some areas with updating. Improvements were seen on the items relating to data sources reported in the abstract, with a significant difference of 17.0 (9.8; 28.7 p = 0.01), review methods, reported in the abstract 35 (24.1; 49.1 p = 0.00), searching methods 18.9 (9.7; 31.6 p = 0.01), and data abstraction 18.9 (11.7; 30.9 p = 0.00). CONCLUSION: The overall quality of Cochrane SRs is fair-to-good. Although reporting quality improved on certain individual items there was no overall improvement seen with updating and methodological quality remained unchanged. Further improvement of quality of reporting is possible. There is room for improvement of methodological quality as well. Authors updating reviews should address identified methodological or reporting weaknesses. We recommend to give full attention to both quality domains when updating SRs.
| S-EPMC1569863 | biostudies-other
Project description:The methodological and reporting quality of burn-specific systematic reviews has not been established. The aim of this study was to evaluate the methodological quality of systematic reviews in burn care management. Computerised searches were performed in Ovid MEDLINE, Ovid EMBASE and The Cochrane Library through to February 2016 for systematic reviews relevant to burn care using medical subject and free-text terms such as 'burn', 'systematic review' or 'meta-analysis'. Additional studies were identified by hand-searching five discipline-specific journals. Two authors independently screened papers, extracted and evaluated methodological quality using the 11-item A Measurement Tool to Assess Systematic Reviews (AMSTAR) tool and reporting quality using the 27-item Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) checklist. Characteristics of systematic reviews associated with methodological and reporting quality were identified. Descriptive statistics and linear regression identified features associated with improved methodological quality. A total of 60 systematic reviews met the inclusion criteria. Six of the 11 AMSTAR items reporting on 'a priori' design, duplicate study selection, grey literature, included/excluded studies, publication bias and conflict of interest were reported in less than 50% of the systematic reviews. Of the 27 items listed for PRISMA, 13 items reporting on introduction, methods, results and the discussion were addressed in less than 50% of systematic reviews. Multivariable analyses showed that systematic reviews associated with higher methodological or reporting quality incorporated a meta-analysis (AMSTAR regression coefficient 2.1; 95% CI: 1.1, 3.1; PRISMA regression coefficient 6·3; 95% CI: 3·8, 8·7) were published in the Cochrane library (AMSTAR regression coefficient 2·9; 95% CI: 1·6, 4·2; PRISMA regression coefficient 6·1; 95% CI: 3·1, 9·2) and included a randomised control trial (AMSTAR regression coefficient 1·4; 95%CI: 0·4, 2·4; PRISMA regression coefficient 3·4; 95% CI: 0·9, 5·8). The methodological and reporting quality of systematic reviews in burn care requires further improvement with stricter adherence by authors to the PRISMA checklist and AMSTAR tool.
| S-EPMC7949759 | biostudies-literature
Project description:BACKGROUND:The methodological quality and completeness of reporting of the systematic reviews (SRs) is fundamental to optimal implementation of evidence-based health care and the reduction of research waste. Methods exist to appraise SRs yet little is known about how they are used in SRs or where there are potential gaps in research best-practice guidance materials. The aims of this study are to identify reports assessing the methodological quality (MQ) and/or reporting quality (RQ) of a cohort of SRs and to assess their number, general characteristics, and approaches to 'quality' assessment over time. METHODS:The Cochrane Library, MEDLINE®, and EMBASE® were searched from January 1990 to October 16, 2014, for reports assessing MQ and/or RQ of SRs. Title, abstract, and full-text screening of all reports were conducted independently by two reviewers. Reports assessing the MQ and/or RQ of a cohort of ten or more SRs of interventions were included. All results are reported as frequencies and percentages of reports. RESULTS:Of 20,765 unique records retrieved, 1189 of them were reviewed for full-text review, of which 76 reports were included. Eight previously published approaches to assessing MQ or reporting guidelines used as proxy to assess RQ were used in 80% (61/76) of identified reports. These included two reporting guidelines (PRISMA and QUOROM) and five quality assessment tools (AMSTAR, R-AMSTAR, OQAQ, Mulrow, Sacks) and GRADE criteria. The remaining 24% (18/76) of reports developed their own criteria. PRISMA, OQAQ, and AMSTAR were the most commonly used published tools to assess MQ or RQ. In conjunction with other approaches, published tools were used in 29% (22/76) of reports, with 36% (8/22) assessing adherence to both PRISMA and AMSTAR criteria and 26% (6/22) using QUOROM and OQAQ. CONCLUSIONS:The methods used to assess quality of SRs are diverse, and none has become universally accepted. The most commonly used quality assessment tools are AMSTAR, OQAQ, and PRISMA. As new tools and guidelines are developed to improve both the MQ and RQ of SRs, authors of methodological studies are encouraged to put thoughtful consideration into the use of appropriate tools to assess quality and reporting.
| S-EPMC5477124 | biostudies-literature
Project description:BackgroundScoping reviews have emerged as a valuable method for synthesizing emerging evidence, providing a comprehensive contextual overview, and influencing policy and practice developments. The objective of this study is to provide an overview of scoping reviews conducted in Chinese academic institutions over the last decades.MethodWe conducted a comprehensive search of nine databases and six grey literature databases for scoping reviews conducted in Chinese academic institutions. The reporting quality of the included reviews was assessed using the Preferred Reporting Items for PRISMA-ScR checklist. We performed both quantitative and qualitative analyses, examining the conduct of the scoping reviews and exploring the breadth of research topics covered. We used Chi-squared and Wilcoxon rank-sum tests to compare methodological issues and reporting quality in English and Chinese-language reviews.ResultsA total of 392 reviews published between 2013 and 2022 were included, 238 English-reported reviews and 154 Chinese-reported reviews, respectively. The primary purposes of these reviews were to map and summarize the evidence, with a particular focus on health and nursing topics. 98.7% of reviews explicitly used the term "scoping review", and the Arksey and O'Malley framework was the most frequently cited framework. Thirty-five English-reported scoping reviews provided a protocol for scoping review. PubMed was the most common source in English-reported reviews and CNKI in Chinese-reported reviews. Reviews published in English were more likely to search the grey literature (P = 0.005), consult information specialists (P < 0.001) and conduct an updated search (P = 0.012) than those in Chinese. Reviews published in English had a significantly high score compared to those published in Chinese (16 vs. 14; P < 0.001). The reporting rates in English-reported reviews were higher than those in Chinese reviews for seven items, but lower for structured summary (P < 0.001), eligibility criteria (P < 0.001), data charting process (P = 0.009) and data items (P = 0.015).ConclusionThere has been a significant increase in the number of scoping reviews conducted in Chinese academic institutions each year since 2020. While the research topics covered are diverse, the overall reporting quality of these reviews is need to be improved. And there is a need for greater standardization in the conduct of scoping reviews in Chinese academic institutions.
| S-EPMC10882808 | biostudies-literature
Project description:BackgroundMixed methods designs are increasingly used in health care research to enrich findings. However, little is known about the frequency of use of this methodology in chiropractic research, or the quality of reporting among chiropractic studies using mixed methods.ObjectiveTo quantify the use and quality of mixed methods in chiropractic research, and explore the association of study characteristics (e.g., authorship, expertise, journal impact factor, country and year of publication) with reporting quality.MethodsWe will conduct a systematic search of MEDLINE, EMBASE, CINAHL, and the Index to Chiropractic Literature to identify all chiropractic mixed methods studies published from inception of each database to December 31, 2020. Articles reporting the use of both qualitative and quantitative methods, or mixed qualitative methods, will be included. Pairs of reviewers will perform article screening, data extraction, risk of bias with the Mixed Methods Appraisal Tool (MMAT), and appraisal of reporting quality using the Good Reporting of A Mixed Methods Study (GRAMMS) guideline. We will explore the correlation between GRAMMS and MMAT scores, and construct generalized estimating equations to explore factors associated with reporting quality.DiscussionThis will be the first methodological review to examine the reporting quality of published mixed methods studies involving chiropractic research. The results of our review will inform opportunities to improve reporting in chiropractic mixed methods studies. Our results will be disseminated in a peer-reviewed publication and presented publicly at conferences and as part of a doctoral thesis.
| S-EPMC8442283 | biostudies-literature
Project description:The methodological quality and reporting practices of laboratory studies of human eating behavior determine the validity and replicability of nutrition science. The aim of this research was to examine basic methodology and reporting practices in recent representative laboratory studies of human eating behavior. We examined laboratory studies of human eating behavior (N?=?140 studies) published during 2016. Basic methodology (e.g., sample size, use of participant blinding) and reporting practices (e.g., information on participant characteristics) were assessed for each study. Some information relating to participant characteristics (e.g., age, gender) and study methodology (e.g., length of washout periods in within-subjects studies) were reported in the majority of studies. However, other aspects of study reporting, including participant eligibility criteria and how sample size was determined were frequently not reported. Studies often did not appear to standardize pre-test meal appetite or attempt to blind participants to study aims. The average sample size of studies was small (between-subjects design studies in particular) and the primary statistical analyses in a number of studies (24%) were reliant on very small sample sizes that would be likely to produce unreliable results. There are basic methodology and reporting practices in the laboratory study of human eating behavior that are sub-optimal and this is likely to be affecting the validity and replicability of research. Recommendations to address these issues are discussed.
| S-EPMC5890731 | biostudies-literature
Project description:BACKGROUND: Systematic reviews can serve as a tool in translation of basic life sciences research from laboratory to human research and healthcare. The extent to which reviews of animal research are systematic and unbiased is not known. METHODS: We searched, without language restrictions, Medline, Embase, bibliographies of known reviews (1996-2004) and contacted experts to identify citations of reviews of basic science literature which, as a minimum, performed search of a publicly available resource. From these we identified reviews of animal studies where laboratory variables were measured or where treatments were administered to live animals to examine their effects, and compared them with reviews of bench studies in which human or animal tissues, cell systems or organ preparations were examined in laboratories to better understand mechanisms of diseases. RESULTS: Systematic reviews of animal studies often lacked methodological features such as specification of a testable hypothesis (9/30, 30%); literature search without language restriction (8/30, 26.6%); assessment of publication bias (5/30, 16.6%), study validity (15/30, 50%) and heterogeneity (10/30, 33.3%); and meta-analysis for quantitative synthesis (12/30, 40%). Compared to reviews of bench studies, they were less prone to bias as they specified the question (96.6% vs. 80%, p = 0.04), searched multiple databases (60% vs. 26.6%, p = 0.01), assessed study quality (50% vs. 20%, p = 0.01), and explored heterogeneity (33.3% vs. 2.2%, p = 0.001) more often. CONCLUSION: There seems to be a gradient of frequency of methodological weaknesses among reviews: Attempted systematic reviews of whole animal research tend to be better than those of bench studies, though compared to systematic reviews of human clinical trials they are apparently poorer. There is a need for rigour when reviewing animal research.
| S-EPMC1435907 | biostudies-other
Project description:BackgroundOur objective was to develop an instrument to assess the methodological quality of systematic reviews, building upon previous tools, empirical evidence and expert consensus.MethodsA 37-item assessment tool was formed by combining 1) the enhanced Overview Quality Assessment Questionnaire (OQAQ), 2) a checklist created by Sacks, and 3) three additional items recently judged to be of methodological importance. This tool was applied to 99 paper-based and 52 electronic systematic reviews. Exploratory factor analysis was used to identify underlying components. The results were considered by methodological experts using a nominal group technique aimed at item reduction and design of an assessment tool with face and content validity.ResultsThe factor analysis identified 11 components. From each component, one item was selected by the nominal group. The resulting instrument was judged to have face and content validity.ConclusionA measurement tool for the 'assessment of multiple systematic reviews' (AMSTAR) was developed. The tool consists of 11 items and has good face and content validity for measuring the methodological quality of systematic reviews. Additional studies are needed with a focus on the reproducibility and construct validity of AMSTAR, before strong recommendations can be made on its use.
| S-EPMC1810543 | biostudies-literature
Project description:ObjectivesTo assess the reporting and methodological quality of COVID-19 systematic reviews, and to analyze trends and gaps in the quality, clinical topics, author countries, and populations of the reviews using an evidence mapping approach.Study design and settingA structured search for systematic reviews concerning COVID-19 was performed using PubMed, Embase, Cochrane Library, Campbell Library, Web of Science, CBM, WanFang Data, CNKI, and CQVIP from inception until June 2020. The quality of each review was assessed using the Assessment of Multiple Systematic Reviews 2 (AMSTAR 2) checklist and the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) checklist.ResultsIn total, 243 systematic reviews met the inclusion criteria, over 50% of which (128, 52.7%) were from 14 developing countries, with China contributing the most reviews (76, 31.3%). In terms of methodological quality of the studies, 30 (12.3%) were of moderate quality, 63 (25.9%) were of low quality, and 150 (61.7%) were of critically low quality. In terms of reporting quality, the median (interquartile range) PRISMA score was 14 (10-18). Regarding the topics of the reviews, 24 (9.9%) focused on the prevalence of COVID-19, 69 (28.4%) focused on the clinical manifestations, 30 (12.3%) focused on etiology, 43 (17.7%) focused on diagnosis, 65 (26.7%) focused on treatment, 104 (42.8%) focused on prognosis, and 25 (10.3%) focused on prevention. These studies mainly focused on general patients with COVID-19 (161, 66.3%), followed by children (22, 9.1%) and pregnant patients (18, 7.4%).ConclusionThis study systematically evaluated the methodological and reporting quality of systematic reviews of COVID-19, summarizing and analyzing trends in their clinical topics, author countries, and study populations.
| S-EPMC8313077 | biostudies-literature
Project description:BackgroundThe quality of nutrition-related systematic reviews (SRs) is an unstudied but important factor affecting their usefulness.ObjectivesThe objectives were to evaluate the reporting quality of published SRs and to identify areas of improvement.DesignDescriptive and exploratory analyses of the reporting quality (7 nutrition items and 28 SR reporting items) of all English-language SRs published through July 2007 linking micronutrients and health outcomes in humans were conducted. Factors that may be associated with reporting quality were also evaluated.ResultsWe identified 141 eligible SRs of 21 micronutrients. Ninety SRs that included only interventional studies met a higher proportion of our reporting criteria (median: 62%; interquartile range: 51%, 72%) than did 31 SRs with only observational studies (median: 53%; interquartile range: 47%, 60%) or 20 SRs with both study designs (median: 47%; interquartile range: 39%, 52%) (P < 0.001). SRs published after consensus reporting standards (since 2003) met a higher proportion of the reporting criteria than did earlier SRs (median: 59% compared with 50%; P = 0.01); however, the reporting of nutrition variables remained unchanged (median: 38% compared with 33%; P = 0.7). The least-reported nutrition criteria were baseline nutrient exposures (28%) and effects of measurement errors from nutrition exposures (24%). Only 58 SRs (41%) used quality scales or checklists to assess the methodologic quality of the primary studies included.ConclusionsThe reporting quality of SRs has improved 3 y after publication of SR reporting standards, but the reporting of nutrition variables has not. Improved adherence to consensus methods and reporting standards should improve the utility of nutrition SRs.
| S-EPMC2667458 | biostudies-literature