Dataset Information

Quality assessment of diagnostic before-after studies: development of methodology in the context of a systematic review.

ABSTRACT:

Background

Quality assessment tools for primary studies of test accuracy are relatively well developed, although only one is validated (QUADAS), but very little work has been done to develop tools to quality-assess studies evaluating the impact of diagnostic testing on management of patients (diagnostic or therapeutic yield). The recent draft NICE Guide to the Methods of Technology Appraisal (2007) suggests QUADAS "as a useful starting point for appraising studies that evaluate the sensitivity and specificity of a test" but does not mention how to quality assess diagnostic or therapeutic yield studies, in particular diagnostic before-after studies. In the context of undertaking a rapid systematic review of structural neuroimaging in psychosis for NICE, we describe the modifications that we made to QUADAS, our experience of this in practice and in relation to published theory on diagnostic or therapeutic yield studies.

Methods

The QUADAS tool was assessed for use in the review by two systematic reviewers with in-depth knowledge of the clinical area being reviewed and the types of studies being found in the searches that could answer the clinical question. Modifications were made following discussion as considered appropriate.

Results

Two QUADAS questions were removed altogether and. four additional questions were developed to capture additional quality issues not addressed by QUADAS. However, the developed checklist only partially helped to discern implications of the study designs on the results given.

Conclusion

The division between topic-specific and more generic quality items of relevance to diagnostic before-after studies is important. With more time, further work could have been done to create a better quality assessment tool, for example by incorporating some of the issues mentioned in previous work in this area. This paper is a discussion around quality assessment and is intended to offer insights into the types of issues that should be assessed. A quality assessment tool for diagnostic before-after studies that incorporates items from QUADAS and published theory needs to be further developed and validated.

SUBMITTER: Meads CA

PROVIDER: S-EPMC2630991 | biostudies-literature |

REPOSITORIES: biostudies-literature

ACCESS DATA

Similar Datasets

Project description:BackgroundMany systematic reviews (SRs) have been published about the various treatments for distal radius fractures (DRF). The heterogeneity of SRs results may come from the misuse of SR methods, and literature overviews have demonstrated that SRs should be considered with caution as they may not always be synonymous with high-quality standards. Our objective is to evaluate the quality of published SRs on the treatment of DRF through these tools.MethodsThe methods utilized in this review were previously published in the PROSPERO database. We considered SRs of surgical and nonsurgical interventions for acute DRF in adults. A comprehensive search strategy was performed in the MEDLINE database (inception to May 2017) and we manually searched the grey literature for non-indexed research. Data were independently extracted by two authors. We assessed SR internal validity and reporting using AMSTAR (Assessing the Methodological Quality of Systematic Reviews and PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyzes). Scores were calculated as the sum of reported items. We also extracted article characteristics and provided Spearman's correlation measurements.ResultsForty-one articles fulfilled the eligibility criteria. The mean score for PRISMA was 15.90 (CI 95%, 13.9-17.89) and AMSTAR was 6.48 (CI 95% 5.72-7.23). SRs that considered only RCTs had better AMSTAR [7.56 (2.1) vs. 5.62 (2.3); p = 0.014] and PRISMA scores [18.61 (5.22) vs. 13.93 (6.47), p = 0.027]. The presence of meta-analysis on the SRs altered PRISMA scores [19.17 (4.75) vs. 10.21 (4.51), p = 0.001] and AMSTAR scores [7.68 (1.9) vs. 4.39 (1.66), p = 0.001]. Journal impact factor or declaration of conflict of interest did not change PRISMA and AMSTAR scores. We found substantial inter observer agreement for PRISMA (0.82, 95% CI 0.62-0.94; p = 0.01) and AMSTAR (0.65, 95% CI 0.43-0.81; p = 0.01), and moderate correlation between PRISMA and AMSTAR scores (0.83, 95% CI 0.62-0.92; p = 0.01).ConclusionsDRF RCT-only SRs have better PRISMA and AMSTAR scores. These tools have substantial inter-observer agreement and moderate inter-tool correlation. We exposed the current research panorama and pointed out some factors that can contribute to improvements on the topic.

Project description:In the literature, there is widespread consensus that methods in plastic research need improvement. Current limitations in quality assurance and harmonization prevent progress in our understanding of the true effects of microplastic in the environment. Following the recent development of quality assessment methods for studies reporting concentrations in biota and water samples, we propose a method to assess the quality of microplastic effect studies. We reviewed 105 microplastic effect studies with aquatic biota, provided a systematic overview of their characteristics, developed 20 quality criteria in four main criteria categories (particle characterization, experimental design, applicability in risk assessment, and ecological relevance), propose a protocol for future effect studies with particles, and, finally, used all the information to define the weight of evidence with respect to demonstrated effect mechanisms. On average, studies scored 44.6% (range 20-77.5%) of the maximum score. No study scored positively on all criteria, reconfirming the urgent need for better quality assurance. Most urgent recommendations for improvement relate to avoiding and verifying background contamination, and to improving the environmental relevance of exposure conditions. The majority of the studies (86.7%) evaluated on particle characteristics properly, nonetheless it should be underlined that by failing to provide characteristics of the particles, an entire experiment can become irreproducible. Studies addressed environmentally realistic polymer types fairly well; however, there was a mismatch between sizes tested and those targeted when analyzing microplastic in environmental samples. In far too many instances, studies suggest and speculate mechanisms that are poorly supported by the design and reporting of data in the study. This represents a problem for decision-makers and needs to be minimized in future research. In their papers, authors frame 10 effects mechanisms as "suggested", whereas 7 of them are framed as "demonstrated". When accounting for the quality of the studies according to our assessment, three of these mechanisms remained. These are inhibition of food assimilation and/or decreased nutritional value of food, internal physical damage, and external physical damage. We recommend that risk assessment addresses these mechanisms with higher priority.

Project description:BackgroundOver the last decade there have been a number of guidelines published, aimed at improving the quality of reporting in published studies and reviews. In systematic reviews this may be measured by their compliance with the PRISMA statement. This review aims to evaluate the quality of reporting in published meta-analyses of diagnostic tests, using the PRISMA statement and establish whether there has been a measurable improvement over time.MethodsEight databases were searched for reviews published prior to 31(st) December 2008. Studies were selected if they evaluated a diagnostic test, measured performance, searched two or more databases, stated the search terms and inclusion criteria, and used a statistical method to summarise a test's performance. Data were extracted on the review characteristics and items of the PRISMA statement. To measure the change in the quality of reporting over time, PRISMA items for two periods of equal duration were compared.ResultsCompliance with the PRISMA statement was generally poor: none of the reviews completely adhered to all 27 checklist items. Of the 236 meta-analyses included following selection: only 2(1%) reported the study protocol; 59(25%) reported the searches used; 76(32%) reported the results of a risk of bias assessment; and 82(35%) reported the abstract as a structured summary. Only 11 studies were published before 2000. Thus, the impact of QUOROM on the quality of reporting was not evaluated. However, the periods 2001-2004 and 2005-2008 (covering 93% of studies) were compared using relative risks (RR). There was an increase in the proportion of reviews reporting on five PRISMA items: eligibility criteria (RR 1.13, 95% CI 1.00 - 1.27); risk of bias across studies (methods) (RR 1.81, 95% CI 1.34 - 2.44); study selection results (RR 1.48, 95% CI 1.05 - 2.09); results of individual studies (RR 1.37, 95% CI 1.09 - 1.72); risk of bias across studies (results) (RR 1.65, 95% CI 1.20 - 2.25).ConclusionAlthough there has been an improvement in the quality of meta-analyses in diagnostic research, there are still many deficiencies in the reporting which future reviewers need to address if readers are to trust the validity of the reported findings.

Project description:BACKGROUND:The implementation of evidence-based healthcare interventions is challenging, with a 17-year gap identified between the generation of evidence and its implementation in routine practice. Although contextual factors such as culture and leadership are strong influences for successful implementation, context remains poorly understood, with a lack of consensus regarding how it should be defined and captured within research. This study addresses this issue by providing insight into how context is defined and assessed within healthcare implementation science literature and develops a definition to enable effective measurement of context. METHODS:Medline, PsychInfo, CINAHL and EMBASE were searched. Articles were included if studies were empirical and evaluated context during the implementation of a healthcare initiative. These English language articles were published in the previous 10?years and included a definition and assessment of context. Results were synthesised using a narrative approach. RESULTS:Three thousand and twenty-one search records were obtained of which 64 met the eligibility criteria and were included in the review. Studies used a variety of definitions in terms of the level of detail and explanation provided. Some listed contextual factors (n?=?19) while others documented sub-elements of a framework that included context (n?=?19). The remaining studies provide a rich definition of general context (n?=?11) or aspects of context (n?=?15). The Alberta Context Tool was the most frequently used quantitative measure (n?=?4), while qualitative papers used a range of frameworks to evaluate context. Mixed methods studies used diverse approaches; some used frameworks to inform the methods chosen while others used quantitative measures to inform qualitative data collection. Most studies (n?=?50) applied the chosen measure to all aspects of study design with a majority analysing context at an individual level (n?=?29). CONCLUSIONS:This review highlighted inconsistencies in defining and measuring context which emphasised the need to develop an operational definition. By providing this consensus, improvements in implementation processes may result, as a common understanding will help researchers to appropriately account for context in research.