Dataset Information

A new algorithm for reducing the workload of experts in performing systematic reviews.

ABSTRACT:

Objective

To determine whether a factorized version of the complement naïve Bayes (FCNB) classifier can reduce the time spent by experts reviewing journal articles for inclusion in systematic reviews of drug class efficacy for disease treatment.

Design

The proposed classifier was evaluated on a test collection built from 15 systematic drug class reviews used in previous work. The FCNB classifier was constructed to classify each article as containing high-quality, drug class-specific evidence or not. Weight engineering (WE) techniques were added to reduce underestimation for Medical Subject Headings (MeSH)-based and Publication Type (PubType)-based features. Cross-validation experiments were performed to evaluate the classifier's parameters and performance.

Measurements

Work saved over sampling (WSS) at no less than a 95% recall was used as the main measure of performance.

Results

The minimum workload reduction for a systematic review for one topic, achieved with a FCNB/WE classifier, was 8.5%; the maximum was 62.2% and the average over the 15 topics was 33.5%. This is 15.0% higher than the average workload reduction obtained using a voting perceptron-based automated citation classification system.

Conclusion

The FCNB/WE classifier is simple, easy to implement, and produces significantly better results in reducing the workload than previously achieved. The results support it being a useful algorithm for machine-learning-based automation of systematic reviews of drug class efficacy for disease treatment.

SUBMITTER: Matwin S

PROVIDER: S-EPMC2995653 | biostudies-literature | 2010 Jul-Aug

REPOSITORIES: biostudies-literature

ACCESS DATA

Publications

A new algorithm for reducing the workload of experts in performing systematic reviews.

Matwin Stan S Kouznetsov Alexandre A Inkpen Diana D Frunza Oana O O'Blenis Peter P

Journal of the American Medical Informatics Association : JAMIA 20100701 4

<h4>Objective</h4>To determine whether a factorized version of the complement naïve Bayes (FCNB) classifier can reduce the time spent by experts reviewing journal articles for inclusion in systematic reviews of drug class efficacy for disease treatment.<h4>Design</h4>The proposed classifier was evaluated on a test collection built from 15 systematic drug class reviews used in previous work. The FCNB classifier was constructed to classify each article as containing high-quality, drug class-specif ...[more]

PMID: 20595313

Similar Datasets

Project description:ObjectiveOsteoarthritis generates substantial health and socioeconomic burden, which is particularly marked in marginalized groups. It is imperative that practitioners have ready access to summaries of evidence-based interventions for osteoarthritis that incorporate equity considerations. Summaries of systematic reviews can provide this. The present study surveyed experts to prioritize a selection ofinterventions, from which equity focused summaries will be generated. Specifically, the prioritized interventions will be developed into Cochrane Evidence4Equity (E4E) summaries.MethodsTwenty-seven systematic reviews of OA interventions were found. From these, twenty-nine non-surgical treatments for osteoarthritis were identified, based on statistically significant findings for desired outcome variables or adverse events. Key findings from these studies were summarised and provided to 9 experts in the field of osteoarthritis.. Expert participants were asked to rate interventions based on feasibility, health system effects, universality, impact on inequities, and priority for translation into equity based E4E summaries. Expert participants were also encouraged to make comments to provide context for each rating. Free text responses were coded inductively and grouped into subthemes and themes.ResultsExpert participants rated the intervention home land-based exercise for knee OA highest for priority for translation into an E4E summaries, followed by the interventions individual land-based exercise for knee OA, class land-based exercise for knee OA, exercise for hand OA and land-based exercise for hip OA. Upon qualitative analysis of the expert participants' comments, fifteen subthemes were identified and grouped into three overall themes: (1) this intervention or an aspect of this intervention is unnecessary or unsafe; (2) this intervention or an aspect of this intervention may increase health inequities; and (3) experts noted difficulties completing rating exercise.ConclusionThe list of priority interventions and corresponding expert commentary generated information that will be used to direct and support knowledge translation efforts.

Project description:BACKGROUND:Here, we outline a method of applying existing machine learning (ML) approaches to aid citation screening in an on-going broad and shallow systematic review of preclinical animal studies. The aim is to achieve a high-performing algorithm comparable to human screening that can reduce human resources required for carrying out this step of a systematic review. METHODS:We applied ML approaches to a broad systematic review of animal models of depression at the citation screening stage. We tested two independently developed ML approaches which used different classification models and feature sets. We recorded the performance of the ML approaches on an unseen validation set of papers using sensitivity, specificity and accuracy. We aimed to achieve 95% sensitivity and to maximise specificity. The classification model providing the most accurate predictions was applied to the remaining unseen records in the dataset and will be used in the next stage of the preclinical biomedical sciences systematic review. We used a cross-validation technique to assign ML inclusion likelihood scores to the human screened records, to identify potential errors made during the human screening process (error analysis). RESULTS:ML approaches reached 98.7% sensitivity based on learning from a training set of 5749 records, with an inclusion prevalence of 13.2%. The highest level of specificity reached was 86%. Performance was assessed on an independent validation dataset. Human errors in the training and validation sets were successfully identified using the assigned inclusion likelihood from the ML model to highlight discrepancies. Training the ML algorithm on the corrected dataset improved the specificity of the algorithm without compromising sensitivity. Error analysis correction leads to a 3% improvement in sensitivity and specificity, which increases precision and accuracy of the ML algorithm. CONCLUSIONS:This work has confirmed the performance and application of ML algorithms for screening in systematic reviews of preclinical animal studies. It has highlighted the novel use of ML algorithms to identify human error. This needs to be confirmed in other reviews with different inclusion prevalence levels, but represents a promising approach to integrating human decisions and automation in systematic review methodology.

Project description:Concern has been expressed globally over rising caesarean birth rates. Recently, the International Federation of Gynaecology and Obstetrics (FIGO) called for help from governmental bodies, professional organisations, women's groups, and other stakeholders to reduce unnecessary caesareans. As part of a wider research initiative, we conducted an overview of systematic reviews of antenatal and intrapartum interventions, and reports of evidence based recommendations, to identify and highlight those that have been shown to be effective for reducing caesarean birth, promoting vaginal birth and reducing fear of childbirth. Following registration of the review protocol, (PROSPERO 2018 CRD42018090681), we searched The Cochrane Database of Systematic Reviews, PubMed, CINAHL and EMBASE (Jan 2000-Jan 2018) and searched for grey literature in PROSPERO, and on websites of health professional and other relevant bodies. Screening and selection of reviews, quality appraisal using AMSTAR-2, and data extraction were performed independently by pairs of at least two reviewers. Excluding reviews assessed as 'critically low' on AMSTAR-2 (n = 54), 101 systematic reviews, and 10 reports of evidence based recommendations were included in the overview. Narrative synthesis was performed, due to heterogeneity of review methodology and topics. The results highlight twenty-five interventions, across 17 reviews, that reduced the risk of caesarean, nine interventions across eight reviews that increased the risk of caesarean, eight interventions that reduced instrumental vaginal birth, four interventions that increased spontaneous vaginal birth, and two interventions that reduced fear of childbirth. This overview of reviews identifies and highlights interventions that have been shown to be effective for reducing caesarean birth, promoting vaginal births and reducing fear of childbirth. In recognising that clinical practices change over time, this overview includes reviews published from 2000 onwards only, thus providing contemporary evidence, and a valuable resource for clinicians when making decisions on practices that should be implemented for reducing unnecessary caesarean births safely. Protocol Registration: PROSPERO 2018 CRD42018090681. Available from: http://www.crd.york.ac.uk/PROSPERO/display_record.php?ID=CRD42018090681.

Project description:Systematic reviews have been challenged to consider effects on disadvantaged groups. A priori specification of subgroup analyses is recommended to increase the credibility of these analyses. This study aimed to develop and assess inter-rater agreement for an algorithm for systematic review authors to predict whether differences in effect measures are likely for disadvantaged populations relative to advantaged populations (only relative effect measures were addressed).A health equity plausibility algorithm was developed using clinimetric methods with three items based on literature review, key informant interviews and methodology studies. The three items dealt with the plausibility of differences in relative effects across sex or socioeconomic status (SES) due to: 1) patient characteristics; 2) intervention delivery (i.e., implementation); and 3) comparators. Thirty-five respondents (consisting of clinicians, methodologists and research users) assessed the likelihood of differences across sex and SES for ten systematic reviews with these questions. We assessed inter-rater reliability using Fleiss multi-rater kappa.The proportion agreement was 66% for patient characteristics (95% confidence interval: 61%-71%), 67% for intervention delivery (95% confidence interval: 62% to 72%) and 55% for the comparator (95% confidence interval: 50% to 60%). Inter-rater kappa, assessed with Fleiss kappa, ranged from 0 to 0.199, representing very low agreement beyond chance.Users of systematic reviews rated that important differences in relative effects across sex and socioeconomic status were plausible for a range of individual and population-level interventions. However, there was very low inter-rater agreement for these assessments. There is an unmet need for discussion of plausibility of differential effects in systematic reviews. Increased consideration of external validity and applicability to different populations and settings is warranted in systematic reviews to meet this need.

Project description:BACKGROUND: Policy makers and others often require synthesis of knowledge in an area within six months or less. Traditional systematic reviews typically take at least 12 months to conduct. Rapid reviews streamline traditional systematic review methods in order to synthesize evidence within a shortened timeframe. There is great variation in the process of conducting rapid reviews. This review sought to examine methods used for rapid reviews, as well as implications of methodological streamlining in terms of rigour, bias, and results. METHODS: A comprehensive search strategy--including five electronic databases, grey literature, hand searching of relevant journals, and contacting key informants--was undertaken. All titles and abstracts (n = 1,989) were reviewed independently by two reviewers. Relevance criteria included articles published between 1995 and 2009 about conducting rapid reviews or addressing comparisons of rapid reviews versus traditional reviews. Full articles were retrieved for any titles deemed relevant by either reviewer (n = 70). Data were extracted from all relevant methodological articles (n = 45) and from exemplars of rapid review methods (n = 25). RESULTS: Rapid reviews varied from three weeks to six months; various methods for speeding up the process were employed. Some limited searching by years, databases, language, and sources beyond electronic searches. Several employed one reviewer for title and abstract reviewing, full text review, methodological quality assessment, and/or data extraction phases. Within rapid review studies, accelerating the data extraction process may lead to missing some relevant information. Biases may be introduced due to shortened timeframes for literature searching, article retrieval, and appraisal. CONCLUSIONS: This review examined the continuum between diverse rapid review methods and traditional systematic reviews. It also examines potential implications of streamlined review methods. More of these rapid reviews need to be published in the peer-reviewed literature with an emphasis on articulating methods employed. While one consistent methodological approach may not be optimal or appropriate, it is important that researchers undertaking reviews within the rapid to systematic continuum provide detailed descriptions of methods used and discuss the implications of their chosen methods in terms of potential bias introduced. Further research comparing full systematic reviews with rapid reviews will enhance understanding of the limitations of these methods.

Dataset Information

A new algorithm for reducing the workload of experts in performing systematic reviews.

Objective

Design

Measurements

Results

Conclusion

Publications

A new algorithm for reducing the workload of experts in performing systematic reviews.

Similar Datasets

OmicsDI is part of the ELIXIR infrastructure