Dataset Information

Data Extraction and Synthesis in Systematic Reviews of Diagnostic Test Accuracy: A Corpus for Automating and Evaluating the Process.

ABSTRACT: BACKGROUND:Systematic reviews are critical for obtaining accurate estimates of diagnostic test accuracy, yet these require extracting information buried in free text articles, an often laborious process. OBJECTIVE:We create a dataset describing the data extraction and synthesis processes in 63 DTA systematic reviews, and demonstrate its utility by using it to replicate the data synthesis in the original reviews. METHOD:We construct our dataset using a custom automated extraction pipeline complemented with manual extraction, verification, and post-editing. We evaluate using manual assessment by two annotators and by comparing against data extracted from source files. RESULTS:The constructed dataset contains 5,848 test results for 1,354 diagnostic tests from 1,738 diagnostic studies. We observe an extraction error rate of 0.06-0.3%. CONCLUSIONS:This constitutes the first dataset describing the later stages of the DTA systematic review process, and is intended to be useful for automating or evaluating the process.

SUBMITTER: Norman C

PROVIDER: S-EPMC6371350 | biostudies-literature | 2018

REPOSITORIES: biostudies-literature

ACCESS DATA

Publications

Data Extraction and Synthesis in Systematic Reviews of Diagnostic Test Accuracy: A Corpus for Automating and Evaluating the Process.

Norman Christopher C Leeflang Mariska M Névéol Aurélie A

AMIA ... Annual Symposium proceedings. AMIA Symposium 20181205

<h4>Background</h4>Systematic reviews are critical for obtaining accurate estimates of diagnostic test accuracy, yet these require extracting information buried in free text articles, an often laborious process.<h4>Objective</h4>We create a dataset describing the data extraction and synthesis processes in 63 DTA systematic reviews, and demonstrate its utility by using it to replicate the data synthesis in the original reviews.<h4>Method</h4>We construct our dataset using a custom automated extra ...[more]

PMID: 30815124

Similar Datasets

Project description:ImportanceSystematic reviews of medical imaging diagnostic test accuracy (DTA) studies are affected by between-study heterogeneity due to a range of factors. Failure to appropriately assess the extent and causes of heterogeneity compromises the interpretability of systematic review findings.ObjectiveTo assess how heterogeneity has been examined in medical imaging DTA studies.Evidence reviewThe PubMed database was searched for systematic reviews of medical imaging DTA studies that performed a meta-analysis. The search was limited to the 40 journals with highest impact factor in the radiology, nuclear medicine, and medical imaging category in the InCites Journal Citation Reports of 2021 to reach a sample size of 200 to 300 included studies. Descriptive analysis was performed to characterize the imaging modality, target condition, type of meta-analysis model used, strategies for evaluating heterogeneity, and sources of heterogeneity identified. Multivariable logistic regression was performed to assess whether any factors were associated with at least 1 source of heterogeneity being identified in the included meta-analyses. Methodological quality evaluation was not performed. Data analysis occurred from October to December 2022.FindingsA total of 242 meta-analyses involving a median (range) of 987 (119-441 510) patients across a diverse range of disease categories and imaging modalities were included. The extent of heterogeneity was adequately described (ie, whether it was absent, low, moderate, or high) in 220 studies (91%) and was most commonly assessed using the I2 statistic (185 studies [76%]) and forest plots (181 studies [75%]). Heterogeneity was rated as moderate to high in 191 studies (79%). Of all included meta-analyses, 122 (50%) performed subgroup analysis and 87 (36%) performed meta-regression. Of the 242 studies assessed, 189 (78%) included 10 or more primary studies. Of these 189 studies, 60 (32%) did not perform meta-regression or subgroup analysis. Reasons for being unable to investigate sources of heterogeneity included inadequate reporting of primary study characteristics and a low number of included primary studies. Use of meta-regression was associated with identification of at least 1 source of variability (odds ratio, 1.90; 95% CI, 1.11-3.23; P = .02).Conclusions and relevanceIn this systematic review of assessment of heterogeneity in medical imaging DTA meta-analyses, most meta-analyses were impacted by a moderate to high level of heterogeneity, presenting interpretive challenges. These findings suggest that, despite the development and availability of more rigorous statistical models, heterogeneity appeared to be incomplete, inconsistently evaluated, or methodologically questionable in many cases, which lessened the interpretability of the analyses performed; comprehensive heterogeneity assessment should be addressed at the author level by improving personal familiarity with appropriate statistical methodology for assessing heterogeneity and involving biostatisticians and epidemiologists in study design, as well as at the editorial level, by mandating adherence to methodologic standards in primary DTA studies and DTA meta-analyses.

Project description:IntroductionSystemic lupus erythematosus (SLE) is a chronic autoimmune disease with multiorgan inflammatory involvement and a mortality rate that is 2.6-fold higher than individuals of the same age and sex in the general population. Approximately 50% of patients with SLE develop renal impairment (lupus nephritis). Delayed diagnosis of lupus nephritis is associated with a higher risk of progression to end-stage renal disease, the need for replacement therapy, and mortality. The initial clinical manifestations of lupus nephritis are often discrete or absent and are usually detected through complementary tests. Although widely used in clinical practice, their accuracy is limited. A great scientific effort has been exerted towards searching for new, more sensitive, and specific biomarkers in recent years. Some systematic reviews have individually evaluated new serum and urinary biomarkers tested in patients with lupus nephritis. This overview aimed to summarize systematic reviews on the accuracy of novel serum and urinary biomarkers for diagnosing lupus nephritis in patients with SLE, discussing how our results can guide the clinical management of the disease and the direction of research in this area.MethodsThe research question is "What is the accuracy of the new serum and urinary biomarkers studied for the diagnosis of LN in patients with SLE?". We searched for systematic reviews of observational studies evaluating the diagnostic accuracy of new serum or urinary biomarkers of lupus nephritis. The following databases were included: PubMed, EMBASE, BIREME/LILACS, Scopus, Web of Science, and Cochrane, including gray literature found via Google Scholar and PROQUEST. Two authors assessed the reviews for inclusion, data extraction, and assessment of the risk of bias (ROBIS tool).ResultsTen SRs on the diagnostic accuracy of new serum and urinary BMs in LN were selected. The SRs evaluated 7 distinct BMs: (a) antibodies (anti-Sm, anti-RNP, and anti-C1q), (b) cytokines (TWEAK and MCP-1), (c) a chemokine (IP-10), and (d) an acute phase glycoprotein (NGAL), in a total of 20 review arms (9 that analyzed serum BMs, and 12 that analyzed BMs in urine). The population evaluated in the primary studies was predominantly adults. Two SRs included strictly adults, 5 reviews also included studies in the paediatric population, and 4 did not report the age groups. The results of the evaluation with the ROBIS tool showed that most of the reviews had a low overall risk of bias.ConclusionsThere are 10 SRs of evidence relating to the diagnostic accuracy of serum and urinary biomarkers for lupus nephritis. Among the BMs evaluated, anti-C1q, urinary MCP-1, TWEAK, and NGAL stood out, highlighting the need for additional research, especially on LN diagnostic panels, and attempting to address methodological issues within diagnostic accuracy research. This would allow for a better understanding of their usefulness and possibly validate their clinical use in the future.RegistrationThis project is registered on the International Prospective Registry of Systematic Reviews (PROSPERO) database (CRD42020196693).

Project description:BACKGROUND:A pretest probability must be selected to calculate data to help clinicians, guideline boards and policy makers interpret diagnostic accuracy parameters. When multiple analyses for the same target condition are compared, identical pretest probabilities might be selected to facilitate the comparison. Some pretest probabilities may lead to exaggerations of the patient harms or benefits, and guidance on how and why to select a specific pretest probability is minimally described. Therefore, the aim of this study was to assess the data sources and methods used in Cochrane diagnostic test accuracy (DTA) reviews for determining pretest probabilities to facilitate the interpretation of DTA parameters. A secondary aim was to assess the use of identical pretest probabilities to compare multiple meta-analyses within the same target condition. METHODS:Cochrane DTA reviews presenting at least one meta-analytic estimate of the sensitivity and/or specificity as a primary analysis published between 2008 and January 2018 were included. Study selection and data extraction were performed by one author and checked by other authors. Observed data sources (e.g. studies in the review, or external sources) and methods to select pretest probabilities (e.g. median) were categorized. RESULTS:Fifty-nine DTA reviews were included, comprising of 308 meta-analyses. A pretest probability was used in 148 analyses. Authors used included studies in the DTA review, external sources, and author consensus as data sources for the pretest probability. Measures of central tendency with or without a measure of dispersion were used to determine the pretest probabilities, with the median most commonly used. Thirty-two target conditions had at least one identical pretest probability for all of the meta-analyses within their target condition. About half of the used identical pretest probabilities were inside the prevalence ranges from all analyses within a target condition. CONCLUSIONS:Multiple sources and methods were used to determine (identical) pretest probabilities in Cochrane DTA reviews. Indirectness and severity of downstream consequences may influence the acceptability of the certainty in calculated data with pretest probabilities. Consider: whether to present normalized frequencies, the influence of pretest probabilities on normalized frequencies, and whether to use identical pretest probabilities for meta-analyses in a target condition.

Dataset Information

Data Extraction and Synthesis in Systematic Reviews of Diagnostic Test Accuracy: A Corpus for Automating and Evaluating the Process.

Publications

Data Extraction and Synthesis in Systematic Reviews of Diagnostic Test Accuracy: A Corpus for Automating and Evaluating the Process.

Similar Datasets

OmicsDI is part of the ELIXIR infrastructure

Tweets