Dataset Information

Meta-analysis of test accuracy studies: an exploratory method for investigating the impact of missing thresholds.

ABSTRACT: Primary studies examining the accuracy of a continuous test evaluate its sensitivity and specificity at one or more thresholds. Meta-analysts then usually perform a separate meta-analysis for each threshold. However, the number of studies available for each threshold is often very different, as primary studies are inconsistent in the thresholds reported. Furthermore, of concern is selective reporting bias, because primary studies may be less likely to report a threshold when it gives low sensitivity and/or specificity estimates. This may lead to biased meta-analysis results. We developed an exploratory method to examine the potential impact of missing thresholds on conclusions from a test accuracy meta-analysis.Our method identifies studies that contain missing thresholds bounded between a pair of higher and lower thresholds for which results are available. The bounded missing threshold results (two-by-two tables) are then imputed, by assuming a linear relationship between threshold value and each of logit-sensitivity and logit-specificity. The imputed results are then added to the meta-analysis, to ascertain if original conclusions are robust. The method is evaluated through simulation, and application made to 13 studies evaluating protein:creatinine ratio (PCR) for detecting proteinuria in pregnancy with 23 different thresholds, ranging from one to seven per study.The simulation shows the imputation method leads to meta-analysis estimates with smaller mean-square error. In the PCR application, it provides 50 additional results for meta-analysis and their inclusion produces lower test accuracy results than originally identified. For example, at a PCR threshold of 0.16, the summary specificity is 0.80 when using the original data, but 0.66 when also including the imputed data. At a PCR threshold of 0.25, the summary sensitivity is reduced from 0.95 to 0.85 when additionally including the imputed data.The imputation method is a practical tool for researchers (often non-statisticians) to explore the potential impact of missing threshold results on their meta-analysis conclusions. Software is available to implement the method. In the PCR example, it revealed threshold results are vulnerable to the missing data, and so stimulates the need for advanced statistical models or, preferably, individual patient data from primary studies.

SUBMITTER: Riley RD

PROVIDER: S-EPMC4417327 | biostudies-literature | 2015

REPOSITORIES: biostudies-literature

ACCESS DATA

Publications

Meta-analysis of test accuracy studies: an exploratory method for investigating the impact of missing thresholds.

Riley Richard D RD Ahmed Ikhlaaq I Ensor Joie J Takwoingi Yemisi Y Kirkham Amanda A Morris R Katie RK Noordzij J Pieter JP Deeks Jonathan J JJ

Systematic reviews 20150204

<h4>Background</h4>Primary studies examining the accuracy of a continuous test evaluate its sensitivity and specificity at one or more thresholds. Meta-analysts then usually perform a separate meta-analysis for each threshold. However, the number of studies available for each threshold is often very different, as primary studies are inconsistent in the thresholds reported. Furthermore, of concern is selective reporting bias, because primary studies may be less likely to report a threshold when i ...[more]

PMID: 25652323

Similar Datasets

Project description:ObjectiveTest accuracy of faecal calprotectin (FC) testing in primary care is inconclusive. We aimed to assess the test accuracy of FC testing in primary care and compare it to secondary care estimates for the detection of inflammatory bowel disease (IBD).MethodsSystematic review and meta-analysis of test accuracy using a bivariate random effects model. We searched MEDLINE, EMBASE, Cochrane Library and Web of Science until 31 May 2017 and included studies from auto alerts up until 31 January 2018. Eligible studies measured FC levels in stool samples to detect IBD in adult patients with chronic (at least 6-8 weeks) abdominal symptoms in primary or secondary care. Risk of bias and applicability were assessed using the Quality Assessment of Diagnostic Accuracy Studies-2 criteria. We followed the protocol registered as PROSPERO CRD 42012003287.Results38 out of 2168 studies were eligible including five from primary care. Comparison of test accuracy by setting was precluded by extensive heterogeneity. Overall, summary estimates of sensitivity and specificity were not recorded. At a threshold of 50 µg/g, sensitivity from separate meta-analysis of four assay types ranged from 0.85 (95% CI 0.75 to 0.92) to 0.94 (95% CI 0.75 to 0.90) and specificity from 0.67 (95% CI 0.56 to 0.76) to 0.88 (95% CI 0.77 to 0.94). Across three different definitions of disease, sensitivity ranged from 0.80 (95% CI 0.76 to 0.84) to 0.97 (95% CI 0.91 to 0.99) and specificity from 0.67 (95% CI 0.58 to 0.75) to 0.76 (95% CI 0.66 to 0.84). Sensitivity appears to be lower in primary care and is further reduced at a revised threshold of 100 µg/g.ConclusionsConclusive estimates of sensitivity and specificity of FC testing in primary care for the detection of IBD are still missing. There is insufficient evidence in the published literature to support the decision to introduce FC testing in primary care. Studies evaluating FC testing in an appropriate primary care setting are needed.

Project description:BackgroundThe large and increasing number of new studies published each year is making literature identification in systematic reviews ever more time-consuming and costly. Technological assistance has been suggested as an alternative to the conventional, manual study identification to mitigate the cost, but previous literature has mainly evaluated methods in terms of recall (search sensitivity) and workload reduction. There is a need to also evaluate whether screening prioritization methods leads to the same results and conclusions as exhaustive manual screening. In this study, we examined the impact of one screening prioritization method based on active learning on sensitivity and specificity estimates in systematic reviews of diagnostic test accuracy.MethodsWe simulated the screening process in 48 Cochrane reviews of diagnostic test accuracy and re-run 400 meta-analyses based on a least 3 studies. We compared screening prioritization (with technological assistance) and screening in randomized order (standard practice without technology assistance). We examined if the screening could have been stopped before identifying all relevant studies while still producing reliable summary estimates. For all meta-analyses, we also examined the relationship between the number of relevant studies and the reliability of the final estimates.ResultsThe main meta-analysis in each systematic review could have been performed after screening an average of 30% of the candidate articles (range 0.07 to 100%). No systematic review would have required screening more than 2308 studies, whereas manual screening would have required screening up to 43,363 studies. Despite an average 70% recall, the estimation error would have been 1.3% on average, compared to an average 2% estimation error expected when replicating summary estimate calculations.ConclusionScreening prioritization coupled with stopping criteria in diagnostic test accuracy reviews can reliably detect when the screening process has identified a sufficient number of studies to perform the main meta-analysis with an accuracy within pre-specified tolerance limits. However, many of the systematic reviews did not identify a sufficient number of studies that the meta-analyses were accurate within a 2% limit even with exhaustive manual screening, i.e., using current practice.

Project description:BackgroundDiabetic retinopathy (DR), a common complication of diabetes mellitus, is the leading cause of impaired vision in adults worldwide. Smartphone ophthalmoscopy involves using a smartphone camera for digital retinal imaging. Utilizing smartphones to detect DR is potentially more affordable, accessible, and easier to use than conventional methods.ObjectiveThis study aimed to determine the diagnostic accuracy of various smartphone ophthalmoscopy approaches for detecting DR in diabetic patients.MethodsWe performed an electronic search on the Medical Literature Analysis and Retrieval System Online (MEDLINE), EMBASE, and Cochrane Library for literature published from January 2000 to November 2018. We included studies involving diabetic patients, which compared the diagnostic accuracy of smartphone ophthalmoscopy for detecting DR to an accurate or commonly employed reference standard, such as indirect ophthalmoscopy, slit-lamp biomicroscopy, and tabletop fundus photography. Two reviewers independently screened studies against the inclusion criteria, extracted data, and assessed the quality of included studies using the Quality Assessment of Diagnostic Accuracy Studies-2 tool, with disagreements resolved via consensus. Sensitivity and specificity were pooled using the random effects model. A summary receiver operating characteristic (SROC) curve was constructed. This review is reported in line with the Preferred Reporting Items for a Systematic Review and Meta-analysis of Diagnostic Test Accuracy Studies guidelines.ResultsIn all, nine studies involving 1430 participants were included. Most studies were of high quality, except one study with limited applicability because of its reference standard. The pooled sensitivity and specificity for detecting any DR was 87% (95% CI 74%-94%) and 94% (95% CI 81%-98%); mild nonproliferative DR (NPDR) was 39% (95% CI 10%-79%) and 95% (95% CI 91%-98%); moderate NPDR was 71% (95% CI 57%-81%) and 95% (95% CI 88%-98%); severe NPDR was 80% (95% CI 49%-94%) and 97% (95% CI 88%-99%); proliferative DR (PDR) was 92% (95% CI 79%-97%) and 99% (95% CI 96%-99%); diabetic macular edema was 79% (95% CI 63%-89%) and 93% (95% CI 82%-97%); and referral-warranted DR was 91% (95% CI 86%-94%) and 89% (95% CI 56%-98%). The area under SROC curve ranged from 0.879 to 0.979. The diagnostic odds ratio ranged from 11.3 to 1225.ConclusionsWe found heterogeneous evidence showing that smartphone ophthalmoscopy performs well in detecting DR. The diagnostic accuracy for PDR was highest. Future studies should standardize reference criteria and classification criteria and evaluate other available forms of smartphone ophthalmoscopy in primary care settings.

Dataset Information

Meta-analysis of test accuracy studies: an exploratory method for investigating the impact of missing thresholds.

Publications

Meta-analysis of test accuracy studies: an exploratory method for investigating the impact of missing thresholds.

Similar Datasets

OmicsDI is part of the ELIXIR infrastructure

Tweets