Dataset Information

Sample sizes and precision of estimates of sensitivity and specificity from primary studies on the diagnostic accuracy of depression screening tools: a survey of recently published studies.

ABSTRACT: Depression screening tools are useful to the extent that they accurately discriminate between depressed and non-depressed patients. Studies without enough patients to generate precise estimates make it difficult to evaluate accuracy. We conducted a survey of recently published studies on depression screening tool accuracy to evaluate the percentage with sample size calculations; the percentage that provided confidence intervals; and precision, based on the width and lower bounds of 95% confidence intervals for sensitivity and specificity. We calculated 95% confidence intervals, if possible, when not provided. Only three of 89 studies (3%) described a viable sample size calculation. Only 30 studies (34%) provided reasonably accurate confidence intervals. Of 86 studies where 95% confidence intervals were provided or could be calculated, only seven (8%) had interval widths for sensitivity of ? 10%, whereas 53 (62%) had widths of ? 21%. Lower bounds of confidence intervals were < 80% for 84% of studies for sensitivity and 66% of studies for specificity. Overall, few studies on the diagnostic accuracy of depression screening tools reported sample size calculations, and the number of patients in most studies was too small to generate reasonably precise accuracy estimates. The failure to provide confidence intervals in published reports may obscure these shortcomings. Copyright © 2016 John Wiley & Sons, Ltd.

SUBMITTER: Thombs BD

PROVIDER: S-EPMC6877233 | biostudies-literature | 2016 Jun

REPOSITORIES: biostudies-literature

ACCESS DATA

Publications

Sample sizes and precision of estimates of sensitivity and specificity from primary studies on the diagnostic accuracy of depression screening tools: a survey of recently published studies.

Thombs Brett D BD Rice Danielle B DB

International journal of methods in psychiatric research 20160408 2

Depression screening tools are useful to the extent that they accurately discriminate between depressed and non-depressed patients. Studies without enough patients to generate precise estimates make it difficult to evaluate accuracy. We conducted a survey of recently published studies on depression screening tool accuracy to evaluate the percentage with sample size calculations; the percentage that provided confidence intervals; and precision, based on the width and lower bounds of 95% confidenc ...[more]

PMID: 27060912

Similar Datasets

Project description:Background. National valuation studies are costly, with ∼1000 face-to-face interviews recommended, and some countries may deem such studies infeasible. Building on previous studies exploring sample size, we determined the effect of sample size and alternative model specifications on prediction accuracy of modeled coefficients in EQ-5D-5L value set generating regression analyses. Methods. Data sets (n = 50 to ∼1000) were simulated from 3 valuation studies, resampled at the respondent level and randomly drawn 1000 times with replacement. We estimated utilities for each subsample with leave-one-out at the block level using regression models (8 or 20 parameter; with or without a random intercept; time tradeoff [TTO] data only or TTO + discrete choice experiment [DCE] data). Prediction accuracy, root mean square error (RMSE), was calculated by comparing to censored mean predicted values to the left-out block in the full data set. Linear regression was used to estimate the relative effect of changes in sample size and each model specification. Results. Results showed that doubling the sample size decreased RMSE by on average 0.012. Effects of other model specifications were smaller but can when combined compensate for loss in prediction accuracy from a small sample size. For models using TTO data only, 8-parameter models clearly outperformed 20-parameter models. Adding a random intercept, or including DCE responses, also improved mean RMSE, most prominently for variants of the 20-parameter models. Conclusions. The prediction accuracy impact of further increases in sample size after 300 to 500 were smaller than the impact of combining alternative modeling choices. Hybrid modeling, use of constrained models, and inclusion of random intercepts all substantially improve the expected prediction accuracy. Beyond a minimum of 300 to 500 respondents, the sample size may be better informed by other considerations, such as legitimacy and representativeness, than by the technical prediction accuracy achievable.HighlightsIncreases in sample size beyond a minimum in the range of 300 to 500 respondents provide smaller gains in expected prediction accuracy than alternative modeling approaches.Constrained, nonlinear models; time tradeoff + discrete choice experiment hybrid modeling; and including a random intercept all improved the prediction accuracy of models estimating values for the EQ-5D-5L based on data from 3 different valuation studies.The tested modeling choices can compensate for smaller sample sizes.

Project description:Applicability of intravoxel incoherent motion (IVIM) imaging in the clinical setting is hampered by the limited reliability in particular of the perfusion-related parameter estimates. To alleviate this problem, various advanced postprocessing methods have been introduced. However, the underlying algorithms are not readily available and generally suffer from an increased computational burden. Contrary, several computationally fast image denoising methods have recently been proposed which are accessible online and may improve reliability of IVIM parameter estimates. The objective of the present work is to investigate the impact of image denoising on accuracy and precision of IVIM parameter estimates using comprehensive in-silico and in-vivo experiments. Image denoising is performed with four different algorithms that work on magnitude data: two algorithms which are based on nonlocal means (NLM) filtering, one algorithm that relies on local principal component analysis (LPCA) of the diffusion-weighted images, and another algorithms that exploits joint rank and edge constraints (JREC). Accuracy and precision of IVIM parameter estimates is investigated in an in-silico brain phantom and an in-vivo ground truth as a function of the signal-to-noise ratio for spatially homogenous and inhomogenous levels of Rician noise. Moreover, precision is evaluated using bootstrap analysis of in-vivo measurements. In the experiments, IVIM parameters are computed a) by using a segmented fit method and b) by performing a biexponential fit of the entire attenuation curve based on nonlinear least squares estimates. Irrespective of the fit method, the results demonstrate that reliability of IVIM parameter estimates is substantially improved by image denoising. The experiments show that the LPCA and the JREC algorithms perform in a similar manner and outperform the NLM-related methods. Relative to noisy data, accuracy of the IVIM parameters in the in-silico phantom improves after image denoising by 76-79%, 79-81%, 84-99% and precision by 74-80%, 80-83%, 84-95% for the perfusion fraction, the diffusion coefficient, and the pseudodiffusion coefficient, respectively, when the segmented fit method is used. Beyond that, the simulations reveal that denoising performance is not impeded by spatially inhomogeneous levels of Rician noise in the image. Since all investigated algorithms are freely available and work on magnitude data they can be readily applied in the clinical setting which may foster transition of IVIM imaging into clinical practice.

Project description:BackgroundEstimates of under-5 mortality at the national level for countries without high-quality vital registration systems are routinely derived from birth history data in censuses and surveys. Subnational or stratified analyses of under-5 mortality could also be valuable, but the usefulness of under-5 mortality estimates derived from birth histories from relatively small samples of women is not known. We aim to assess the magnitude and direction of error that can be expected for estimates derived from birth histories with small samples of women using various analysis methods.MethodsWe perform a data-based simulation study using Demographic and Health Surveys. Surveys are treated as populations with known under-5 mortality, and samples of women are drawn from each population to mimic surveys with small sample sizes. A variety of methods for analyzing complete birth histories and one method for analyzing summary birth histories are used on these samples, and the results are compared to corresponding true under-5 mortality. We quantify the expected magnitude and direction of error by calculating the mean error, mean relative error, mean absolute error, and mean absolute relative error.ResultsAll methods are prone to high levels of error at the smallest sample size with no method performing better than 73% error on average when the sample contains 10 women. There is a high degree of variation in performance between the methods at each sample size, with methods that contain considerable pooling of information generally performing better overall. Additional stratified analyses suggest that performance varies for most methods according to the true level of mortality and the time prior to survey. This is particularly true of the summary birth history method as well as complete birth history methods that contain considerable pooling of information across time.ConclusionsPerformance of all birth history analysis methods is extremely poor when used on very small samples of women, both in terms of magnitude of expected error and bias in the estimates. Even with larger samples there is no clear best method to choose for analyzing birth history data. The methods that perform best overall are the same methods where performance is noticeably different at different levels of mortality and lengths of time prior to survey. At the same time, methods that perform more uniformly across levels of mortality and lengths of time prior to survey also tend to be among the worst performing overall.

Project description:BackgroundMediation analysis can be used to evaluate the effect of an exposure on an outcome acting through an intermediate variable or mediator. For studies with small sample sizes, permutation testing may be useful in evaluating the indirect effect (i.e., the effect of exposure on the outcome through the mediator) while maintaining the appropriate type I error rate. For mediation analysis in studies with small sample sizes, existing permutation testing methods permute the residuals under the full or alternative model, but have not been evaluated under situations where covariates are included. In this article, we consider and evaluate two additional permutation approaches for testing the indirect effect in mediation analysis based on permutating the residuals under the reduced or null model which allows for the inclusion of covariates.MethodsSimulation studies were used to empirically evaluate the behavior of these two additional approaches: (1) the permutation test of the Indirect Effect under Reduced Models (IERM) and (2) the Permutation Supremum test under Reduced Models (PSRM). The performance of these methods was compared to the standard permutation approach for mediation analysis, the permutation test of the Indirect Effect under Full Models (IEFM). We evaluated the type 1 error rates and power of these methods in the presence of covariates since mediation analysis assumes no unmeasured confounders of the exposure-mediator-outcome relationships.ResultsThe proposed PSRM approach maintained type I error rates below nominal levels under all conditions, while the proposed IERM approach exhibited grossly inflated type I rates in many conditions and the standard IEFM exhibited inflated type I error rates under a small number of conditions. Power did not differ substantially between the proposed PSRM approach and the standard IEFM approach.ConclusionsThe proposed PSRM approach is recommended over the existing IEFM approach for mediation analysis in studies with small sample sizes.

Dataset Information

Sample sizes and precision of estimates of sensitivity and specificity from primary studies on the diagnostic accuracy of depression screening tools: a survey of recently published studies.

Publications

Sample sizes and precision of estimates of sensitivity and specificity from primary studies on the diagnostic accuracy of depression screening tools: a survey of recently published studies.

Similar Datasets

OmicsDI is part of the ELIXIR infrastructure

Tweets