Dataset Information

Minimizing signal detection time in postmarket sequential analysis: balancing positive predictive value and sensitivity.

ABSTRACT: PURPOSE:Outcome misclassification in retrospective epidemiologic analyses has been well-studied, but little is known about such misclassification with respect to sequential statistical analysis during surveillance of medical product-associated risks, a planned capability of the US Food and Drug Administration's Sentinel System. METHODS:Using a vaccine example, we model and simulate sequential database surveillance in an observational data network using a variety of outcome detection algorithms. We consider how these algorithms, as characterized by sensitivity and positive predictive value, impact the length of surveillance and timeliness of safety signal detection. We show investigators/users of these networks how they can perform preparatory study design calculations that consider outcome misclassification in sequential database surveillance. RESULTS:Non-differential outcome misclassification generates longer surveillance times and less timely safety signal detection as compared with the case of no misclassification. Inclusive algorithms characterized by high sensitivity but low positive predictive value outperform more narrow algorithms when detecting rare outcomes. This decision calculus may change considerably if medical chart validation procedures were required. CONCLUSIONS:These findings raise important questions regarding the design of observational data networks used for pharmacovigilance. Specifically, there are tradeoffs involved when choosing to populate such networks with component databases that are large as compared with smaller integrated delivery system databases that can more easily access laboratory or clinical data and perform medical chart validation.

SUBMITTER: Maro JC

PROVIDER: S-EPMC7181336 | biostudies-literature | 2014 Aug

REPOSITORIES: biostudies-literature

ACCESS DATA

Publications

Minimizing signal detection time in postmarket sequential analysis: balancing positive predictive value and sensitivity.

Maro Judith C JC Brown Jeffrey S JS Dal Pan Gerald J GJ Kulldorff Martin M

Pharmacoepidemiology and drug safety 20140403 8

<h4>Purpose</h4>Outcome misclassification in retrospective epidemiologic analyses has been well-studied, but little is known about such misclassification with respect to sequential statistical analysis during surveillance of medical product-associated risks, a planned capability of the US Food and Drug Administration's Sentinel System.<h4>Methods</h4>Using a vaccine example, we model and simulate sequential database surveillance in an observational data network using a variety of outcome detecti ...[more]

PMID: 24700557

Similar Datasets

Project description:To evaluate the positive predictive values (PPVs) of Breast Imaging and Reporting Data Systems (BI-RADS) assessment categories for breast magnetic resonance (MR) imaging and to identify the BI-RADS MR imaging lesion features most predictive of cancer.This institutional review board-approved HIPAA-compliant prospective multicenter study was performed with written informed consent. Breast MR imaging studies of the contralateral breast in women with a recent diagnosis of breast cancer were prospectively evaluated. Contralateral breast MR imaging BI-RADS assessment categories, morphologic descriptors for foci, masses, non-masslike enhancement (NMLE), and kinetic features were assessed for predictive values for malignancy. PPV of each imaging characteristic of interest was estimated, and logistic regression analysis was used to examine the predictive ability of combinations of characteristics.Of 969 participants, 71.3% had a BI-RADS category 1 or 2 assessment; 10.9%, a BI-RADS category 3 assessment; 10.0%, a BI-RADS category 4 or 5 assessment; and 7.7%, a BI-RADS category 0 assessment on the basis of initial MR images. Thirty-one cancers were detected with MR imaging. Overall PPV for BI-RADS category 4 and 5 lesions was 0.278, with 17 cancers in patients with a BI-RADS category 4 lesion (PPV, 0.205) and 10 cancers in patients with a BI-RADS category 5 lesion (PPV, 0.714). Of the cancers, one was a focus, 17 were masses, and 13 were NMLEs. For masses, irregular shape, irregular margins, spiculated margins, and marked internal enhancement were most predictive of malignancy. For NMLEs, ductal, clumped, and reticular or dendritic enhancement were the features most frequently seen with malignancy. Kinetic enhancement features were less predictive of malignancy than were morphologic features.Standardized terminology of the BI-RADS lexicon enables quantification of the likelihood of malignancy for MR imaging-detected lesions through careful evaluation of lesion features. In particular, BI-RADS assessment categories and morphologic descriptors for masses and NMLE were useful in estimating the probability of cancer.

Project description:BACKGROUND:The expression of suicidal ideation is considered to be an important warning sign for suicide. However, the predictive properties of suicidal ideation as a test of later suicide are unclear.AimsTo assess the strength of the association between suicidal ideation and later suicide measured by odds ratio (OR), sensitivity, specificity and positive predictive value (PPV). METHOD:We located English-language studies indexed in PubMed that reported the expression or non-expression of suicidal ideation among people who later died by suicide or did not. A random effects meta-analysis was used to assess the pooled OR, sensitivity, specificity and PPV of suicidal ideation for later suicide among groups of people from psychiatric and non-psychiatric settings. RESULTS:There was a moderately strong but highly heterogeneous association between suicidal ideation and later suicide (n = 71, OR = 3.41, 95% CI 2.59-4.49, 95% prediction interval 0.42-28.1, I2 = 89.4, Q-value = 661, d.f.(Q) = 70, P ?0.001). Studies conducted in primary care and other non-psychiatric settings had similar pooled odds to studies of current and former psychiatric patients (OR = 3.86 v. OR = 3.23, P = 0.7). The pooled sensitivity of suicidal ideation for later suicide was 41% (95% CI 35-48) and the pooled specificity was 86% (95% CI 76-92), with high between-study heterogeneity. Studies of suicidal ideation expressed by current and former psychiatric patients had a significantly higher pooled sensitivity (46% v. 22%) and lower pooled specificity (81% v. 96%) than studies conducted in non-psychiatric settings. The PPV among non-psychiatric cohorts (0.3%, 95% CI 0.1%-0.5%) was significantly lower (Q-value = 35.6, P < 0.001) than among psychiatric samples (3.9%, 95% CI 2.2-6.6). CONCLUSIONS:Estimates of the extent of the association between suicidal ideation and later suicide are limited by unexplained between-study heterogeneity. The utility of suicidal ideation as a test for later suicide is limited by a modest sensitivity and low PPV.Declaration interestM.M.L. and C.J.R. have provided expert evidence in civil, criminal and coronial matters. I.B.H. has been a Commissioner in Australia's National Mental Health Commission since 2012. He is the Co-Director, Health and Policy at the Brain and Mind Centre (BMC) University of Sydney. The BMC operates an early-intervention youth services at Camperdown under contract to Headspace. I.B.H. has previously led community-based and pharmaceutical industry-supported (Wyeth, Eli Lily, Servier, Pfizer, AstraZeneca) projects focused on the identification and better management of anxiety and depression. He is a Board Member of Psychosis Australia Trust and a member of Veterans Mental Health Clinical Reference group. He was a member of the Medical Advisory Panel for Medibank Private until October 2017. He is the Chief Scientific Advisor to, and an equity shareholder in, InnoWell. InnoWell has been formed by the University of Sydney and PricewaterhouseCoopers to administer the $30 M Australian Government Funded Project Synergy. Project Synergy is a 3-year programme for the transformation of mental health services through the use of innovative technologies.

Project description:Although medical school programs oriented toward postgraduate specialty training have the potential to reduce the duration and cost of medical school for US medical students, success depends on the ability of students to predict their postgraduate specialties. It is clear that first-year choices are poorly predictive, but it is not known when predictions become sufficiently reliable to support specialty-oriented learning programs. We therefore examined the predictive value of specialty preferences expressed at the ends of the first, second and third years of medical school and asked whether concurrent expressions of confidence in choices improved predictive ability. We also investigated the possibility that discrepancies between predicted and actual postgraduate specialty training were related to scores on an examination of knowledge in basic biomedical sciences required for US medical school graduation (the United States Medical Licensing Examination (USLME) Step 1 examination).We calculated positive and negative predictive values (PPV and NPV, respectively) for specialty choices and the sensitivity and specificity of asking for choices for 634 University of Colorado School of Medicine students who trained in 23 accredited residencies from 2011 through 2015. We examined the effect of confidence in first choices in 609 students, and in 334 students, sought an association between USMLE Step 1 scores and switching from postgraduate training specialties predicted at the end of year 2.The PPV of first choices improved from years 1 through 3. NPV was high throughout. PPVs of year 3 first choices ranged from 79% in Anesthesiology to 95% in Psychiatry. Expressions of confidence in first choices did not improve PPV. Sensitivity of asking for first choices increased with time; specificity was consistently high. USLME Step 1 scores were higher for students who ultimately trained in specialties more competitive than first-choice specialties at the end of year 2.Specialty-oriented learning programs during medical school must accommodate students who change career plans. The PPV of specialty first choices improves each year, but even year 3 predictions can be inaccurate with potential loss of students from specialty-specific programs. USMLE Step 1 scores appeared to affect career plans expressed at the end of year 2.

Project description:ImportanceMyelin oligodendrocyte glycoprotein-IgG1-associated disorder (MOGAD) is a distinct central nervous system-demyelinating disease. Positive results on MOG-IgG1 testing by live cell-based assays can confirm a MOGAD diagnosis, but false-positive results may occur.ObjectiveTo determine the positive predictive value (PPV) of MOG-IgG1 testing in a tertiary referral center.Design, setting, and participantsThis diagnostic study was conducted over 2 years, from January 1, 2018, through December 31, 2019. Patients in the Mayo Clinic who were consecutively tested for MOG-IgG1 by live cell-based flow cytometry during their diagnostic workup were included. Patients without research authorization were excluded.Main outcomes and measuresMedical records of patients who were tested were initially reviewed by 2 investigators blinded to MOG-IgG1 serostatus, and pretest probability was classified as high or low (suggestive of MOGAD or not). Testing of MOG-IgG1 used a live-cell fluorescence-activated cell-sorting assay; an IgG binding index value of 2.5 or more with an end titer of 1:20 or more was considered positive. Cases positive for MOG-IgG1 were independently designated by 2 neurologists as true-positive or false-positive results at last follow-up, based on current international recommendations on diagnosis or identification of alternative diagnoses; consensus was reached for cases in which disagreement existed.ResultsA total of 1617 patients were tested, and 357 were excluded. Among 1260 included patients tested over 2 years, the median (range) age at testing was 46 (0-98) years, and 792 patients were female (62.9%). A total of 92 of 1260 (7.3%) were positive for MOG-IgG1. Twenty-six results (28%) were designated as false positive by the 2 raters, with an overall agreement on 91 of 92 cases (99%) for true and false positivity. Alternative diagnoses included multiple sclerosis (n = 11), infarction (n = 3), B12 deficiency (n = 2), neoplasia (n = 2), genetically confirmed adrenomyeloneuropathy (n = 1), and other conditions (n = 7). The overall PPV (number of true-positive results/total positive results) was 72% (95% CI, 62%-80%) and titer dependent (PPVs: 1:1000, 100%; 1:100, 82%; 1:20-40, 51%). The median titer was higher with true-positive results (1:100 [range, 1:20-1:10000]) than false-positive results (1:40 [range, 1:20-1:100]; P < .001). The PPV was higher for children (94% [95% CI, 72%-99%]) vs adults (67% [95% CI, 56%-77%]) and patients with high pretest probability (85% [95% CI, 76%-92%]) vs low pretest probability (12% [95% CI, 3%-34%]). The specificity of MOG-IgG1 testing was 97.8%.Conclusions and relevanceThis study confirms MOG-IgG1 as a highly specific biomarker for MOGAD, but when using a cutoff of 1:20, it has a low PPV of 72%. Caution is advised in the interpretation of low titers among patients with atypical phenotypes, because ordering MOG-IgG1 in low pretest probability situations will increase the proportion of false-positive results.

Project description:Environmental DNA (eDNA) analysis has gained traction as a precise and cost-effective method for species and waterways management. To date, publications on eDNA protocol optimization have focused primarily on DNA yield. Therefore, it has not been possible to evaluate the cost and speed of specific components of the eDNA protocol, such as water filtration and DNA extraction method when designing or choosing an eDNA protocol. At the same time, these two parameters are essential for the experimental design of a project. Here we evaluate and rank 27 different eDNA protocols in the context of Chinook salmon (Oncorhynchus tshawytscha) eDNA detection in an estuarine environment. We present a comprehensive evaluation of multiple eDNA protocol parameters, balancing time, cost and DNA yield. We collected samples composed of 500 mL estuarine water from Deverton Slough (38°11'16.7"N 121°58'34.5"W) and 500 mL from tank water containing 1.3 juvenile Chinook Salmon per liter. Then, we compared extraction methods, filter types, use of inhibitor removal kit for DNA yield, processing time, and protocol cost. Lastly, we used an MCMC algorithm together with machine learning to understand the DNA yield of each step of the protocol as well as the interactions between those steps. Glass fiber filtration was to be the most resilient to high turbidites, filtering the samples in 2.32 ± 0.08 min instead of 14.16 ± 1.86 min and 6.72 ± 1.99 min for nitrocellulose and paper filter N1, respectively. The filtration DNA yield percentages for paper filter N1, glass fiber, and nitrocellulose were 0.00045 ± 0.00013, 0.00107 ± 0.00013, 0.00172 ± 0.00013. The DNA extraction yield percentage for QIagen, dipstick, NaOH, magnetic beads, and direct dipstick ranged from 0.047 ± 0.0388 to 0.475 ± 0.0357. For estuarine waters, which are challenging for eDNA studies due to high turbidity, variable salinity, and the presence of PCR inhibitors, we found that a protocol combining glass filters, magnetic beads, and an extra step for PCR inhibitor removal, is the method that best balances time, cost, and yield. In addition, we provide a generalized decision tree for determining the optimal eDNA protocol for other studies in aquatic systems. Our findings should be applicable to most aquatic environments and provide a clear guide for determining which eDNA protocol should be used under different study constraints.

Dataset Information

Minimizing signal detection time in postmarket sequential analysis: balancing positive predictive value and sensitivity.

Publications

Minimizing signal detection time in postmarket sequential analysis: balancing positive predictive value and sensitivity.

Similar Datasets

OmicsDI is part of the ELIXIR infrastructure

Tweets