Approaches to lung nodule risk assessment: clinician intuition versus prediction models.
Ontology highlight
ABSTRACT:
SUBMITTER: Fox AH
PROVIDER: S-EPMC7330782 | biostudies-literature | 2020 Jun
REPOSITORIES: biostudies-literature
Similar Datasets
Project description:BackgroundClinical intuition and nonanalytic reasoning play a major role in clinical hypothesis generation; however, clinicians' intuition about whether a critically ill child is bacteremic has not been explored. We endeavored to assess pediatric critical care clinicians' ability to predict bacteremia and to evaluate what affected the accuracy of those predictions.MethodsWe conducted a retrospective review of clinicians' responses to a sepsis screening tool ("Early Sepsis Detection Tool" or "ESDT") over 6 months. The ESDT was completed during the initial evaluation of a possible sepsis episode. If a culture was ordered, they were asked to predict if the culture would be positive or negative. Culture results were compared to predictions for each episode as well as vital signs and laboratory data from the preceding 24 h.ResultsFrom January to July 2017, 266 ESDTs were completed. Of the 135 blood culture episodes, 15% of cultures were positive. Clinicians correctly predicted patients with bacteremia in 82% of cases, but the positive predictive value was just 28% as there was a tendency to overestimate the presence of bacteremia. The negative predictive value was 96%. The presence of bandemia, thrombocytopenia, and abnormal CRP were associated with increased likelihood of correct positive prediction.ConclusionsClinicians are accurate in predicting critically ill children whose blood cultures, obtained for symptoms of sepsis, will be negative. Clinicians frequently overestimate the presence of bacteremia. The combination of evidence-based practice guidelines and bedside judgment should be leveraged to optimize diagnosis of bacteremia.
| S-EPMC7330962 | biostudies-literature
Project description:Case-cohort studies are increasingly used to quantify the association of novel factors with disease risk. Conventional measures of predictive ability need modification for this design. We show how Harrell's C-index, Royston's D, and the category-based and continuous versions of the net reclassification index (NRI) can be adapted.We simulated full cohort and case-cohort data, with sampling fractions ranging from 1% to 90%, using covariates from a cohort study of coronary heart disease, and two incidence rates. We then compared the accuracy and precision of the proposed risk prediction metrics.The C-index and D must be weighted in order to obtain unbiased results. The NRI does not need modification, provided that the relevant non-subcohort cases are excluded from the calculation. The empirical standard errors across simulations were consistent with analytical standard errors for the C-index and D but not for the NRI. Good relative efficiency of the prediction metrics was observed in our examples, provided the sampling fraction was above 40% for the C-index, 60% for D, or 30% for the NRI. Stata code is made available.Case-cohort designs can be used to provide unbiased estimates of the C-index, D measure and NRI.
| S-EPMC3848813 | biostudies-other
Project description:We compare the calibration and variability of risk prediction models that were estimated using various approaches for combining information on new predictors, termed 'markers', with parameter information available for other variables from an earlier model, which was estimated from a large data source. We assess the performance of risk prediction models updated based on likelihood ratio (LR) approaches that incorporate dependence between new and old risk factors as well as approaches that assume independence ('naive Bayes' methods). We study the impact of estimating the LR by (i) fitting a single model to cases and non-cases when the distribution of the new markers is in the exponential family or (ii) fitting separate models to cases and non-cases. We also evaluate a new constrained maximum likelihood method. We study updating the risk prediction model when the new data arise from a cohort and extend available methods to accommodate updating when the new data source is a case-control study. To create realistic correlations between predictors, we also based simulations on real data on response to antiviral therapy for hepatitis C. From these studies, we recommend the LR method fit using a single model or constrained maximum likelihood. Copyright © 2016 John Wiley & Sons, Ltd.
| S-EPMC8182952 | biostudies-literature
Project description:ObjectivesTo determine if predictions of the Lung Cancer Prediction convolutional neural network (LCP-CNN) artificial intelligence (AI) model are analogous to the Brock model.MethodsIn total, 10,485 lung nodules in 4660 participants from the National Lung Screening Trial (NLST) were analysed. Both manual and automated nodule measurements were inputted into the Brock model, and this was compared to LCP-CNN. The performance of an experimental AI model was tested after ablating imaging features in a manner analogous to removing predictors from the Brock model. First, the nodule was ablated leaving lung parenchyma only. Second, a sphere of the same size as the nodule was implanted in the parenchyma. Third, internal texture of both nodule and parenchyma was ablated.ResultsAutomated axial diameter (AUC 0.883) and automated equivalent spherical diameter (AUC 0.896) significantly improved the accuracy of Brock when compared to manual measurement (AUC 0.873), although not to the level of the LCP-CNN (AUC 0.936). Ablating nodule and parenchyma texture (AUC 0.915) led to a small drop in AI predictive accuracy, as did implanting a sphere of the same size as the nodule (AUC 0.889). Ablating the nodule leaving parenchyma only led to a large drop in AI performance (AUC 0.717).ConclusionsFeature ablation is a feasible technique for understanding AI model predictions. Nodule size and morphology play the largest role in AI prediction, with nodule internal texture and background parenchyma playing a limited role. This is broadly analogous to the relative importance of morphological factors over clinical factors within the Brock model.Key points• Brock lung cancer risk prediction accuracy was significantly improved using automated axial or equivalent spherical measurements of lung nodule diameter, when compared to manual measurements. • Predictive accuracy was further improved by using the Lung Cancer Prediction convolutional neural network, an artificial intelligence-based model which obviates the requirement for nodule measurement. • Nodule size and morphology are important factors in artificial intelligence lung cancer risk prediction, with nodule texture and background parenchyma contributing a small, but measurable, role.
| S-EPMC9279235 | biostudies-literature
Project description:ObjectivesPost-imaging mathematical prediction models (MPMs) provide guidance for the management of solid pulmonary nodules by providing a lung cancer risk score from demographic and radiologists-indicated imaging characteristics. We hypothesized calibrating the MPM risk score threshold to a local study cohort would result in improved performance over the original recommended MPM thresholds. We compared the pre- and post-calibration performance of four MPM models and determined if improvement in MPM prediction occurs as nodules are imaged longitudinally.Materials and methodsA common cohort of 317 individuals with computed tomography-detected, solid nodules (80 malignant, 237 benign) were used to evaluate the MPM performance. We created a web-based application for this study that allows others to easily calibrate thresholds and analyze the performance of MPMs on their local cohort. Thirty patients with repeated imaging were tested for improved performance longitudinally.ResultsUsing calibrated thresholds, Mayo Clinic and Brock University (BU) MPMs performed the best (AUC = 0.63, 0.61) compared to the Veteran's Affairs (0.51) and Peking University (0.55). Only BU had consensus with the original MPM threshold; the other calibrated thresholds improved MPM accuracy. No significant improvements in accuracy were found longitudinally between time points.ConclusionsCalibration to a common cohort can select the best-performing MPM for your institution. Without calibration, BU has the most stable performance in solid nodules ≥ 8 mm but has only moderate potential to refine subjects into appropriate workup. Application of MPM is recommended only at initial evaluation as no increase in accuracy was achieved over time.Key points• Post-imaging lung cancer risk mathematical predication models (MPMs) perform poorly on local populations without calibration. • An application is provided to facilitate calibration to new study cohorts: the Mayo Clinic model, the U.S. Department of Veteran's Affairs model, the Brock University model, and the Peking University model. • No significant improvement in risk prediction occurred in nodules with repeated imaging sessions, indicating the potential value of risk prediction application is limited to the initial evaluation.
| S-EPMC6717521 | biostudies-literature
Project description:BackgroundSurgical risk prediction models traditionally use patient attributes and measures of physiology to generate predictions about postoperative outcomes. However, the surgeon's assessment of the patient may be a valuable predictor, given the surgeon's ability to detect and incorporate factors that existing models cannot capture. We compare the predictive utility of surgeon intuition and a risk calculator derived from the American College of Surgeons (ACS) NSQIP.Study designFrom January 10, 2021 to January 9, 2022, surgeons were surveyed immediately before performing surgery to assess their perception of a patient's risk of developing any postoperative complication. Clinical data were abstracted from ACS NSQIP. Both sources of data were independently used to build models to predict the likelihood of a patient experiencing any 30-day postoperative complication as defined by ACS NSQIP.ResultsPreoperative surgeon assessment was obtained for 216 patients. NSQIP data were available for 9,182 patients who underwent general surgery (January 1, 2017 to January 9, 2022). A binomial regression model trained on clinical data alone had an area under the receiver operating characteristic curve (AUC) of 0.83 (95% CI 0.80 to 0.85) in predicting any complication. A model trained on only preoperative surgeon intuition had an AUC of 0.70 (95% CI 0.63 to 0.78). A model trained on surgeon intuition and a subset of clinical predictors had an AUC of 0.83 (95% CI 0.77 to 0.89).ConclusionsPreoperative surgeon intuition alone is an independent predictor of patient outcomes; however, a risk calculator derived from ACS NSQIP is a more robust predictor of postoperative complication. Combining intuition and clinical data did not strengthen prediction.
| S-EPMC10192014 | biostudies-literature
Project description:BackgroundA calibration phantom-based method has been developed for predicting small lung nodule volume measurement bias and precision that is specific to a particular computed tomography (CT) scanner and acquisition protocol.MethodsThe approach involves CT scanning a simple reference object with a specific acquisition protocol, analyzing the scan to estimate the fundamental imaging properties of the CT acquisition system, generating numerous simulated images of a target geometry using the fundamental imaging properties, measuring the simulated images with a standard nodule volume segmentation algorithm, and calculating bias and precision performance statistics from the resulting volume measurements. We evaluated the ability of this approach to predict volume measurement bias and precision of Teflon spheres (diameters =4.76, 6.36, and 7.94 mm) placed within an anthropomorphic chest phantom when using 3M Scotch Magic™ tape as the reference object. CT scanning of the spheres was performed with 0.625, 1.25, and 2.5 mm slice thickness and spacing.ResultsThe study demonstrated good agreement between predicted volumetric performance and observed volume measurement performance for both volumetric measurement bias and precision. The predicted and observed volume mean for all slice thicknesses was found to be 28% and 13% lower on average than the manufactured sphere volume, respectively. When restricted to 0.625 and 1.25 mm slice thickness scans, which are recommended for small lung nodule volume measurement, we found that the difference between predicted and observed volume coefficient of variation was less than 1.0 %. The approach also showed a resilience to varying CT image acquisition protocols, a critical capability when deploying in a real-world clinical setting.ConclusionsThis is the first report of a calibration phantom-based method's ability to predict both small lung nodule volume measurement bias and precision. Volume measurement bias and precision for small lung nodules can be predicted using simple low-cost reference objects to estimate fundamental CT image characteristics and modeling and simulation techniques. The approach demonstrates an improved method for predicting task specific, clinically relevant measurement performance using advanced and fully automated image analysis techniques and low-cost reference objects.
| S-EPMC10498266 | biostudies-literature
Project description:Vaginitis is a common complaint, diagnosed either empirically or using Amsel's criteria and wet mount microscopy. This study sought to determine characteristics of an investigational test (a molecular test for vaginitis), compared to reference, for detection of bacterial vaginosis, Candida spp., and Trichomonas vaginalis Vaginal specimens from a cross-sectional study were obtained from 1,740 women (≥18 years old), with vaginitis symptoms, during routine clinic visits (across 10 sites in the United States). Specimens were analyzed using a commercial PCR/fluorogenic probe-based investigational test that detects bacterial vaginosis, Candida spp., and Trichomonas vaginalis Clinician diagnosis and in-clinic testing (Amsel's test, potassium hydroxide preparation, and wet mount) were also employed to detect the three vaginitis causes. All testing methods were compared to the respective reference methods (Nugent Gram stain for bacterial vaginosis, detection of the Candida gene its2, and Trichomonas vaginalis culture). The investigational test, clinician diagnosis, and in-clinic testing were compared to reference methods for bacterial vaginosis, Candida spp., and Trichomonas vaginalis The investigational test resulted in significantly higher sensitivity and negative predictive value than clinician diagnosis or in-clinic testing. In addition, the investigational test showed a statistically higher overall percent agreement with each of the three reference methods than did clinician diagnosis or in-clinic testing. The investigational test showed significantly higher sensitivity for detecting vaginitis, involving more than one cause, than did clinician diagnosis. Taken together, these results suggest that a molecular investigational test can facilitate accurate detection of vaginitis.
| S-EPMC5971525 | biostudies-literature
Project description:Lung adenocarcinoma (ADC), the most common lung cancer type, is recognized increasingly as a disease spectrum. To guide individualized patient care, a non-invasive means of distinguishing indolent from aggressive ADC subtypes is needed urgently. Computer-Aided Nodule Assessment and Risk Yield (CANARY) is a novel computed tomography (CT) tool that characterizes early ADCs by detecting nine distinct CT voxel classes, representing a spectrum of lepidic to invasive growth, within an ADC. CANARY characterization has been shown to correlate with ADC histology and patient outcomes. This study evaluated the inter-observer variability of CANARY analysis. Three novice observers segmented and analyzed independently 95 biopsy-confirmed lung ADCs from Vanderbilt University Medical Center/Nashville Veterans Administration Tennessee Valley Healthcare system (VUMC/TVHS) and the Mayo Clinic (Mayo). Inter-observer variability was measured using intra-class correlation coefficient (ICC). The average ICC for all CANARY classes was 0.828 (95% CI 0.76, 0.895) for the VUMC/TVHS cohort, and 0.852 (95% CI 0.804, 0.901) for the Mayo cohort. The most invasive voxel classes had the highest ICC values. To determine whether nodule size influenced inter-observer variability, an additional cohort of 49 sub-centimeter nodules from Mayo were also segmented by three observers, with similar ICC results. Our study demonstrates that CANARY ADC classification between novice CANARY users has an acceptably low degree of variability, and supports the further development of CANARY for clinical application.
| S-EPMC5983856 | biostudies-literature
Project description:The National Lung Screening Trial (NLST) demonstrated that low dose computed tomography (LDCT) screening could reduce lung cancer mortality by 20% in high-risk individuals. The United States Preventive Services Task Force (USPSTF) and Centers for Medicare and Medicaid Services (CMS) approved lung cancer screening. The NLST, USPSTF and CMS define high risk as smoking ?30 pack-years, smoking within the past 15 years, and being ages 55-74, 55-80 or 55-77. Retrospective studies demonstrated selection using model-estimated risk is superior to NLST-like criteria: higher sensitivity and positive predictive value (PPV), more deaths averted and higher cost-effectiveness. Projects are underway that may additionally support use of risk to determine eligibility. Firstly, the International Lung Screen Trial (ILST) is prospectively enrolling 4,000 individuals for screening if individuals have PLCOm2012 model risk ?1.5% or are USPSTF+ve. Six-year follow-up will allow comparisons. Interim results support the risk approach. Secondly, Cancer Care Ontario started the Lung Cancer Screening Pilot for People at High Risk in order to find optimal design for province-wide programmatic screening. They are enrolling 3,000 individuals to screening based on PLCOm2012 risk ?2%. Some hesitation to recommend screening selection based on model risk comes from the observation that selected individuals are older, have more comorbidities, are expected to have fewer life years and quality-adjusted life years (QALY) and are more likely to die from competing causes. We show that 25.6% of NLST eligible smokers are at low risk (6-year lung cancer incidence proportion =0.008). This group will not benefit from screening but has lower age, fewer comorbidities and fewer competing causes of death. When they are excluded from the NLST+ve group, age, comorbidity count and competing causes of death are similar to those in the PLCOm2012+ve group. In some jurisdictions, model-based lung cancer screening selection needs to take into consideration the elevated risk in blacks and indigenous peoples.
| S-EPMC6037970 | biostudies-other