Developing clinical prediction models: a step-by-step guide.
Ontology highlight
ABSTRACT:
SUBMITTER: Efthimiou O
PROVIDER: S-EPMC11369751 | biostudies-literature | 2024 Sep
REPOSITORIES: biostudies-literature
Publications
Developing clinical prediction models: a step-by-step guide.
Efthimiou Orestis O Seo Michael M Chalkou Konstantina K Debray Thomas T Egger Matthias M Salanti Georgia G
BMJ (Clinical research ed.) 20240903
Similar Datasets
Project description:BackgroundMany chronic conditions, such as epilepsy and asthma, are typified by recurrent events-repeated acute deterioration events of a similar type. Statistical models for these conditions often focus on evaluating the time to the first event. They therefore do not make use of data available on all events. Statistical models for recurrent events exist, but it is not clear how best to evaluate their performance. We compare the relative performance of statistical models for analysing recurrent events for epilepsy and asthma.MethodsWe studied two clinical exemplars of common and infrequent events: asthma exacerbations using the Optimum Patient Clinical Research Database, and epileptic seizures using data from the Standard versus New Antiepileptic Drug Study. In both cases, count-based models (negative binomial and zero-inflated negative binomial) and variants on the Cox model (Andersen-Gill and Prentice, Williams and Peterson) were used to assess the risk of recurrence (of exacerbations or seizures respectively). Performance of models was evaluated via numerical (root mean square prediction error, mean absolute prediction error, and prediction bias) and graphical (calibration plots and Bland-Altman plots) approaches.ResultsThe performance of the prediction models for asthma and epilepsy recurrent events could be evaluated via the selected numerical and graphical measures. For both the asthma and epilepsy exemplars, the Prentice, Williams and Peterson model showed the closest agreement between predicted and observed outcomes.ConclusionInappropriate models can lead to incorrect conclusions which disadvantage patients. Therefore, prediction models for outcomes associated with chronic conditions should include all repeated events. Such models can be evaluated via the promoted numerical and graphical approaches alongside modified calibration measures.
| S-EPMC11912649 | biostudies-literature
Project description:Combining imaging and genetic information to predict disease presence and behavior is being codified into an emerging discipline called "radiogenomics." Optimal evaluation methodologies for radiogenomics techniques have not been established. We aim to develop a clinical decision framework based on utility analysis to assess prediction models for breast cancer. Our data comes from a retrospective case-control study, collecting Gail model risk factors, genetic variants (single nucleotide polymorphisms-SNPs), and mammographic features in Breast Imaging Reporting and Data System (BI-RADS) lexicon. We first constructed three logistic regression models built on different sets of predictive features: (1) Gail, (2) Gail+SNP, and (3) Gail+SNP+BI-RADS. Then, we generated ROC curves for three models. After we assigned utility values for each category of findings (true negative, false positive, false negative and true positive), we pursued optimal operating points on ROC curves to achieve maximum expected utility (MEU) of breast cancer diagnosis. We used McNemar's test to compare the predictive performance of the three models. We found that SNPs and BI-RADS features augmented the baseline Gail model in terms of the area under ROC curve (AUC) and MEU. SNPs improved sensitivity of the Gail model (0.276 vs. 0.147) and reduced specificity (0.855 vs. 0.912). When additional mammographic features were added, sensitivity increased to 0.457 and specificity to 0.872. SNPs and mammographic features played a significant role in breast cancer risk estimation (p-value < 0.001). Our decision framework comprising utility analysis and McNemar's test provides a novel framework to evaluate prediction models in the realm of radiogenomics.
| S-EPMC4834184 | biostudies-literature
Project description:Clinical prediction models (CPMs) can inform decision making about treatment initiation, which requires predicted risks assuming no treatment is given. However, this is challenging since CPMs are usually derived using data sets where patients received treatment, often initiated postbaseline as "treatment drop-ins." This study proposes the use of marginal structural models (MSMs) to adjust for treatment drop-in. We illustrate the use of MSMs in the CPM framework through simulation studies that represent randomized controlled trials and real-world observational data and the example of statin initiation for cardiovascular disease prevention. The simulations include a binary treatment and a covariate, each recorded at two timepoints and having a prognostic effect on a binary outcome. The bias in predicted risk was examined in a model ignoring treatment, a model fitted on treatment-naïve patients (at baseline), a model including baseline treatment, and the MSM. In all simulation scenarios, all models except the MSM underestimated the risk of outcome given absence of treatment. These results were supported in the statin initiation example, which showed that ignoring statin initiation postbaseline resulted in models that significantly underestimated the risk of a cardiovascular disease event occurring within 10 years. Consequently, CPMs that do not acknowledge treatment drop-in can lead to underallocation of treatment. In conclusion, when developing CPMs to predict treatment-naïve risk, researchers should consider using MSMs to adjust for treatment drop-in, and also seek to exploit the ability of MSMs to allow estimation of individual treatment effects.
| S-EPMC6282523 | biostudies-literature
Project description:BackgroundThe quality of communication between healthcare professionals (HCPs) and patients affects health outcomes. Different coding systems have been developed to unravel the interaction. Most schemes consist of predefined categories that quantify the content of communication (the what). Though the form (the how) of the interaction is equally important, protocols that systematically code variations in form are lacking. Patterns of form and how they may differ between groups therefore remain unnoticed. To fill this gap, we present CLECI, Coding Linguistic Elements in Clinical Interactions, a protocol for the development of a quantitative codebook analyzing communication form in medical interactions.MethodsAnalyzing with a CLECI codebook is a four-step process, i.e. preparation, codebook development, (double-)coding, and analysis and report. Core activities within these phases are research question formulation, data collection, selection of utterances, iterative deductive and inductive category refinement, reliability testing, coding, analysis, and reporting.Results and conclusionWe present step-by-step instructions for a CLECI analysis and illustrate this process in a case study. We highlight theoretical and practical issues as well as the iterative codebook development which combines theory-based and data-driven coding. Theory-based codes assess how relevant linguistic elements occur in natural interactions, whereas codes derived from the data accommodate linguistic elements to real-life interactions and contribute to theory-building. This combined approach increases research validity, enhances theory, and adjusts to fit naturally occurring data. CLECI will facilitate the study of communication form in clinical interactions and other institutional settings.
| S-EPMC9277943 | biostudies-literature
Project description:Existing dementia prediction models using non-neuroimaging clinical measures have been limited in their ability to identify disease. This study used machine learning to re-examine the diagnostic potential of clinical measures for dementia. Data was sourced from the Australian Imaging, Biomarkers, and Lifestyle Flagship Study of Ageing (AIBL) and the Alzheimer's Disease Neuroimaging Initiative (ADNI). Clinical variables included 21 measures across medical history, hematological and other blood tests, and APOE genotype. Tree-based machine learning algorithms and artificial neural networks were used. APOE genotype was the best predictor of dementia cases and healthy controls. Our results, however, demonstrated that there are limitations when using publicly accessible cohort data that may limit the generalizability and interpretability of such predictive models. Future research should examine the use of routine APOE genetic testing for dementia diagnostics. It should also focus on clearly unifying data across clinical cohorts.
| S-EPMC11751436 | biostudies-literature
Project description:ObjectiveTo document a step-by-step guide of crossed transeptal vasoepididymostomy with prioritization of crossover vas deferens measurement.DesignVideo presentation.SettingUniversity of Miami Lennar Surgical Center.PatientsThe patient undergoing this procedure signed a written, informed consent for video and audio recording.ResultsNo intraoperative complications were seen during the surgery, and the patient was discharged 6 h after the procedure. Through the steps detailed in the video and manuscript, a proper tubule into vas deferens invagination was formed.ConclusionsAnatomical indications for crossed vasoepididymostomy exists in 6% of azoospermic males and procedural prevalence is increasing. With a standardized step-by-step procedural approach that prioritizes crossover vas deferens measurement, crossed vasoepididymostomy can be considered in couples desiring natural conception, with previous complications of ART or failed ART.
| S-EPMC8291749 | biostudies-literature
Project description:Background There is a need to develop electronic health record-based predictive models for worsening heart failure (WHF) events across clinical settings and across the spectrum of left ventricular ejection fraction (LVEF). Methods and Results We studied adults with heart failure (HF) from 2011 to 2019 within an integrated health care delivery system. WHF encounters were ascertained using natural language processing and structured data. We conducted boosted decision tree ensemble models to predict 1-year hospitalizations, emergency department visits/observation stays, and outpatient encounters for WHF and all-cause death within each LVEF category: HF with reduced ejection fraction (EF) (LVEF <40%), HF with mildly reduced EF (LVEF 40%-49%), and HF with preserved EF (LVEF ≥50%). Model discrimination was evaluated using area under the curve and calibration using mean squared error. We identified 338 426 adults with HF: 61 045 (18.0%) had HF with reduced EF, 49 618 (14.7%) had HF with mildly reduced EF, and 227 763 (67.3%) had HF with preserved EF. The 1-year risks of any WHF event and death were, respectively, 22.3% and 13.0% for HF with reduced EF, 17.0% and 10.1% for HF with mildly reduced EF, and 16.3% and 10.3% for HF with preserved EF. The WHF model displayed an area under the curve of 0.76 and mean squared error of 0.13, whereas the model for death displayed an area under the curve of 0.83 and mean squared error of 0.076. Performance and predictors were similar across WHF encounter types and LVEF categories. Conclusions We developed risk prediction models for 1-year WHF events and death across the LVEF spectrum using structured and unstructured electronic health record data and observed no substantial differences in model performance or predictors except for death, despite differences in underlying HF cause.
| S-EPMC10727243 | biostudies-literature
Project description:Radical prostatectomy remains an important means to treat prostate cancer. A major limiting factor to radical prostatectomy is short- and long-term complications, especially incontinence and sexual dysfunction. With the advent of robotic radical prostatectomy, the ability to easily evaluate technical issues with video has been realized. In this article, we present a step-by-step examination of our procedure and our results over the past 5 years.
| S-EPMC6071518 | biostudies-literature
Project description:Pedigree information is often missing for some animals in a breeding program. Unknown-parent groups (UPGs) are assigned to the missing parents to avoid biased genetic evaluations. Although the use of UPGs is well established for the pedigree model, it is unclear how UPGs are integrated into the inverse of the unified relationship matrix (H-inverse) required for single-step genomic best linear unbiased prediction. A generalization of the UPG model is the metafounder (MF) model. The objectives of this study were to derive 3 H-inverses and to compare genetic trends among models with UPG and MF H-inverses using a simulated purebred population. All inverses were derived using the joint density function of the random breeding values and genetic groups. The breeding values of genotyped animals (u2) were assumed to be adjusted for UPG effects (g) using matrix Q2 as u2∗=u2+Q2g before incorporating genomic information. The Quaas-Pollak-transformed (QP) H-inverse was derived using a joint density function of u2∗ and g updated with genomic information and assuming nonzero cov(u2∗,g'). The modified QP (altered) H-inverse also assumes that the genomic information updates u2∗ and g, but cov(u2∗,g')=0. The UPG-encapsulated (EUPG) H-inverse assumed genomic information updates the distribution of u2∗. The EUPG H-inverse had the same structure as the MF H-inverse. Fifty percent of the genotyped females in the simulation had a missing dam, and missing parents were replaced with UPGs by generation. The simulation study indicated that u2∗ and g in models using the QP and altered H-inverses may be inseparable leading to potential biases in genetic trends. Models using the EUPG and MF H-inverses showed no genetic trend biases. These 2 H-inverses yielded the same genomic EBV (GEBV). The predictive ability and inflation of GEBVs from young genotyped animals were nearly identical among models using the QP, altered, EUPG, and MF H-inverses. Although the choice of H-inverse in real applications with enough data may not result in biased genetic trends, the EUPG and MF H-inverses are to be preferred because of theoretical justification and possibility to reduce biases.
| S-EPMC7896628 | biostudies-literature
Project description:BackgroundGuidelines recommend that clinicians use clinical prediction models to estimate future risk to guide decisions. For example, predicted fracture risk is a major factor in the decision to initiate bisphosphonate medications. However, current methods for developing prediction models often lead to models that are accurate but difficult to use in clinical settings.ObjectiveThe objective of this study was to develop and test whether a new metric that explicitly balances model accuracy with clinical usability leads to accurate, easier-to-use prediction models.MethodsWe propose a new metric called the Time-cost Information Criterion (TCIC) that will penalize potential predictor variables that take a long time to obtain in clinical settings. To demonstrate how the TCIC can be used to develop models that are easier-to-use in clinical settings, we use data from the 2000 wave of the Health and Retirement Study (n=6311) to develop and compare time to mortality prediction models using a traditional metric (Bayesian Information Criterion or BIC) and the TCIC.ResultsWe found that the TCIC models utilized predictors that could be obtained more quickly than BIC models while achieving similar discrimination. For example, the TCIC identified a 7-predictor model with a total time-cost of 44 seconds, while the BIC identified a 7-predictor model with a time-cost of 119 seconds. The Harrell C-statistic of the TCIC and BIC 7-predictor models did not differ (0.7065 vs. 0.7088, P=0.11).ConclusionAccounting for the time-costs of potential predictor variables through the use of the TCIC led to the development of an easier-to-use mortality prediction model with similar discrimination.
| S-EPMC8026517 | biostudies-literature