Dataset Information

Design Characteristics Influence Performance of Clinical Prediction Rules in Validation: A Meta-Epidemiological Study.

ABSTRACT:

Background

Many new clinical prediction rules are derived and validated. But the design and reporting quality of clinical prediction research has been less than optimal. We aimed to assess whether design characteristics of validation studies were associated with the overestimation of clinical prediction rules' performance. We also aimed to evaluate whether validation studies clearly reported important methodological characteristics.

Methods

Electronic databases were searched for systematic reviews of clinical prediction rule studies published between 2006 and 2010. Data were extracted from the eligible validation studies included in the systematic reviews. A meta-analytic meta-epidemiological approach was used to assess the influence of design characteristics on predictive performance. From each validation study, it was assessed whether 7 design and 7 reporting characteristics were properly described.

Results

A total of 287 validation studies of clinical prediction rule were collected from 15 systematic reviews (31 meta-analyses). Validation studies using case-control design produced a summary diagnostic odds ratio (DOR) 2.2 times (95% CI: 1.2-4.3) larger than validation studies using cohort design and unclear design. When differential verification was used, the summary DOR was overestimated by twofold (95% CI: 1.2 -3.1) compared to complete, partial and unclear verification. The summary RDOR of validation studies with inadequate sample size was 1.9 (95% CI: 1.2 -3.1) compared to studies with adequate sample size. Study site, reliability, and clinical prediction rule was adequately described in 10.1%, 9.4%, and 7.0% of validation studies respectively.

Conclusion

Validation studies with design shortcomings may overestimate the performance of clinical prediction rules. The quality of reporting among studies validating clinical prediction rules needs to be improved.

SUBMITTER: Ban JW

PROVIDER: S-EPMC4701404 | biostudies-literature | 2016

REPOSITORIES: biostudies-literature

ACCESS DATA

Publications

Design Characteristics Influence Performance of Clinical Prediction Rules in Validation: A Meta-Epidemiological Study.

Ban Jong-Wook JW Emparanza José Ignacio JI Urreta Iratxe I Burls Amanda A

PloS one 20160105 1

<h4>Background</h4>Many new clinical prediction rules are derived and validated. But the design and reporting quality of clinical prediction research has been less than optimal. We aimed to assess whether design characteristics of validation studies were associated with the overestimation of clinical prediction rules' performance. We also aimed to evaluate whether validation studies clearly reported important methodological characteristics.<h4>Methods</h4>Electronic databases were searched for s ...[more]

PMID: 26730980

Similar Datasets

Project description:OBJECTIVES:To empirically assess the relation between study characteristics and prognostic model performance in external validation studies of multivariable prognostic models. DESIGN:Meta-epidemiological study. DATA SOURCES AND STUDY SELECTION:On 16 October 2018, we searched electronic databases for systematic reviews of prognostic models. Reviews from non-overlapping clinical fields were selected if they reported common performance measures (either the concordance (c)-statistic or the ratio of observed over expected number of events (OE ratio)) from 10 or more validations of the same prognostic model. DATA EXTRACTION AND ANALYSES:Study design features, population characteristics, methods of predictor and outcome assessment, and the aforementioned performance measures were extracted from the included external validation studies. Random effects meta-regression was used to quantify the association between the study characteristics and model performance. RESULTS:We included 10 systematic reviews, describing a total of 224 external validations, of which 221 reported c-statistics and 124 OE ratios. Associations between study characteristics and model performance were heterogeneous across systematic reviews. C-statistics were most associated with variation in population characteristics, outcome definitions and measurement and predictor substitution. For example, validations with eligibility criteria comparable to the development study were associated with higher c-statistics compared with narrower criteria (difference in logit c-statistic 0.21(95% CI 0.07 to 0.35), similar to an increase from 0.70 to 0.74). Using a case-control design was associated with higher OE ratios, compared with using data from a cohort (difference in log OE ratio 0.97(95% CI 0.38 to 1.55), similar to an increase in OE ratio from 1.00 to 2.63). CONCLUSIONS:Variation in performance of prognostic models across studies is mainly associated with variation in case-mix, study designs, outcome definitions and measurement methods and predictor substitution. Researchers developing and validating prognostic models should realise the potential influence of these study characteristics on the predictive performance of prognostic models.

Project description:IntroductionExternal validation studies are essential to study the generalizability of prediction models. Recently a permutation test, focusing on discrimination as quantified by the c-statistic, was proposed to judge whether a prediction model is transportable to a new setting. We aimed to evaluate this test and compare it to previously proposed procedures to judge any changes in c-statistic from development to external validation setting.MethodsWe compared the use of the permutation test to the use of benchmark values of the c-statistic following from a previously proposed framework to judge transportability of a prediction model. In a simulation study we developed a prediction model with logistic regression on a development set and validated them in the validation set. We concentrated on two scenarios: 1) the case-mix was more heterogeneous and predictor effects were weaker in the validation set compared to the development set, and 2) the case-mix was less heterogeneous in the validation set and predictor effects were identical in the validation and development set. Furthermore we illustrated the methods in a case study using 15 datasets of patients suffering from traumatic brain injury.ResultsThe permutation test indicated that the validation and development set were homogenous in scenario 1 (in almost all simulated samples) and heterogeneous in scenario 2 (in 17%-39% of simulated samples). Previously proposed benchmark values of the c-statistic and the standard deviation of the linear predictors correctly pointed at the more heterogeneous case-mix in scenario 1 and the less heterogeneous case-mix in scenario 2.ConclusionThe recently proposed permutation test may provide misleading results when externally validating prediction models in the presence of case-mix differences between the development and validation population. To correctly interpret the c-statistic found at external validation it is crucial to disentangle case-mix differences from incorrect regression coefficients.

Project description:BackgroundClinical prediction rules (CPRs) should be externally validated by independent researchers. Although there are many cardiovascular CPRs, most have not been externally validated. It is not known why some CPRs are externally validated by independent researchers and others are not.MethodsWe analyzed cardiovascular risk CPRs included in a systematic review. Independent external validations were identified by forward citation searches of derivation studies. Time between the publication of a cardiovascular CPR and the first independent external validation was calculated. We assessed Kaplan-Meier estimates of the probability to have an independent external validation. Using univariable Cox regression, we explored whether characteristics of derivation (design, location, sample size, number of predictors, presentation format, validation in derivation), reporting (participants, predictors, outcomes, performance measure, information for risk calculation), and publication (journal impact factor) are associated with time to the first independent external validation.ResultsOf 125 cardiovascular risk CPRs, 29 had an independent external validation. The median follow-up was 118 months (95% CI, 99-130). The 25th percentile of event time was 122 months (95% CI, 91-299). Cardiovascular risk CPRs from the USA were 4.15 times (95% CI, 1.89-9.13) more likely to have an independent external validation. Increasing the sample size of derivation by ten times was associated with a 2.32-fold (95% CI, 1.37-3.91) increase in the probability of having an independent external validation. CPRs presented with an internal validation tend to get an independent external validation sooner (HR = 1.73, 95% CI, 0.77-3.93). CPRs reporting all the information necessary for calculating individual risk were 2.65 (95% CI, 1.01-6.96) times more likely to have an independent external validation. Publishing a cardiovascular risk CPR in a journal that has one unit higher impact factor was associated with a 6% (95% CI, 3-9) higher likelihood of an independent external validation.ConclusionsThe probability for cardiovascular risk CPRs to get an independent external validation was low even many years after their derivations. Authors of new cardiovascular risk CPRs should consider using adequate sample size, conducting an internal validation, and reporting all the information needed for risk calculation as these features were associated with an independent external validation.

Project description:BACKGROUND: During an influenza epidemic prompt diagnosis of influenza is important. This diagnosis however is still essentially based on the interpretation of symptoms and signs by general practitioners. No single symptom is specific enough to be useful in differentiating influenza from other respiratory infections. Our objective is to formulate prediction rules for the diagnosis of influenza with the best diagnostic performance, combining symptoms, signs and context among patients with influenza-like illness. METHODS: During five consecutive winter periods (2002-2007) 138 sentinel general practitioners sampled (naso- and oropharyngeal swabs) 4597 patients with an influenza-like illness (ILI) and registered their symptoms and signs, general characteristics and contextual information. The samples were analysed by a DirectigenFlu-A&B and RT-PCR tests. 4584 records were useful for further analysis.Starting from the most relevant variables in a Generalized Estimating Equations (GEE) model, we calculated the area under the Receiver Operating Characteristic curve (ROC AUC), sensitivity, specificity and likelihood ratios for positive (LR+) and negative test results (LR-) of single and combined signs, symptoms and context taking into account pre-test and post-test odds. RESULTS: In total 52.6% (2409/4584) of the samples were positive for influenza virus: 64% (2066/3212) during and 25% (343/1372) pre/post an influenza epidemic. During and pre/post an influenza epidemic the LR+ of 'previous flu-like contacts', 'coughing', 'expectoration on the first day of illness' and 'body temperature above 37.8°C' is 3.35 (95%CI 2.67-4.03) and 1.34 (95%CI 0.97-1.72), respectively. During and pre/post an influenza epidemic the LR- of 'coughing' and 'a body temperature above 37.8°C' is 0.34 (95%CI 0.27-0.41) and 0.07 (95%CI 0.05-0.08), respectively. CONCLUSIONS: Ruling out influenza using clinical and contextual information is easier than ruling it in. Outside an influenza epidemic the absence of cough and fever (> 37,8°C) makes influenza 14 times less likely in ILI patients. During an epidemic the presence of 'previous flu-like contacts', cough, 'expectoration on the first day of illness' and fever (>37,8°C) increases the likelihood for influenza threefold. The additional diagnostic value of rapid point of care tests especially for confirming influenza still has to be established.

Project description:ImportanceDelirium at admission is associated with increased hospital morbidity and mortality, but it may be missed in up to 70% of cases. Use of a predictive algorithm in an electronic medical record (EMR) system could provide critical information to target assessment of those with delirium at admission.ObjectivesTo develop and assess a prediction rule for delirium using 2 populations of veterans and compare this rule with previously confirmed rules.Design, setting, and participantsIn a diagnostic study, randomly selected EMRs of hospitalized veterans from the Veterans Affairs (VA) External Peer Review Program at 118 VA medical centers with inpatient facilities were reviewed for delirium risk factors associated with the National Institute for Health and Clinical Excellence (NICE) delirium rule in a derivation cohort (October 1, 2012, to September 30, 2013) and a confirmation cohort (October 1, 2013, to March 31, 2014). Delirium within 24 hours of admission was identified using key word terms. A total of 39 377 veterans 65 years or older who were admitted to a VA medical center for congestive heart failure, acute coronary syndrome, community-acquired pneumonia, and chronic obstructive pulmonary disease were included in the study.ExposureThe EMR calculated delirium risk.Main outcomes and measuresDelirium at admission as identified by trained nurse reviewers was the main outcome measure. Random forest methods were used to identify accurate risk factors for prevalent delirium. A prediction rule for prevalent delirium was developed, and its diagnostic accuracy was tested in the confirmation cohort. This consolidated NICE rule was compared with previously confirmed scoring algorithms (electronic NICE and Pendlebury NICE).ResultsA total of 27?625 patients were included in the derivation cohort (28 118 [92.2%] male; mean [SD] age, 75.95 [8.61] years) and 11 752 in the confirmation cohort (11 536 [98.2%] male; mean [SD] age, 75.43 [8.55] years). Delirium at admission was identified in 2343 patients (8.5%) in the derivation cohort and 882 patients (7.0%) in the confirmation cohort. Modeling techniques identified cognitive impairment, infection, sodium level, and age of 80 years or older as the dominant risk factors. The consolidated NICE rule (area under the receiver operating characteristic [AUROC] curve, 0.91; 95% CI, 0.91-0.92; P?<?.001) had significantly higher discriminatory function than the eNICE rule (AUROC curve, 0.81; 95% CI, 0.80-0.82; P?<?.001) or Pendlebury NICE rule (AUROC curve, 0.87; 95% CI, 0.86-0.88; P?<?.001). These findings were confirmed in the confirmation cohort.Conclusions and relevanceThis analysis identified preexisting cognitive impairment, infection, sodium level, and age of 80 years or older as delirium screening targets. Use of this algorithm in an EMR system could direct clinical assessment efforts to patients with delirium at admission.

Project description:BACKGROUND:Kawasaki disease (KD) is an acute multisystem vascular syndrome of unknown etiology that is the leading cause of acquired heart disease in children of developed counties. OBJECTIVES:We aimed to evaluate the epidemiological characteristics and clinical manifestations of KD in children residing in the southwest of Iran. PATIENTS AND METHODS:In this retrospective study, we reviewed the medical records of all children with KD who had been admitted to the main children's hospital of Ahvaz, southwest Iran, from March 2000 to March 2010. Data regarding clinical and epidemiological characteristics, management, and the outcome of disease for each patient were obtained. The patients were divided into cardiac and non-cardiac groups based on echocardiographic results. RESULTS:In total, 104 patients with KD (66 boys and 38 girls) were enrolled in this study. The male to female ratio was 1.7:1. The mean ± SD age of the patients was 33.6 ± 24.2 months. Most (87.2%) cases were from urban areas. The disease occurred more frequently during winter and spring. Furthermore, 61.5% of the children had the criteria of classic KD, and 38.5% were labeled as incomplete KD. The mean ± SD of the duration of hospital stay was 6.9 ± 2.4 days. The mean time between illness and admission to the hospital was 6.47 ± 2.6 days. The most common sign was fever, followed by conjunctivitis and oral changes. In total, 20% of the patients had cardiac abnormalities. There was no significant statistical difference between the cardiac and non-cardiac groups according to age, sex, clinical manifestations, laboratory findings, and cessation of fever. The duration of hospital stay and the time between onset of illness and diagnosis were longer in the cardiac group. All patients received intravenous immunoglobulin and aspirin. Only one patient continued to have cardiac abnormalities after 6 months of follow-up. CONCLUSIONS:Kawasaki disease is not rare in southwest of Iran. The age, gender distribution and clinical findings are similar to that of other reports. Patients with cardiac abnormalities had delayed treatment and prolonged hospital stays.

Dataset Information

Design Characteristics Influence Performance of Clinical Prediction Rules in Validation: A Meta-Epidemiological Study.

Background

Methods

Results

Conclusion

Publications

Design Characteristics Influence Performance of Clinical Prediction Rules in Validation: A Meta-Epidemiological Study.

Similar Datasets

OmicsDI is part of the ELIXIR infrastructure

Tweets