Dataset Information

Design Characteristics Influence Performance of Clinical Prediction Rules in Validation: A Meta-Epidemiological Study.

ABSTRACT:

Background

Many new clinical prediction rules are derived and validated. But the design and reporting quality of clinical prediction research has been less than optimal. We aimed to assess whether design characteristics of validation studies were associated with the overestimation of clinical prediction rules' performance. We also aimed to evaluate whether validation studies clearly reported important methodological characteristics.

Methods

Electronic databases were searched for systematic reviews of clinical prediction rule studies published between 2006 and 2010. Data were extracted from the eligible validation studies included in the systematic reviews. A meta-analytic meta-epidemiological approach was used to assess the influence of design characteristics on predictive performance. From each validation study, it was assessed whether 7 design and 7 reporting characteristics were properly described.

Results

A total of 287 validation studies of clinical prediction rule were collected from 15 systematic reviews (31 meta-analyses). Validation studies using case-control design produced a summary diagnostic odds ratio (DOR) 2.2 times (95% CI: 1.2-4.3) larger than validation studies using cohort design and unclear design. When differential verification was used, the summary DOR was overestimated by twofold (95% CI: 1.2 -3.1) compared to complete, partial and unclear verification. The summary RDOR of validation studies with inadequate sample size was 1.9 (95% CI: 1.2 -3.1) compared to studies with adequate sample size. Study site, reliability, and clinical prediction rule was adequately described in 10.1%, 9.4%, and 7.0% of validation studies respectively.

Conclusion

Validation studies with design shortcomings may overestimate the performance of clinical prediction rules. The quality of reporting among studies validating clinical prediction rules needs to be improved.

SUBMITTER: Ban JW

PROVIDER: S-EPMC4701404 | biostudies-literature | 2016

REPOSITORIES: biostudies-literature

ACCESS DATA

Publications

Design Characteristics Influence Performance of Clinical Prediction Rules in Validation: A Meta-Epidemiological Study.

Ban Jong-Wook JW Emparanza José Ignacio JI Urreta Iratxe I Burls Amanda A

PloS one 20160105 1

<h4>Background</h4>Many new clinical prediction rules are derived and validated. But the design and reporting quality of clinical prediction research has been less than optimal. We aimed to assess whether design characteristics of validation studies were associated with the overestimation of clinical prediction rules' performance. We also aimed to evaluate whether validation studies clearly reported important methodological characteristics.<h4>Methods</h4>Electronic databases were searched for s ...[more]

PMID: 26730980

Similar Datasets

Project description:OBJECTIVES:To empirically assess the relation between study characteristics and prognostic model performance in external validation studies of multivariable prognostic models. DESIGN:Meta-epidemiological study. DATA SOURCES AND STUDY SELECTION:On 16 October 2018, we searched electronic databases for systematic reviews of prognostic models. Reviews from non-overlapping clinical fields were selected if they reported common performance measures (either the concordance (c)-statistic or the ratio of observed over expected number of events (OE ratio)) from 10 or more validations of the same prognostic model. DATA EXTRACTION AND ANALYSES:Study design features, population characteristics, methods of predictor and outcome assessment, and the aforementioned performance measures were extracted from the included external validation studies. Random effects meta-regression was used to quantify the association between the study characteristics and model performance. RESULTS:We included 10 systematic reviews, describing a total of 224 external validations, of which 221 reported c-statistics and 124 OE ratios. Associations between study characteristics and model performance were heterogeneous across systematic reviews. C-statistics were most associated with variation in population characteristics, outcome definitions and measurement and predictor substitution. For example, validations with eligibility criteria comparable to the development study were associated with higher c-statistics compared with narrower criteria (difference in logit c-statistic 0.21(95% CI 0.07 to 0.35), similar to an increase from 0.70 to 0.74). Using a case-control design was associated with higher OE ratios, compared with using data from a cohort (difference in log OE ratio 0.97(95% CI 0.38 to 1.55), similar to an increase in OE ratio from 1.00 to 2.63). CONCLUSIONS:Variation in performance of prognostic models across studies is mainly associated with variation in case-mix, study designs, outcome definitions and measurement methods and predictor substitution. Researchers developing and validating prognostic models should realise the potential influence of these study characteristics on the predictive performance of prognostic models.

Project description:IntroductionExternal validation studies are essential to study the generalizability of prediction models. Recently a permutation test, focusing on discrimination as quantified by the c-statistic, was proposed to judge whether a prediction model is transportable to a new setting. We aimed to evaluate this test and compare it to previously proposed procedures to judge any changes in c-statistic from development to external validation setting.MethodsWe compared the use of the permutation test to the use of benchmark values of the c-statistic following from a previously proposed framework to judge transportability of a prediction model. In a simulation study we developed a prediction model with logistic regression on a development set and validated them in the validation set. We concentrated on two scenarios: 1) the case-mix was more heterogeneous and predictor effects were weaker in the validation set compared to the development set, and 2) the case-mix was less heterogeneous in the validation set and predictor effects were identical in the validation and development set. Furthermore we illustrated the methods in a case study using 15 datasets of patients suffering from traumatic brain injury.ResultsThe permutation test indicated that the validation and development set were homogenous in scenario 1 (in almost all simulated samples) and heterogeneous in scenario 2 (in 17%-39% of simulated samples). Previously proposed benchmark values of the c-statistic and the standard deviation of the linear predictors correctly pointed at the more heterogeneous case-mix in scenario 1 and the less heterogeneous case-mix in scenario 2.ConclusionThe recently proposed permutation test may provide misleading results when externally validating prediction models in the presence of case-mix differences between the development and validation population. To correctly interpret the c-statistic found at external validation it is crucial to disentangle case-mix differences from incorrect regression coefficients.

Project description:BackgroundClinical prediction rules (CPRs) should be externally validated by independent researchers. Although there are many cardiovascular CPRs, most have not been externally validated. It is not known why some CPRs are externally validated by independent researchers and others are not.MethodsWe analyzed cardiovascular risk CPRs included in a systematic review. Independent external validations were identified by forward citation searches of derivation studies. Time between the publication of a cardiovascular CPR and the first independent external validation was calculated. We assessed Kaplan-Meier estimates of the probability to have an independent external validation. Using univariable Cox regression, we explored whether characteristics of derivation (design, location, sample size, number of predictors, presentation format, validation in derivation), reporting (participants, predictors, outcomes, performance measure, information for risk calculation), and publication (journal impact factor) are associated with time to the first independent external validation.ResultsOf 125 cardiovascular risk CPRs, 29 had an independent external validation. The median follow-up was 118 months (95% CI, 99-130). The 25th percentile of event time was 122 months (95% CI, 91-299). Cardiovascular risk CPRs from the USA were 4.15 times (95% CI, 1.89-9.13) more likely to have an independent external validation. Increasing the sample size of derivation by ten times was associated with a 2.32-fold (95% CI, 1.37-3.91) increase in the probability of having an independent external validation. CPRs presented with an internal validation tend to get an independent external validation sooner (HR = 1.73, 95% CI, 0.77-3.93). CPRs reporting all the information necessary for calculating individual risk were 2.65 (95% CI, 1.01-6.96) times more likely to have an independent external validation. Publishing a cardiovascular risk CPR in a journal that has one unit higher impact factor was associated with a 6% (95% CI, 3-9) higher likelihood of an independent external validation.ConclusionsThe probability for cardiovascular risk CPRs to get an independent external validation was low even many years after their derivations. Authors of new cardiovascular risk CPRs should consider using adequate sample size, conducting an internal validation, and reporting all the information needed for risk calculation as these features were associated with an independent external validation.

Project description:ObjectivesInfectious mononucleosis (IM) is a clinical syndrome that is characterised by lymphadenopathy, fever and sore throat. Although generally not considered a serious illness, IM can lead to significant loss of time from school or work due to profound fatigue, or the development of chronic illness. This study aimed to derive and externally validate clinical prediction rules (CPRs) for IM caused by Epstein-Barr virus (EBV).DesignProspective cohort study.Setting and participants328 participants were recruited prospectively for the derivation cohort, from seven university-affiliated student health centres in Ireland. Participants were young adults (17-39 years old, mean age 20.6 years) with sore throat and one other additional symptom suggestive of IM. The validation cohort was a retrospective cohort of 1498 participants from a student health centre at the University of Georgia, USA.Main outcome measuresRegression analyses were used to develop four CPR models, internally validated in the derivation cohort. External validation was carried out in the geographically separate validation cohort.ResultsIn the derivation cohort, there were 328 participants, of whom 42 (12.8%) had a positive EBV serology test result. Of 1498 participants in the validation cohort, 243 (16.2%) had positive heterophile antibody tests for IM. Four alternative CPR models were developed and compared. There was moderate discrimination and good calibration for all models. The sparsest CPR included presence of enlarged/tender posterior cervical lymph nodes and presence of exudate on the pharynx. This model had moderate discrimination (area under the receiver operating characteristic curve (AUC): 0.70; 95% CI: 0.62-0.79) and good calibration. On external validation, this model demonstrated reasonable discrimination (AUC: 0.69; 95% CI: 0.67-0.72) and good calibration.ConclusionsThe alternative CPRs proposed can provide quantitative probability estimates of IM. Used in conjunction with serological testing for atypical lymphocytosis and immunoglobulin testing for viral capsid antigen, CPRs can enhance diagnostic decision-making for IM in community settings.

Project description:BACKGROUND: During an influenza epidemic prompt diagnosis of influenza is important. This diagnosis however is still essentially based on the interpretation of symptoms and signs by general practitioners. No single symptom is specific enough to be useful in differentiating influenza from other respiratory infections. Our objective is to formulate prediction rules for the diagnosis of influenza with the best diagnostic performance, combining symptoms, signs and context among patients with influenza-like illness. METHODS: During five consecutive winter periods (2002-2007) 138 sentinel general practitioners sampled (naso- and oropharyngeal swabs) 4597 patients with an influenza-like illness (ILI) and registered their symptoms and signs, general characteristics and contextual information. The samples were analysed by a DirectigenFlu-A&B and RT-PCR tests. 4584 records were useful for further analysis.Starting from the most relevant variables in a Generalized Estimating Equations (GEE) model, we calculated the area under the Receiver Operating Characteristic curve (ROC AUC), sensitivity, specificity and likelihood ratios for positive (LR+) and negative test results (LR-) of single and combined signs, symptoms and context taking into account pre-test and post-test odds. RESULTS: In total 52.6% (2409/4584) of the samples were positive for influenza virus: 64% (2066/3212) during and 25% (343/1372) pre/post an influenza epidemic. During and pre/post an influenza epidemic the LR+ of 'previous flu-like contacts', 'coughing', 'expectoration on the first day of illness' and 'body temperature above 37.8°C' is 3.35 (95%CI 2.67-4.03) and 1.34 (95%CI 0.97-1.72), respectively. During and pre/post an influenza epidemic the LR- of 'coughing' and 'a body temperature above 37.8°C' is 0.34 (95%CI 0.27-0.41) and 0.07 (95%CI 0.05-0.08), respectively. CONCLUSIONS: Ruling out influenza using clinical and contextual information is easier than ruling it in. Outside an influenza epidemic the absence of cough and fever (> 37,8°C) makes influenza 14 times less likely in ILI patients. During an epidemic the presence of 'previous flu-like contacts', cough, 'expectoration on the first day of illness' and fever (>37,8°C) increases the likelihood for influenza threefold. The additional diagnostic value of rapid point of care tests especially for confirming influenza still has to be established.

Dataset Information

Design Characteristics Influence Performance of Clinical Prediction Rules in Validation: A Meta-Epidemiological Study.

Background

Methods

Results

Conclusion

Publications

Design Characteristics Influence Performance of Clinical Prediction Rules in Validation: A Meta-Epidemiological Study.

Similar Datasets

OmicsDI is part of the ELIXIR infrastructure

Tweets