Dataset Information

Measuring the statistical validity of summary meta-analysis and meta-regression results for use in clinical practice.

ABSTRACT: An important question for clinicians appraising a meta-analysis is: are the findings likely to be valid in their own practice-does the reported effect accurately represent the effect that would occur in their own clinical population? To this end we advance the concept of statistical validity-where the parameter being estimated equals the corresponding parameter for a new independent study. Using a simple ('leave-one-out') cross-validation technique, we demonstrate how we may test meta-analysis estimates for statistical validity using a new validation statistic, Vn, and derive its distribution. We compare this with the usual approach of investigating heterogeneity in meta-analyses and demonstrate the link between statistical validity and homogeneity. Using a simulation study, the properties of Vn and the Q statistic are compared for univariate random effects meta-analysis and a tailored meta-regression model, where information from the setting (included as model covariates) is used to calibrate the summary estimate to the setting of application. Their properties are found to be similar when there are 50 studies or more, but for fewer studies Vn has greater power but a higher type 1 error rate than Q. The power and type 1 error rate of Vn are also shown to depend on the within-study variance, between-study variance, study sample size, and the number of studies in the meta-analysis. Finally, we apply Vn to two published meta-analyses and conclude that it usefully augments standard methods when deciding upon the likely validity of summary meta-analysis estimates in clinical practice. © 2017 The Authors. Statistics in Medicine published by John Wiley & Sons Ltd.

SUBMITTER: Willis BH

PROVIDER: S-EPMC5575530 | biostudies-literature | 2017 Sep

REPOSITORIES: biostudies-literature

ACCESS DATA

Publications

Measuring the statistical validity of summary meta-analysis and meta-regression results for use in clinical practice.

Willis Brian H BH Riley Richard D RD

Statistics in medicine 20170615 21

An important question for clinicians appraising a meta-analysis is: are the findings likely to be valid in their own practice-does the reported effect accurately represent the effect that would occur in their own clinical population? To this end we advance the concept of statistical validity-where the parameter being estimated equals the corresponding parameter for a new independent study. Using a simple ('leave-one-out') cross-validation technique, we demonstrate how we may test meta-analysis e ...[more]

PMID: 28620945

Similar Datasets

Project description:BackgroundPractitioner-level implementation interventions such as audit and feedback, communities of practice, and local opinion leaders have shown potential to change nurses' behaviour in clinical practice and improve patients' health. However, their effectiveness remains unclear. Moreover, we have a paucity of data regarding the use of theory in implementation studies with nurses, the causal processes-i.e. mechanisms of action-targeted by interventions to change nurses' behaviour in clinical practice, and the constituent components-i.e. behaviour change techniques-included in interventions. Thus, our objectives are threefold: (1) to examine the effectiveness of practitioner-level implementation interventions in changing nurses' behaviour in clinical practice; (2) to identify, in included studies, the type and degree of theory use, the mechanisms of action targeted by interventions and the behaviour change techniques constituting interventions and (3) to examine whether intervention effectiveness is associated with the use of theory or with specific mechanisms of action and behaviour change techniques.MethodsWe will conduct a systematic review based on the Cochrane Effective Practice and Organization of Care (EPOC) Group guidelines. We will search six databases (CINAHL, EMBASE, ERIC, PsycINFO, PubMed and Web of Science) with no time limitation for experimental and quasi-experimental studies that evaluated practitioner-level implementation interventions aiming to change nurses' behaviour in clinical practice. We will also hand-search reference lists of included studies. We will perform screening, full-text review, risk of bias assessment, and data extraction independently with the Covidence systematic review software. We will assess the quality of evidence using the GRADEpro software. We will code included studies independently for theory use (Theory Coding Scheme), mechanisms of action (coding guidelines from Michie) and behaviour change techniques (Behaviour Change Technique Taxonomy v1) with QSR International's NVivo qualitative data analysis software. Meta-analyses will be performed using the Review Manager (RevMan) software. Meta-regression analyses will be performed with IBM SPSS Statistics software.DiscussionThis review will inform knowledge users and researchers interested in designing, developing and evaluating implementation interventions to support nurses' behaviour change in clinical practice. Results will provide key insights regarding which causal processes-i.e. mechanisms of action-should be targeted by these interventions, and which constituent components-i.e. behaviour change techniques-should be included in these interventions to increase their effectiveness.Systematic review registrationThe protocol has been registered at the International Prospective Register of Systematic Reviews (PROSPERO; registration number: CRD42019130446).

Project description:Catatonia is an independent syndrome that co-occurs with several mental and medical conditions. We performed a systematic literature review in PubMed/Scopus until February 2017 and meta-analyzed studies reporting catatonia prevalence. Across 74 studies (cross-sectional = 32, longitudinal = 26, retrospective = 16) providing data collected from 1935 to 2017 across all continents, mean catatonia prevalence was 9.0% (k = 80, n = 110764; 95% CI = 6.9-11.7, I2 = 98%, publication bias P < .01), decreasing to 7.8% (k = 19, n = 7612, 95% CI = 7-8.7, I2 = 38.9%) in a subgroup with low heterogeneity. Catatonia prevalence was 23.9% (k = 8, n = 1168, 95% CI = 10-46.9, I2 = 96%) in patients undergoing ECT/having elevated creatinine phosphokinase. Excluding ECT samples, the catatonia prevalence was 8.1% (k = 72, n = 109606, 95% CI = 6.1-10.5, I2 = 98%, publication bias P < .01), with sensitivity analyses demonstrating that country of study origin (P < .001), treatment setting (P = .003), main underlying condition (P < .001), and sample size (P < .001)moderated catatonia prevalence, being highest in Uganda (48.5%, k = 1) and lowest in Mexico (1.9%, 95% CI = 0.4-8.8, I2 = 67%, k = 2), highest in nonpsychiatric out- or inpatient services (15.8%, 95% CI = 8.1-28.4, I2 = 97%, k = 15)and lowest in psychiatric outpatients services (3.2%, 95% CI = 1.7-6.1, I2 = 50%, k = 3), highest in presence of medical or neurological illness with no comorbid psychiatric condition (20.6%, 95% CI = 11.5-34.2, I2 = 95%, k = 10)and lowest in mixed psychiatric samples (5.7%, 95% CI = 4.2-7.7, I2 =98%, k = 43), highest in studies with sample sizes <100 (20.7%, 95% CI = 12.8-31.6, I2 = 90%, k = 17) and lowest in studies with sample sizes >1000 (2.3%, 95% CI = 1.3-3.9, I2 = 99%, k = 16). Meta-regression showed that smaller sample size (P < .01) and less major depressive disorder (P = .02) moderated higher catatonia prevalence. Year of data collection did not significantly moderate the results. Results from this first meta-analysis of catatonia frequencies across time and disorders suggest that catatonia is an epidemiologically and clinically relevant condition that occurs throughout several mental and medical conditions, whose prevalence has not decreased over time and does not seem to depend on different rating scales/criteria. However, results were highly heterogeneous, calling for a cautious interpretation.

Project description:BackgroundComputerized clinical decision support systems (CDSSs) are a promising knowledge translation tool, but often fail to meaningfully influence the outcomes they target. Low CDSS provider uptake is a potential contributor to this problem but has not been systematically studied. The objective of this systematic review and meta-regression was to determine reported CDSS uptake and identify which CDSS features may influence uptake.MethodsMedline, Embase, CINAHL, and the Cochrane Database of Controlled Trials were searched from January 2000 to August 2020. Randomized, non-randomized, and quasi-experimental trials reporting CDSS uptake in any patient population or setting were included. The main outcome extracted was CDSS uptake, reported as a raw proportion, and representing the number of times the CDSS was used or accessed over the total number of times it could have been interacted with. We also extracted context, content, system, and implementation features that might influence uptake, for each CDSS. Overall weighted uptake was calculated using random-effects meta-analysis and determinants of uptake were investigated using multivariable meta-regression.ResultsAmong 7995 citations screened, 55 studies involving 373,608 patients and 3607 providers met full inclusion criteria. Meta-analysis revealed that overall CDSS uptake was 34.2% (95% CI 23.2 to 47.1%). Uptake was only reported in 12.4% of studies that otherwise met inclusion criteria. Multivariable meta-regression revealed the following factors significantly associated with uptake: (1) formally evaluating the availability and quality of the patient data needed to inform CDSS advice; and (2) identifying and addressing other barriers to the behaviour change targeted by the CDSS.Conclusions and relevanceSystem uptake was seldom reported in CDSS trials. When reported, uptake was low. This represents a major and potentially modifiable barrier to overall CDSS effectiveness. We found that features relating to CDSS context and implementation strategy best predicted uptake. Future studies should measure the impact of addressing these features as part of the CDSS implementation strategy. Uptake reporting must also become standard in future studies reporting CDSS intervention effects.RegistrationPre-registered on PROSPERO, CRD42018092337.

Dataset Information

Measuring the statistical validity of summary meta-analysis and meta-regression results for use in clinical practice.

Publications

Measuring the statistical validity of summary meta-analysis and meta-regression results for use in clinical practice.

Similar Datasets

OmicsDI is part of the ELIXIR infrastructure

Tweets