Dataset Information

Internal and external validity of cluster randomised trials: systematic review of recent trials.

ABSTRACT:

Objectives

To assess aspects of the internal validity of recently published cluster randomised trials and explore the reporting of information useful in assessing the external validity of these trials.

Design

Review of 34 cluster randomised trials in primary care published in 2004 and 2005 in seven journals (British Medical Journal, British Journal of General Practice, Family Practice, Preventive Medicine, Annals of Internal Medicine, Journal of General Internal Medicine, Pediatrics).

Data sources

National Library of Medicine (Medline) via PubMed.

Data extraction

To assess aspects of internal validity we extracted data on appropriateness of sample size calculations and analyses, methods of identifying and recruiting individual participants, and blinding. To explore reporting of information useful in assessing external validity we extracted data on cluster eligibility, cluster inclusion and retention, cluster generalisability, and the feasibility and acceptability of the intervention to health providers in clusters.

Results

21 (62%) trials accounted for clustering in sample size calculations and 30 (88%) in the analysis; about a quarter were potentially biased because of procedures surrounding recruitment and identification of patients; individual participants were blind to allocation status in 19 (56%) and outcome assessors were blind in 15 (44%). In almost half the reports, information relating to generalisability of clusters was poorly reported, and in two fifths there was no information about the feasibility and acceptability of the intervention.

Conclusions

Cluster randomised trials are essential for evaluating certain types of interventions. Issues affecting their internal validity, such as appropriate sample size calculations and analysis, have been widely disseminated and are now better addressed by researchers. Blinding of those identifying and recruiting patients to allocation status is recommended but is not always carried out. There may be fewer barriers to internal validity in trials in which individual participants are not recruited. External validity seems poorly addressed in many trials, yet is arguably as important as internal validity in judging quality as a basis for healthcare intervention.

SUBMITTER: Eldridge S

PROVIDER: S-EPMC2323095 | biostudies-literature |

REPOSITORIES: biostudies-literature

ACCESS DATA

Similar Datasets

Project description:BackgroundTreatment non-adherence in randomised trials refers to situations where some participants do not receive their allocated treatment as intended. For cluster randomised trials, where the unit of randomisation is a group of participants, non-adherence may occur at the cluster or individual level. When non-adherence occurs, randomisation no longer guarantees that the relationship between treatment receipt and outcome is unconfounded, and the power to detect the treatment effects in intention-to-treat analysis may be reduced. Thus, recording adherence and estimating the causal treatment effect adequately are of interest for clinical trials.ObjectivesTo assess the extent of reporting of non-adherence issues in published cluster trials and to establish which methods are currently being used for addressing non-adherence, if any, and whether clustering is accounted for in these.MethodsWe systematically reviewed 132 cluster trials published in English in 2011 previously identified through a search in PubMed.ResultsOne-hundred and twenty three cluster trials were included in this systematic review. Non-adherence was reported in 56 cluster trials. Among these, 19 reported a treatment efficacy estimate: per protocol in 15 and as treated in 4. No study discussed the assumptions made by these methods, their plausibility or the sensitivity of the results to deviations from these assumptions.LimitationsThe year of publication of the cluster trials included in this review (2011) could be considered a limitation of this study; however, no new guidelines regarding the reporting and the handling of non-adherence for cluster trials have been published since. In addition, a single reviewer undertook the data extraction. To mitigate this, a second reviewer conducted a validation of the extraction process on 15 randomly selected reports. Agreement was satisfactory (93%).ConclusionDespite the recommendations of the Consolidated Standards of Reporting Trials statement extension to cluster randomised trials, treatment adherence is under-reported. Among the trials providing adherence information, there was substantial variation in how adherence was defined, handled and reported. Researchers should discuss the assumptions required for the results to be interpreted causally and whether these are scientifically plausible in their studies. Sensitivity analyses to study the robustness of the results to departures from these assumptions should be performed.

Project description:The purpose of this review was to determine the degree to which physical activity interventions for Latin American populations reported on internal and external validity factors using the RE-AIM framework (reach & representativeness, effectiveness, adoption, implementation, maintenance). We systematically identified English (PubMed; EbscoHost) and Spanish (SCIELO; Biblioteca Virtual en Salud) language studies published between 2001 and 2012 that tested physical activity, exercise, or fitness promotion interventions in Latin American populations. Cross-sectional/descriptive studies, conducted in Brazil or Spain, published in Portuguese, not including a physical activity/fitness/exercise outcome, and with one time point assessment were excluded. We reviewed 192 abstracts and identified 46 studies that met the eligibility criteria (34 in English, 12 in Spanish). A validated 21-item RE-AIM abstraction tool was used to determine the quality of reporting across studies (0-7 = low, 8-14 = moderate, and 15-21 = high). The number of indicators reported ranged from 3-14 (mean = 8.1 ± 2.6), with the majority of studies falling in the moderate quality reporting category. English and Spanish language articles did not differ on the number of indicators reported (8.1 vs. 8.3, respectively). However, Spanish articles reported more across reach indicators (62% vs. 43% of indicators), while English articles reported more across effectiveness indicators (69% vs 62%). Across RE-AIM dimensions, indicators for reach (48%), efficacy/effectiveness (67%), and implementation (41%) were reported more often than indicators of adoption (25%) and maintenance (10%). Few studies reported on the representativeness of participants, staff that delivered interventions, or the settings where interventions were adopted. Only 13% of the studies reported on quality of life and/or potential negative outcomes, 20% reported on intervention fidelity, and 11% on cost of implementation. Outcomes measured after six months of intervention, information on continued delivery and institutionalization of interventions, were also seldom reported. Regardless of language of publication, physical activity intervention research for Latin Americans should increase attention to and measurement of external validity and cost factors that are critical in the decision making process in practice settings and can increase the likelihood of translation into community or clinical practice.

Project description:BACKGROUND: Controlled clinical trials of health care interventions are either explanatory or pragmatic. Explanatory trials test whether an intervention is efficacious; that is, whether it can have a beneficial effect in an ideal situation. Pragmatic trials measure effectiveness; they measure the degree of beneficial effect in real clinical practice. In pragmatic trials, a balance between external validity (generalizability of the results) and internal validity (reliability or accuracy of the results) needs to be achieved. The explanatory trial seeks to maximize the internal validity by assuring rigorous control of all variables other than the intervention. The pragmatic trial seeks to maximize external validity to ensure that the results can be generalized. However the danger of pragmatic trials is that internal validity may be overly compromised in the effort to ensure generalizability. We are conducting two pragmatic randomized controlled trials on interventions in the management of hypertension in primary care. We describe the design of the trials and the steps taken to deal with the competing demands of external and internal validity. DISCUSSION: External validity is maximized by having few exclusion criteria and by allowing flexibility in the interpretation of the intervention and in management decisions. Internal validity is maximized by decreasing contamination bias through cluster randomization, and decreasing observer and assessment bias, in these non-blinded trials, through baseline data collection prior to randomization, automating the outcomes assessment with 24 hour ambulatory blood pressure monitors, and blinding the data analysis. SUMMARY: Clinical trials conducted in community practices present investigators with difficult methodological choices related to maintaining a balance between internal validity (reliability of the results) and external validity (generalizability). The attempt to achieve methodological purity can result in clinically meaningless results, while attempting to achieve full generalizability can result in invalid and unreliable results. Achieving a creative tension between the two is crucial.

Project description:OBJECTIVE:To conduct the first meta-analysis evaluating the internal and external validity of the sluggish cognitive tempo (SCT) construct as related to or distinct from attention-deficit/hyperactivity disorder (ADHD) and as associated with functional impairment and neuropsychological functioning. METHOD:Electronic databases were searched through September 2015 for studies examining the factor structure and/or correlates of SCT in children or adults. The search procedures identified 73 papers. The core SCT behaviors included across studies, as well as factor loadings and reliability estimates, were reviewed to evaluate internal validity. Pooled correlation effect sizes using random effects models were used to evaluate SCT in relation to external validity domains (i.e., demographics, other psychopathologies, functional impairment, and neuropsychological functioning). RESULTS:Strong support was found for the internal validity of the SCT construct. Specifically, across factor analytic studies including more than 19,000 individuals, 13 SCT items loaded consistently on an SCT factor as opposed to an ADHD factor. Findings also support the reliability (i.e., internal consistency, test-retest reliability, interrater reliability) of SCT. In terms of external validity, there is some indication that SCT may increase with age (r = 0.11) and be associated with lower socioeconomic status (r = 0.10). Modest (potentially negligible) support was found for SCT symptoms being higher in males than females in children (r = 0.05) but not in adults. SCT is more strongly associated with ADHD inattention (r = 0.63 in children, r = 0.72 in adults) than with ADHD hyperactivity-impulsivity (r = 0.32 in children, r = 0.46 in adults), and it likewise appears that SCT is more strongly associated with internalizing symptoms than with externalizing symptoms. SCT is associated with significant global, social, and academic impairment (r = 0.38-0.44). Effects for neuropsychological functioning are mixed, although there is initial support for SCT being associated with processing speed, sustained attention, and metacognitive deficits. CONCLUSION:This meta-analytic review provides strong support for the internal validity of SCT and preliminary support for the external validity of SCT. In terms of diagnostic validity, there is currently not enough evidence to describe SCT in diagnostic terms. Key directions for future research are discussed, including evaluating the conceptualization of SCT as a transdiagnostic construct and the need for longitudinal research.