Assessment of generalisability in trials of health interventions: suggested framework and systematic review.
Ontology highlight
ABSTRACT:
SUBMITTER: Bonell C
PROVIDER: S-EPMC1539056 | biostudies-literature | 2006 Aug
REPOSITORIES: biostudies-literature
Publications
Assessment of generalisability in trials of health interventions: suggested framework and systematic review.
Bonell C C Oakley A A Hargreaves J J Strange V V Rees R R
BMJ (Clinical research ed.) 20060801 7563
Similar Datasets
Project description:Sub-Saharan Africa (SSA) is facing a double burden of disease with a rising prevalence of non-communicable diseases (NCDs) while the burden of communicable diseases (CDs) remains high. Despite these challenges, there remains a significant need to understand how or under what conditions health interventions implemented in sub-Saharan Africa are sustained. The purpose of this study was to conduct a systematic review of empirical literature to explore how health interventions implemented in SSA are sustained.We searched MEDLINE, Biological Abstracts, CINAHL, Embase, PsycInfo, SCIELO, Web of Science, and Google Scholar for available research investigating the sustainability of health interventions implemented in sub-Saharan Africa. We also used narrative synthesis to examine factors whether positive or negative that may influence the sustainability of health interventions in the region.The search identified 1819 citations, and following removal of duplicates and our inclusion/exclusion criteria, only 41 papers were eligible for inclusion in the review. Twenty-six countries were represented in this review, with Kenya and Nigeria having the most representation of available studies examining sustainability. Study dates ranged from 1996 to 2015. Of note, majority of these studies (30 %) were published in 2014. The most common framework utilized was the sustainability framework, which was discussed in four of the studies. Nineteen out of 41 studies (46 %) reported sustainability outcomes focused on communicable diseases, with HIV and AIDS represented in majority of the studies, followed by malaria. Only 21 out of 41 studies had clear definitions of sustainability. Community ownership and mobilization were recognized by many of the reviewed studies as crucial facilitators for intervention sustainability, both early on and after intervention implementation, while social and ecological conditions as well as societal upheavals were barriers that influenced the sustainment of interventions in sub-Saharan Africa.The sustainability of health interventions implemented in sub-Saharan Africa is inevitable given the double burden of diseases, health care worker shortage, weak health systems, and limited resources. We propose a conceptual framework that draws attention to sustainability as a core component of the overall life cycle of interventions implemented in the region.
| S-EPMC4804528 | biostudies-other
Project description:BackgroundThe application of Health Information Technologies (HITs) can be an effective way to advance medical research and health services provision. The two-fold objective of this work is to: (i) identify and review state-of-the-art HITs that facilitate the aims of evidence-based medicine and (ii) propose a methodology for HIT assessment.MethodsThe systematic review was conducted according to the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) guidelines. Furthermore, we consolidated existing knowledge in the field and proposed a Synthesis Framework for the Assessment of Health Information Technology (SF/HIT) in order to evaluate the joint use of Randomized Controlled Trials (RCTs) along with HITs in the field of evidence-based medicine.Results55 articles met the inclusion criteria and refer to 51 (RCTs) published between 2008 and 2016. Significant improvements in healthcare through the use of HITs were observed in the findings of 31 out of 51 trials-60.8%. We also confirmed that RCTs are valuable tools for assessing the effectiveness, acceptability, safety, privacy, appropriateness, satisfaction, performance, usefulness and adherence.ConclusionsTo improve health service delivery, RCTs apply and exhibit formalization by providing measurable outputs. Towards this direction, we propose the SF/HIT as a framework which may help researchers to carry out appropriate evaluations and extend their studies.
| S-EPMC6165327 | biostudies-literature
Project description:BACKGROUND: The purpose of the study was to evaluate systematic reviews of research into two public health priorities, tobacco consumption and HIV infection, in terms of the reporting of data related to the applicability of trial results (i.e., whether the results of a trial can be reasonably applied or generalized to a definable group of patients in a particular setting in routine practice, also called external validity or generalisability). METHODS: All systematic reviews of interventions aimed at reducing or stopping tobacco use and treating or preventing HIV infection published in the Cochrane database of systematic reviews and in journals indexed in MEDLINE between January 1997 and December 2007 were selected. We used a standardized data abstraction form to extract data related to applicability in terms of the context of the trial, (country, centres, settings), participants (recruitment, inclusion and exclusion criteria, baseline characteristics of participants such as age, sex, ethnicity, coexisting diseases or co-morbidities, and socioeconomic status), treatment (duration, intensity/dose of treatment, timing and delivery format), and the outcomes assessment from selected reviews. RESULTS: A total of 98 systematic reviews were selected (57 Cochrane reviews and 41 non-Cochrane reviews); 49 evaluated interventions aimed at reducing or stopping tobacco use and 49 treating or preventing HIV infection. The setting of the individual studies was reported in 45 (46%) of the systematic reviews, the number of centres in 21 (21%), and the country where the trial took place in 62 (63%). Inclusion and exclusion criteria of the included studies were reported in 16 (16%) and 13 (13%) of the reviews, respectively. Baseline characteristics of participants in the included studies were described in 59 (60%) of the reviews. These characteristics concerned age in about half of the reviews, sex in 46 (47%), and ethnicity in 9 (9%).Applicability of results was discussed in 13 (13%) of the systematic reviews. The reporting was better in systematic reviews by the Cochrane Collaboration than by non-Cochrane groups. CONCLUSIONS: Our study highlighted the lack of consideration of applicability of results in systematic reviews of research into 2 public health priorities: tobacco consumption and HIV infection.
| S-EPMC2838881 | biostudies-other
Project description:Neuropsychiatric disorders pose a high societal cost, but their treatment is hindered by lack of objective outcomes and fidelity metrics. AI technologies and specifically Natural Language Processing (NLP) have emerged as tools to study mental health interventions (MHI) at the level of their constituent conversations. However, NLP's potential to address clinical and research challenges remains unclear. We therefore conducted a pre-registered systematic review of NLP-MHI studies using PRISMA guidelines (osf.io/s52jh) to evaluate their models, clinical applications, and to identify biases and gaps. Candidate studies (n = 19,756), including peer-reviewed AI conference manuscripts, were collected up to January 2023 through PubMed, PsycINFO, Scopus, Google Scholar, and ArXiv. A total of 102 articles were included to investigate their computational characteristics (NLP algorithms, audio features, machine learning pipelines, outcome metrics), clinical characteristics (clinical ground truths, study samples, clinical focus), and limitations. Results indicate a rapid growth of NLP MHI studies since 2019, characterized by increased sample sizes and use of large language models. Digital health platforms were the largest providers of MHI data. Ground truth for supervised learning models was based on clinician ratings (n = 31), patient self-report (n = 29) and annotations by raters (n = 26). Text-based features contributed more to model accuracy than audio markers. Patients' clinical presentation (n = 34), response to intervention (n = 11), intervention monitoring (n = 20), providers' characteristics (n = 12), relational dynamics (n = 14), and data preparation (n = 4) were commonly investigated clinical categories. Limitations of reviewed studies included lack of linguistic diversity, limited reproducibility, and population bias. A research framework is developed and validated (NLPxMHI) to assist computational and clinical researchers in addressing the remaining gaps in applying NLP to MHI, with the goal of improving clinical utility, data access, and fairness.
| S-EPMC10556019 | biostudies-literature
Project description: Not available
| S-EPMC11344322 | biostudies-literature
Project description:ImportanceDespite the potential of machine learning to improve multiple aspects of patient care, barriers to clinical adoption remain. Randomized clinical trials (RCTs) are often a prerequisite to large-scale clinical adoption of an intervention, and important questions remain regarding how machine learning interventions are being incorporated into clinical trials in health care.ObjectiveTo systematically examine the design, reporting standards, risk of bias, and inclusivity of RCTs for medical machine learning interventions.Evidence reviewIn this systematic review, the Cochrane Library, Google Scholar, Ovid Embase, Ovid MEDLINE, PubMed, Scopus, and Web of Science Core Collection online databases were searched and citation chasing was done to find relevant articles published from the inception of each database to October 15, 2021. Search terms for machine learning, clinical decision-making, and RCTs were used. Exclusion criteria included implementation of a non-RCT design, absence of original data, and evaluation of nonclinical interventions. Data were extracted from published articles. Trial characteristics, including primary intervention, demographics, adherence to the CONSORT-AI reporting guideline, and Cochrane risk of bias were analyzed.FindingsLiterature search yielded 19 737 articles, of which 41 RCTs involved a median of 294 participants (range, 17-2488 participants). A total of 16 RCTS (39%) were published in 2021, 21 (51%) were conducted at single sites, and 15 (37%) involved endoscopy. No trials adhered to all CONSORT-AI standards. Common reasons for nonadherence were not assessing poor-quality or unavailable input data (38 trials [93%]), not analyzing performance errors (38 [93%]), and not including a statement regarding code or algorithm availability (37 [90%]). Overall risk of bias was high in 7 trials (17%). Of 11 trials (27%) that reported race and ethnicity data, the median proportion of participants from underrepresented minority groups was 21% (range, 0%-51%).Conclusions and relevanceThis systematic review found that despite the large number of medical machine learning-based algorithms in development, few RCTs for these technologies have been conducted. Among published RCTs, there was high variability in adherence to reporting standards and risk of bias and a lack of participants from underrepresented minority groups. These findings merit attention and should be considered in future RCT design and reporting.
| S-EPMC9523495 | biostudies-literature
Project description:ObjectivesConversational agents (CAs) with emerging artificial intelligence present new opportunities to assist in health interventions but are difficult to evaluate, deterring their applications in the real world. We aimed to synthesize existing evidence and knowledge and outline an evaluation framework for CA interventions.Materials and methodsWe conducted a systematic scoping review to investigate designs and outcome measures used in the studies that evaluated CAs for health interventions. We then nested the results into an overarching digital health framework proposed by the World Health Organization (WHO).ResultsThe review included 81 studies evaluating CAs in experimental (n = 59), observational (n = 15) trials, and other research designs (n = 7). Most studies (n = 72, 89%) were published in the past 5 years. The proposed CA-evaluation framework includes 4 evaluation stages: (1) feasibility/usability, (2) efficacy, (3) effectiveness, and (4) implementation, aligning with WHO's stepwise evaluation strategy. Across these stages, this article presents the essential evidence of different study designs (n = 8), sample sizes, and main evaluation categories (n = 7) with subcategories (n = 40). The main evaluation categories included (1) functionality, (2) safety and information quality, (3) user experience, (4) clinical and health outcomes, (5) costs and cost benefits, (6) usage, adherence, and uptake, and (7) user characteristics for implementation research. Furthermore, the framework highlighted the essential evaluation areas (potential primary outcomes) and gaps across the evaluation stages.Discussion and conclusionThis review presents a new framework with practical design details to support the evaluation of CA interventions in healthcare research.Protocol registrationThe Open Science Framework (https://osf.io/9hq2v) on March 22, 2021.
| S-EPMC10873847 | biostudies-literature
Project description:Disturbed interoception (i.e., the sensing, awareness, and regulation of internal body signals) has been found across several mental disorders, leading to the development of interoception-based interventions (IBIs). Searching PubMed and PsycINFO, we conducted the first systematic review of randomized-controlled trials (RCTs) investigating the efficacy of behavioral IBIs at improving interoception and target symptoms of mental disorders in comparison to a non-interoception-based control condition [CRD42021297993]. Thirty-one RCTs fulfilled inclusion criteria. Across all studies, a pattern emerged with 20 (64.5%) RCTs demonstrating IBIs to be more efficacious at improving interoception compared to control conditions. The most promising results were found for post-traumatic stress disorder, irritable bowel syndrome, fibromyalgia and substance use disorders. Regarding symptom improvement, the evidence was inconclusive. The IBIs were heterogenous in their approach to improving interoception. The quality of RCTs was moderate to good. In conclusion, IBIs are potentially efficacious at improving interoception for some mental disorders. In terms of symptom reduction, the evidence is less promising. Future research on the efficacy of IBIs is needed.
| S-EPMC7615164 | biostudies-literature
Project description:BackgroundIntimate partner violence (IPV) is a major public health concern. eHealth interventions may reduce exposure to violence and health-related consequences as the technology provides a safe and flexible space for the target population. However, the evidence is unclear.ObjectiveThe goal of the review is to examine the effect of eHealth interventions compared with standard care on reducing IPV, depression, and posttraumatic stress disorder (PTSD) among women exposed to IPV.MethodsWe searched EMBASE, MEDLINE, Cochrane Central Register of Controlled Trials, PsycInfo, Scopus, Global Health Library, ClinicalTrials.gov, and International Clinical Trials Registry Platform for published and unpublished trials from inception until April 2019. Trials with an eHealth intervention targeting women exposed to violence were included. We assessed risk of bias using the Cochrane Risk of Bias Tool. Trials that reported effect estimates on overall IPV; physical, sexual, and psychological violence; depression; or posttraumatic stress disorder were included in meta-analyses.ResultsA total of 14 trials were included in the review; 8 published trials, 3 unpublished trials and 3 ongoing trials. Of the 8 published trials, 2 were judged as overall low risk of bias trials. The trials reported 23 types of outcomes, and 7 of the trials had outcomes that were eligible for meta-analyses. Our pooled analyses found no effect of eHealth interventions on any of our prespecified outcomes: overall IPV (SMD -0.01; 95% CI -0.11 to 0.08; I2=0%; 5 trials, 1668 women); physical violence (SMD 0.01; 95% CI -0.22 to 0.24; I2=58%; 4 trials, 1128 women); psychological violence (SMD 0.07; 95% CI -0.12 to 0.25; I2=40%; 4 trials, 1129 women); sexual violence (MD 0.36; 95% CI -0.18 to 0.91; I2=0%; 2 trials, 1029 women); depression (SMD -0.13; 95% CI -0.37 to 0.11; I2=78%; 5 trials, 1600 women); and PTSD (MD -0.11; 95% CI -1.04 to 0.82; I2=0%; 5 trials, 1267 women).ConclusionsThere is no evidence from randomized trials of a beneficial effect of eHealth interventions on IPV. More high-quality trials are needed, and we recommend harmonizing outcome reporting in IPV trials by establishing core outcome sets.Trial registrationPROSPERO International Prospective Register of Systematic Reviews CRD42019130124; https://www.crd.york.ac.uk/prospero/display_record.php?RecordID=130124.
| S-EPMC7762681 | biostudies-literature
Project description:IntroductionPersonalization is a much-discussed approach to improve adherence and outcomes for Digital Mental Health interventions (DMHIs). Yet, major questions remain open, such as (1) what personalization is, (2) how prevalent it is in practice, and (3) what benefits it truly has.MethodsWe address this gap by performing a systematic literature review identifying all empirical studies on DMHIs targeting depressive symptoms in adults from 2015 to September 2022. The search in Pubmed, SCOPUS and Psycinfo led to the inclusion of 138 articles, describing 94 distinct DMHIs provided to an overall sample of approximately 24,300 individuals.ResultsOur investigation results in the conceptualization of personalization as purposefully designed variation between individuals in an intervention's therapeutic elements or its structure. We propose to further differentiate personalization by what is personalized (i.e., intervention content, content order, level of guidance or communication) and the underlying mechanism [i.e., user choice, provider choice, decision rules, and machine-learning (ML) based approaches]. Applying this concept, we identified personalization in 66% of the interventions for depressive symptoms, with personalized intervention content (32% of interventions) and communication with the user (30%) being particularly popular. Personalization via decision rules (48%) and user choice (36%) were the most used mechanisms, while the utilization of ML was rare (3%). Two-thirds of personalized interventions only tailored one dimension of the intervention.DiscussionWe conclude that future interventions could provide even more personalized experiences and especially benefit from using ML models. Finally, empirical evidence for personalization was scarce and inconclusive, making further evidence for the benefits of personalization highly needed.Systematic review registrationIdentifier: CRD42022357408.
| S-EPMC10239832 | biostudies-literature