Dataset Information

A scoping review of the evaluation and effectiveness of technical assistance.

ABSTRACT:

Background

Although the benefits of evidence-based practices (EBPs) for advancing community outcomes are well-recognized, challenges with the uptake of EBPs are considerable. Technical assistance (TA) is a core capacity building strategy that has been widely used to support EBP implementation and other community development and improvement efforts. Yet despite growing reliance on TA, no reviews have systematically examined the evaluation of TA across varying implementation contexts and capacity building aims. This study draws on two decades of peer-reviewed publications to summarize the evidence on the evaluation and effectiveness of TA.

Methods

Guided by Arksey and O'Malley's six-stage methodological framework, we used a scoping review methodology to map research on TA evaluation. We included peer-reviewed articles published in English between 2000 and 2020. Our search involved five databases: Business Source Complete, Cumulative Index to Nursing and Allied Health Literature (CINAHL), Education Resources Information Center (ERIC), PsycInfo, and PubMed.

Results

A total of 125 evaluation research studies met the study criteria. Findings indicate that publications have increased over the last two decades, signaling a growth in the recognition and reporting of TA. Technical assistance is being implemented across diverse settings, often serving socially vulnerable and under-resourced populations. Most evaluation research studies involved summative evaluations, with TA outcomes mostly reported at the organizational level. Only 5% of the studies examined sustainability of TA outcomes. This review also demonstrates that there is a lack of consistent standards regarding the definition of TA and the level of reporting across relevant TA evaluation categories (e.g., cadence of contact, and directionality).

Conclusions

Advances in the science and practice of TA hinge on understanding what aspects of TA are effective and when, how, and for whom these aspects of TA are effective. Addressing these core questions requires (i) a standard definition for TA; (ii) more robust and rigorous evaluation research designs that involve comparison groups and assessment of direct, indirect, and longitudinal outcomes; (iii) increased use of reliable and objective TA measures; and (iv) development of reporting standards. We view this scoping review as a foundation for improving the state of the science and practice of evaluating TA.

SUBMITTER: Scott VC

PROVIDER: S-EPMC9238031 | biostudies-literature |

REPOSITORIES: biostudies-literature

ACCESS DATA

Similar Datasets

Project description:Significance:Prior research suggests that Community Engagement and Planning (CEP) for coalition support compared with Resources for Services (RS) for program technical assistance to implement depression quality improvement programs improves 6- and 12-month client mental-health related quality of life (MHRQL); however, effects for clients with multiple chronic medical conditions (MCC) are unknown. Objective:To explore effectiveness of CEP vs RS in MCC and non-MCC subgroups. Design:Secondary analyses of a cluster-randomized trial. Setting:93 health care and community-based programs in two neighborhoods. Participants:Of 4,440 clients screened, 1,322 depressed (Patient Health Questionnaire, PHQ8) provided contact information, 1,246 enrolled and 1,018 (548 with ?3 MCC) completed baseline, 6- or 12-month surveys. Intervention:CEP or RS for implementing depression quality improvement programs. Outcomes and Analyses:Primary: depression (PHQ9 <10), poor MHRQL (Short Form Health Survey, SF-12<40); Secondary: mental wellness, good physical health, behavioral health hospitalization, chronic homelessness risk, work/workloss days, services use at 6 and 12 months. End-point regressions were used to estimate intervention effects on outcomes for subgroups with ?3 MCC, non-MCC, and intervention-by-MCC interactions (exploratory). Results:Among MCC clients at 6 months, CEP vs RS lowered likelihoods of depression and poor MHRQL; increased likelihood of mental wellness; reduced work-loss days among employed and likelihoods of ?4 behavioral-health hospitalization nights and chronic homelessness risk, while increasing faith-based and park community center depression services; and at 12 months, likelihood of good physical health and park community center depression services use (each P<.05). There were no significant interactions or primary outcome effects for non-MCC. Conclusions:CEP was more effective than RS in improving 6-month primary outcomes among depressed MCC clients, without significant interactions.

Project description:BACKGROUND:Dialog agents (chatbots) have a long history of application in health care, where they have been used for tasks such as supporting patient self-management and providing counseling. Their use is expected to grow with increasing demands on health systems and improving artificial intelligence (AI) capability. Approaches to the evaluation of health care chatbots, however, appear to be diverse and haphazard, resulting in a potential barrier to the advancement of the field. OBJECTIVE:This study aims to identify the technical (nonclinical) metrics used by previous studies to evaluate health care chatbots. METHODS:Studies were identified by searching 7 bibliographic databases (eg, MEDLINE and PsycINFO) in addition to conducting backward and forward reference list checking of the included studies and relevant reviews. The studies were independently selected by two reviewers who then extracted data from the included studies. Extracted data were synthesized narratively by grouping the identified metrics into categories based on the aspect of chatbots that the metrics evaluated. RESULTS:Of the 1498 citations retrieved, 65 studies were included in this review. Chatbots were evaluated using 27 technical metrics, which were related to chatbots as a whole (eg, usability, classifier performance, speed), response generation (eg, comprehensibility, realism, repetitiveness), response understanding (eg, chatbot understanding as assessed by users, word error rate, concept error rate), and esthetics (eg, appearance of the virtual agent, background color, and content). CONCLUSIONS:The technical metrics of health chatbot studies were diverse, with survey designs and global usability metrics dominating. The lack of standardization and paucity of objective measures make it difficult to compare the performance of health chatbots and could inhibit advancement of the field. We suggest that researchers more frequently include metrics computed from conversation logs. In addition, we recommend the development of a framework of technical metrics with recommendations for specific circumstances for their inclusion in chatbot studies.

Project description:BackgroundChatbots are applications that can conduct natural language conversations with users. In the medical field, chatbots have been developed and used to serve different purposes. They provide patients with timely information that can be critical in some scenarios, such as access to mental health resources. Since the development of the first chatbot, ELIZA, in the late 1960s, much effort has followed to produce chatbots for various health purposes developed in different ways.ObjectiveThis study aimed to explore the technical aspects and development methodologies associated with chatbots used in the medical field to explain the best methods of development and support chatbot development researchers on their future work.MethodsWe searched for relevant articles in 8 literature databases (IEEE, ACM, Springer, ScienceDirect, Embase, MEDLINE, PsycINFO, and Google Scholar). We also performed forward and backward reference checking of the selected articles. Study selection was performed by one reviewer, and 50% of the selected studies were randomly checked by a second reviewer. A narrative approach was used for result synthesis. Chatbots were classified based on the different technical aspects of their development. The main chatbot components were identified in addition to the different techniques for implementing each module.ResultsThe original search returned 2481 publications, of which we identified 45 studies that matched our inclusion and exclusion criteria. The most common language of communication between users and chatbots was English (n=23). We identified 4 main modules: text understanding module, dialog management module, database layer, and text generation module. The most common technique for developing text understanding and dialogue management is the pattern matching method (n=18 and n=25, respectively). The most common text generation is fixed output (n=36). Very few studies relied on generating original output. Most studies kept a medical knowledge base to be used by the chatbot for different purposes throughout the conversations. A few studies kept conversation scripts and collected user data and previous conversations.ConclusionsMany chatbots have been developed for medical use, at an increasing rate. There is a recent, apparent shift in adopting machine learning-based approaches for developing chatbot systems. Further research can be conducted to link clinical outcomes to different chatbot development techniques and technical characteristics.

Project description:IntroductionMedical assistance in dying (MAID), a term encompassing both euthanasia and assisted suicide, was decriminalised in Canada in 2015. Although Bill C-14 legislated eligibility criteria under which patients could receive MAID, it did not provide guidance regarding the technical aspects of providing an assisted death. Therefore, we propose a scoping review to map the characteristics of the existing medical literature describing the medications, settings, participants and outcomes of MAID, in order to identify knowledge gaps and areas for future research.Methods and analysisWe will search electronic databases (MEDLINE, EMBASE, CINAHL, CENTRAL, PsycINFO), clinical trial registries, conference abstracts, and professional guidelines and recommendations from jurisdictions where MAID is legal, up to June 2017. Eligible report types will include technical summaries, institutional policies, practice surveys, practice guidelines and clinical studies. We will include all descriptions of MAID provision (either euthanasia or assisted suicide) in adults who have provided informed consent for MAID, for any reason, including reports where patients have provided consent to MAID in advance of the development of incapacity (eg, dementia). We will exclude reports in which patients receive involuntary euthanasia (eg, capital punishment). Two independent investigators will screen and select retrieved reports using pilot-tested screening and eligibility forms, and collect data using standardised data collection forms. We will summarise extracted data in tabular format with accompanying descriptive statistics and use narrative format to describe their clinical relevance, identify knowledge gaps and suggest topics for future research.Ethics and disseminationThis scoping review will map the range and scope of the existing literature on the provision of MAID in jurisdictions where the practice has been decriminalised. The review will be disseminated through conference presentations and publication in a peer-reviewed journal. These results will be useful to clinicians, policy makers and researchers involved with MAID.

Project description:BACKGROUND:Food control is defined as a mandatory regulatory activity of enforcement aimed at ensuring that all foods during production, handling, storage, processing, and distribution are safe, wholesome, and fit for human consumption; conform to safety and quality requirements; and are honestly and accurately labeled as prescribed by law. This applies to food served by the conventional food supply chain as well as the charitable food assistance programs (CFAPs). This review sought to map the available evidence on the food control in the CFAPs globally. METHODS:In order to identify the literature, we developed a series of search terms, as well as parameters for including articles to review the literature using African Index Medicus, PubMed, Google Scholar, and EBSCOhost (MEDLINE with full text, Academic search complete, MEDLINE) search engines. Articles were also searched through the "Cited by" search as well as citations included in the reference lists of included articles. We included studies reported in all languages and published from inception to 2018. We included studies if they presented evidence of the CFAPs, namely food banks, food charitable organizations (FCOs), pantries, community soup kitchens, and emergency shelters. We presented the results of our search using thematic analysis in order to reveal the emerging themes. RESULTS:Beyond inconsistencies with the classification of the CFAPs, our study found significant knowledge gaps in crucial areas, namely food vulnerability, food traceability, vulnerability of beneficiary populations, and lack of food control. Our search yielded a total of 23 articles, which we included in the analysis. Results show that while food is the critical commodity to saving lives, if not controlled properly, it can have an adverse effect, especially on people it is meant to benefit (the vulnerable). CONCLUSION:With no previous comprehensive review to assess what is known about food control in the CFAPs, we undertook a scoping review, focusing on mapping the key concepts, including the main sources and types of evidence available. By drawing conclusions about the overall state of research activity and identifying research gaps and priorities in the existing literature, this study provides a baseline assessment of the CFAP research published in peer-reviewed journals from inception to 2018.

Project description:BackgroundDespite reported positive results of telemonitoring effectiveness in various health care domains, this new technology is rarely used in prenatal care. A few isolated investigations were performed in the past years but with conflicting results.ObjectiveThe aim of this review was to (1) assess whether telemonitoring adds any substantial benefit to this patient population and (2) identify research gaps in this area to suggest goals for future research.MethodsThis review includes studies exploring the effectiveness of telemonitoring interventions for pregnant women reported in the English language. Due to the paucity of research in this area, all reports including uncontrolled nonrandomized and randomized controlled studies were selected.ResultsFourteen studies, which performed their data collection from 1988 to 2010, met the inclusion criteria and were published from 1995 to present; four of the 14 published papers were multicenter randomized controlled trials (RCTs), five papers were single-center RCTs, three papers were retrospective studies, one paper was an observational study, and one paper was a qualitative study. Of the 14 papers, nine were available for a risk of bias assessment: three papers were classified as low risk, one as medium risk, and five as high risk. Furthermore, of those 14 papers, 13 focused on telemonitoring for maternal outcomes, and nine of the 14 papers focused on telemonitoring for fetal or neonatal outcomes. The studies reviewed report that telemonitoring can contribute to significant reductions in health care costs, (unscheduled) face-to-face visits, low neonatal birth weight, and admissions to the neonatal intensive care unit (NICU), as well as prolonged gestational age and improved feelings of maternal satisfaction when compared with a control group. When only studies with low risk of bias were taken into account, the added value of telemonitoring became less pronounced: the only added value of telemonitoring is for pregnant women who transmitted their uterine activity by telecommunication. They had significant prolonged pregnancy survivals, and the newborns were less likely to be of low birth weight or to be admitted to the NICU. Following these results, telemonitoring can only be recommended by pregnant women at risk for preterm delivery. It is however important to consider that these studies were published in the mid-90s, which limits their direct applicability given the current technologies and practice.ConclusionsThis review shows that telemonitoring can be tentatively recommended for pregnant women at risk for preterm delivery. More recent RCTs with a blinded protocol are needed to strengthen the level of evidence around this topic and to have an insight in the added value of the technologies that are available nowadays. In addition, studies investigating patient satisfaction and economic effects in relation to telemonitoring are suggested for future research.