Dataset Information

A scoping review of the evaluation and effectiveness of technical assistance.

ABSTRACT:

Background

Although the benefits of evidence-based practices (EBPs) for advancing community outcomes are well-recognized, challenges with the uptake of EBPs are considerable. Technical assistance (TA) is a core capacity building strategy that has been widely used to support EBP implementation and other community development and improvement efforts. Yet despite growing reliance on TA, no reviews have systematically examined the evaluation of TA across varying implementation contexts and capacity building aims. This study draws on two decades of peer-reviewed publications to summarize the evidence on the evaluation and effectiveness of TA.

Methods

Guided by Arksey and O'Malley's six-stage methodological framework, we used a scoping review methodology to map research on TA evaluation. We included peer-reviewed articles published in English between 2000 and 2020. Our search involved five databases: Business Source Complete, Cumulative Index to Nursing and Allied Health Literature (CINAHL), Education Resources Information Center (ERIC), PsycInfo, and PubMed.

Results

A total of 125 evaluation research studies met the study criteria. Findings indicate that publications have increased over the last two decades, signaling a growth in the recognition and reporting of TA. Technical assistance is being implemented across diverse settings, often serving socially vulnerable and under-resourced populations. Most evaluation research studies involved summative evaluations, with TA outcomes mostly reported at the organizational level. Only 5% of the studies examined sustainability of TA outcomes. This review also demonstrates that there is a lack of consistent standards regarding the definition of TA and the level of reporting across relevant TA evaluation categories (e.g., cadence of contact, and directionality).

Conclusions

Advances in the science and practice of TA hinge on understanding what aspects of TA are effective and when, how, and for whom these aspects of TA are effective. Addressing these core questions requires (i) a standard definition for TA; (ii) more robust and rigorous evaluation research designs that involve comparison groups and assessment of direct, indirect, and longitudinal outcomes; (iii) increased use of reliable and objective TA measures; and (iv) development of reporting standards. We view this scoping review as a foundation for improving the state of the science and practice of evaluating TA.

SUBMITTER: Scott VC

PROVIDER: S-EPMC9238031 | biostudies-literature |

REPOSITORIES: biostudies-literature

ACCESS DATA

Similar Datasets

Project description:One of the most basic needs globally, food assistance refers to the multitude of programs, both governmental and non-governmental, to improve food access and consumption by food-insecure individuals and families. Despite the importance of digital and mobile Health (mHealth) strategies in food insecurity contexts, little is known about their specific use in food assistance programs. Therefore, the purpose of this study was to address that gap by conducting a scoping review of the literature. Keywords were defined within the concepts of food assistance and digital technology. The search included relevant peer-reviewed and grey literature from 2011 to 2021. Excluded articles related to agriculture and non-digital strategies. PRISMA guidelines were followed to perform a partnered, two-round scoping literature review. The final synthesis included 39 studies of which most (84.6%) were from the last five years and United States-based (93.2%). The top three types of articles or studies included text and opinion, qualitative research, and website, application, or model development (17.9%). The top three types of digital tools were websites (56.4%), smartphone applications (20.5%), and chatbots (5.1%). Nineteen digital features were identified as desirable. Most tools included just one or two features. The most popular feature to include was online shopping (n = 14), followed by inventory management, and client tracking. Digital tools for individual food assistance represent an opportunity for equitable and stable access to programs that can enhance or replace in-person services. While this review identified 39 tools, all are in early development and/or implementation stages. Review findings highlight an overall lack of these tools, an absence of user-centered design in their development, and a critical need for research on their effectiveness globally. Further analysis and testing of current digital tool usage and interventions examining the health and food security impacts of such tools should be explored in future studies, including in the context of pandemics, where digital tools allow for help from a distance.

Project description:Significance:Prior research suggests that Community Engagement and Planning (CEP) for coalition support compared with Resources for Services (RS) for program technical assistance to implement depression quality improvement programs improves 6- and 12-month client mental-health related quality of life (MHRQL); however, effects for clients with multiple chronic medical conditions (MCC) are unknown. Objective:To explore effectiveness of CEP vs RS in MCC and non-MCC subgroups. Design:Secondary analyses of a cluster-randomized trial. Setting:93 health care and community-based programs in two neighborhoods. Participants:Of 4,440 clients screened, 1,322 depressed (Patient Health Questionnaire, PHQ8) provided contact information, 1,246 enrolled and 1,018 (548 with ?3 MCC) completed baseline, 6- or 12-month surveys. Intervention:CEP or RS for implementing depression quality improvement programs. Outcomes and Analyses:Primary: depression (PHQ9 <10), poor MHRQL (Short Form Health Survey, SF-12<40); Secondary: mental wellness, good physical health, behavioral health hospitalization, chronic homelessness risk, work/workloss days, services use at 6 and 12 months. End-point regressions were used to estimate intervention effects on outcomes for subgroups with ?3 MCC, non-MCC, and intervention-by-MCC interactions (exploratory). Results:Among MCC clients at 6 months, CEP vs RS lowered likelihoods of depression and poor MHRQL; increased likelihood of mental wellness; reduced work-loss days among employed and likelihoods of ?4 behavioral-health hospitalization nights and chronic homelessness risk, while increasing faith-based and park community center depression services; and at 12 months, likelihood of good physical health and park community center depression services use (each P<.05). There were no significant interactions or primary outcome effects for non-MCC. Conclusions:CEP was more effective than RS in improving 6-month primary outcomes among depressed MCC clients, without significant interactions.

Project description:BackgroundChatbots are applications that can conduct natural language conversations with users. In the medical field, chatbots have been developed and used to serve different purposes. They provide patients with timely information that can be critical in some scenarios, such as access to mental health resources. Since the development of the first chatbot, ELIZA, in the late 1960s, much effort has followed to produce chatbots for various health purposes developed in different ways.ObjectiveThis study aimed to explore the technical aspects and development methodologies associated with chatbots used in the medical field to explain the best methods of development and support chatbot development researchers on their future work.MethodsWe searched for relevant articles in 8 literature databases (IEEE, ACM, Springer, ScienceDirect, Embase, MEDLINE, PsycINFO, and Google Scholar). We also performed forward and backward reference checking of the selected articles. Study selection was performed by one reviewer, and 50% of the selected studies were randomly checked by a second reviewer. A narrative approach was used for result synthesis. Chatbots were classified based on the different technical aspects of their development. The main chatbot components were identified in addition to the different techniques for implementing each module.ResultsThe original search returned 2481 publications, of which we identified 45 studies that matched our inclusion and exclusion criteria. The most common language of communication between users and chatbots was English (n=23). We identified 4 main modules: text understanding module, dialog management module, database layer, and text generation module. The most common technique for developing text understanding and dialogue management is the pattern matching method (n=18 and n=25, respectively). The most common text generation is fixed output (n=36). Very few studies relied on generating original output. Most studies kept a medical knowledge base to be used by the chatbot for different purposes throughout the conversations. A few studies kept conversation scripts and collected user data and previous conversations.ConclusionsMany chatbots have been developed for medical use, at an increasing rate. There is a recent, apparent shift in adopting machine learning-based approaches for developing chatbot systems. Further research can be conducted to link clinical outcomes to different chatbot development techniques and technical characteristics.

Project description:BackgroundDialog agents (chatbots) have a long history of application in health care, where they have been used for tasks such as supporting patient self-management and providing counseling. Their use is expected to grow with increasing demands on health systems and improving artificial intelligence (AI) capability. Approaches to the evaluation of health care chatbots, however, appear to be diverse and haphazard, resulting in a potential barrier to the advancement of the field.ObjectiveThis study aims to identify the technical (nonclinical) metrics used by previous studies to evaluate health care chatbots.MethodsStudies were identified by searching 7 bibliographic databases (eg, MEDLINE and PsycINFO) in addition to conducting backward and forward reference list checking of the included studies and relevant reviews. The studies were independently selected by two reviewers who then extracted data from the included studies. Extracted data were synthesized narratively by grouping the identified metrics into categories based on the aspect of chatbots that the metrics evaluated.ResultsOf the 1498 citations retrieved, 65 studies were included in this review. Chatbots were evaluated using 27 technical metrics, which were related to chatbots as a whole (eg, usability, classifier performance, speed), response generation (eg, comprehensibility, realism, repetitiveness), response understanding (eg, chatbot understanding as assessed by users, word error rate, concept error rate), and esthetics (eg, appearance of the virtual agent, background color, and content).ConclusionsThe technical metrics of health chatbot studies were diverse, with survey designs and global usability metrics dominating. The lack of standardization and paucity of objective measures make it difficult to compare the performance of health chatbots and could inhibit advancement of the field. We suggest that researchers more frequently include metrics computed from conversation logs. In addition, we recommend the development of a framework of technical metrics with recommendations for specific circumstances for their inclusion in chatbot studies.

Project description:IntroductionMedical assistance in dying (MAID), a term encompassing both euthanasia and assisted suicide, was decriminalised in Canada in 2015. Although Bill C-14 legislated eligibility criteria under which patients could receive MAID, it did not provide guidance regarding the technical aspects of providing an assisted death. Therefore, we propose a scoping review to map the characteristics of the existing medical literature describing the medications, settings, participants and outcomes of MAID, in order to identify knowledge gaps and areas for future research.Methods and analysisWe will search electronic databases (MEDLINE, EMBASE, CINAHL, CENTRAL, PsycINFO), clinical trial registries, conference abstracts, and professional guidelines and recommendations from jurisdictions where MAID is legal, up to June 2017. Eligible report types will include technical summaries, institutional policies, practice surveys, practice guidelines and clinical studies. We will include all descriptions of MAID provision (either euthanasia or assisted suicide) in adults who have provided informed consent for MAID, for any reason, including reports where patients have provided consent to MAID in advance of the development of incapacity (eg, dementia). We will exclude reports in which patients receive involuntary euthanasia (eg, capital punishment). Two independent investigators will screen and select retrieved reports using pilot-tested screening and eligibility forms, and collect data using standardised data collection forms. We will summarise extracted data in tabular format with accompanying descriptive statistics and use narrative format to describe their clinical relevance, identify knowledge gaps and suggest topics for future research.Ethics and disseminationThis scoping review will map the range and scope of the existing literature on the provision of MAID in jurisdictions where the practice has been decriminalised. The review will be disseminated through conference presentations and publication in a peer-reviewed journal. These results will be useful to clinicians, policy makers and researchers involved with MAID.

Project description:BACKGROUND:Food control is defined as a mandatory regulatory activity of enforcement aimed at ensuring that all foods during production, handling, storage, processing, and distribution are safe, wholesome, and fit for human consumption; conform to safety and quality requirements; and are honestly and accurately labeled as prescribed by law. This applies to food served by the conventional food supply chain as well as the charitable food assistance programs (CFAPs). This review sought to map the available evidence on the food control in the CFAPs globally. METHODS:In order to identify the literature, we developed a series of search terms, as well as parameters for including articles to review the literature using African Index Medicus, PubMed, Google Scholar, and EBSCOhost (MEDLINE with full text, Academic search complete, MEDLINE) search engines. Articles were also searched through the "Cited by" search as well as citations included in the reference lists of included articles. We included studies reported in all languages and published from inception to 2018. We included studies if they presented evidence of the CFAPs, namely food banks, food charitable organizations (FCOs), pantries, community soup kitchens, and emergency shelters. We presented the results of our search using thematic analysis in order to reveal the emerging themes. RESULTS:Beyond inconsistencies with the classification of the CFAPs, our study found significant knowledge gaps in crucial areas, namely food vulnerability, food traceability, vulnerability of beneficiary populations, and lack of food control. Our search yielded a total of 23 articles, which we included in the analysis. Results show that while food is the critical commodity to saving lives, if not controlled properly, it can have an adverse effect, especially on people it is meant to benefit (the vulnerable). CONCLUSION:With no previous comprehensive review to assess what is known about food control in the CFAPs, we undertook a scoping review, focusing on mapping the key concepts, including the main sources and types of evidence available. By drawing conclusions about the overall state of research activity and identifying research gaps and priorities in the existing literature, this study provides a baseline assessment of the CFAP research published in peer-reviewed journals from inception to 2018.