Dataset Information

Review of Evaluation Metrics Used in Digital and Traditional Tobacco Control Campaigns.

ABSTRACT: BACKGROUND:Mass media campaigns for public health are increasingly using digital media platforms, such as web-based advertising and social media; however, there is a lack of evidence on how to best use these digital platforms for public health campaigns. To generate this evidence, appropriate campaign evaluations are needed, but with the proliferation of digital media-related metrics, there is no clear consensus on which evaluation metrics should be used. Public health campaigns are diverse in nature, so to facilitate analysis, this review has selected tobacco control campaigns as the scope of the study. OBJECTIVE:This literature review aimed to examine how tobacco control campaigns that use traditional and digital media platforms have been evaluated. METHODS:Medicine and science databases (Medical Literature Analysis and Retrieval System Online [MEDLINE], EMBASE, PsycINFO, Cumulative Index to Nursing and Allied Health Literature [CINAHL], and Scopus), and a marketing case study database (World Advertising Research Center) were searched for articles published between 2013 and 2018. Two authors established the eligibility criteria and reviewed articles for inclusion. Individual campaigns were identified from the articles, and information on campaigns and their evaluations were supplemented with searches on Google, Google Scholar, and social media platforms. Data about campaign evaluations were tabulated and mapped to a conceptual framework. RESULTS:In total, 17 campaigns were included in this review, with evaluations reported on by 51 articles, 17 marketing reports, and 4 grey literature reports. Most campaigns were from English-speaking countries, with behavioral change as the primary objective. In the process evaluations, a wide range of metrics were used to assess the reach of digital campaign activities, making comparison between campaigns difficult. Every campaign in the review, except one, reported some type of engagement impact measure, with website visits being the most commonly reported metric (11 of the 17 campaigns). Other commonly reported evaluation measures identified in this review include engagement on social media, changes in attitudes, and number of people contacting smoking cessation services. Of note, only 7 of the 17 campaigns attempted to measure media platform attribution, for example, by asking participants where they recalled seeing the campaign or using unique website tracking codes for ads on different media platforms. CONCLUSIONS:One of the key findings of this review is the numerous and diverse range of measures and metrics used in tobacco control campaign evaluations. To address this issue, we propose principles to guide the selection of digital media-related metrics for campaign evaluations, and also outline a conceptual framework to provide a coherent organization to the diverse range of metrics. Future research is needed to specifically investigate whether engagement metrics are associated with desired campaign outcomes, to determine whether reporting of engagement metrics is meaningful in campaign evaluations.

SUBMITTER: Chan L

PROVIDER: S-EPMC7448186 | biostudies-literature | 2020 Aug

REPOSITORIES: biostudies-literature

ACCESS DATA

Publications

Review of Evaluation Metrics Used in Digital and Traditional Tobacco Control Campaigns.

Chan Lilian L O'Hara Blythe B Phongsavan Philayrath P Bauman Adrian A Freeman Becky B

Journal of medical Internet research 20200811 8

<h4>Background</h4>Mass media campaigns for public health are increasingly using digital media platforms, such as web-based advertising and social media; however, there is a lack of evidence on how to best use these digital platforms for public health campaigns. To generate this evidence, appropriate campaign evaluations are needed, but with the proliferation of digital media-related metrics, there is no clear consensus on which evaluation metrics should be used. Public health campaigns are dive ...[more]

PMID: 32348272

Similar Datasets

Project description:The purpose of this study was to compare a new digitized cognitive test battery, Minnemera, with its correspondent traditional paper-based cognitive tests. Eighty-one healthy adults between the ages of 21 and 85 participated in the study. Participants performed the two different test versions (traditional paper-based and digitized) with an interval of four weeks between the tests. Test presentation (the order of the test versions presented) was counterbalanced in order to control for any possible test learning effects. The digitized tests were constructed so that there were only minor differences when compared to the traditional paper-based tests. Test results from the paper-based and digitized versions of the cognitive screening were compared within individuals by means of a correlation analysis and equivalence tests. The effects of demographic variables (age, gender and level of education) and test presentation were explored for each test measure and each test version through linear regression models. For each test measure, a significant correlation between traditional and digitized version was observed ranging between r = 0.34 and r = 0.67 with a median of r = 0.53 (corresponding to a large effect size). Score equivalence was observed for five out of six tests. In line with previous traditional cognitive studies, age was found to be the most prominent predictor of performance in all digitized tests, with younger participants performing better than older adults. Gender was the second strongest predictor, where women outperformed men in tests measuring verbal memory; men performed better than women in tests with a strong visual component. Finally, the educational level of the test subjects had an effect on executive functions, with a higher educational level linked to a better inhibition response and working memory span. This study suggests that the tests in the Minnemera cognitive screening battery are acceptably comparable to the traditional paper-based counterparts.

Project description:This review reports the pork quality attributes, Warner-Bratzler Shear Force, Slice Shear Force, Star Probe, pH, marbling, color (Minolta L*/L or Hunter L*/L), and sensory tenderness evaluation, in control groups used in comparative nutrition experiments over the past 20 yr. The original aim of this study was to evaluate if changes in pork quality based on the above metrics occurred over time. To address this question, it was anticipated that data may come from 3 sources with decreasing relevance: representative retail pork surveys, representative post-harvest carcass surveys, and control groups from comparative nutrition experiments. To identify the study population, a review of studies reported in Centre for Agricultural Biosciences International Abstracts (Web of Knowledge; 1994-2014) was conducted. Two national level surveys of retail pork and 146 relevant nutritional experiments studies, with 228 control groups, were identified by the search. It was not possible to conduct a meta-analysis of the retail pork surveys based on only 2 time points. For the comparative studies, a random effects meta-analysis was conducted with year as a covariate to assess the impact of time on the outcome. In the absence of modifiers, there was no evidence of meaningful change in the mean Warner-Bratzler Shear Force, pH, color, marbling, or sensory scores over the study period. There was evidence of substantial between-study heterogeneity in the characteristics of control pigs used over the years for Warner-Bratzler Shear Force and measures of color. The absence of publicly-available representative surveys of pork quality meant the changes in pork quality over time were not clear. If changes in pork quality have occurred, the data suggest that pigs used as controls in experiments may have become less representative of commercial pigs over time and the translatability of study findings from nutrition experiments might be reduced over time. Alternately, if commercial pigs have not changed, then control pigs reflect this. The study does not address if control groups in other experimental intervention studies had similar tenderness patterns as reported here for nutritional interventions. A large amount of potentially available data was excluded from the analysis due to incomplete reporting in the original study reports.

Project description:BackgroundSerious games are now widely used in many contexts, including psychological research and clinical use. One area of growing interest is that of cognitive assessment, which seeks to measure different cognitive functions such as memory, attention, and perception. Measuring these functions at both the population and individual levels can inform research and indicate health issues. Attention is an important function to assess, as an accurate measure of attention can help diagnose many common disorders, such as attention-deficit/hyperactivity disorder and dementia. However, using games to assess attention poses unique problems, as games inherently manipulate attention through elements such as sound effects, graphics, and rewards, and research on adding game elements to assessments (ie, gamification) has shown mixed results. The process for developing cognitive tasks is robust, with high psychometric standards that must be met before these tasks are used for assessment. Although games offer more diverse approaches for assessment, there is no standard for how they should be developed or evaluated.ObjectiveTo better understand the field and provide guidance to interdisciplinary researchers, we aim to answer the question: How are digital games used for the cognitive assessment of attention made and measured?MethodsWe searched several databases for papers that described a digital game used to assess attention that could be deployed remotely without specialized hardware. We used Rayyan, a systematic review software, to screen the records before conducting a systematic review.ResultsThe initial database search returned 49,365 papers. Our screening process resulted in a total of 74 papers that used a digital game to measure cognitive functions related to attention. Across the studies in our review, we found three approaches to making assessment games: gamifying cognitive tasks, creating custom games based on theories of cognition, and exploring potential assessment properties of commercial games. With regard to measuring the assessment properties of these games (eg, how accurately they assess attention), we found three approaches: comparison to a traditional cognitive task, comparison to a clinical diagnosis, and comparison to knowledge of cognition; however, most studies in our review did not evaluate the game's properties (eg, if participants enjoyed the game).ConclusionsOur review provides an overview of how games used for the assessment of attention are developed and evaluated. We further identified three barriers to advancing the field: reliance on assumptions, lack of evaluation, and lack of integration and standardization. We then recommend the best practices to address these barriers. Our review can act as a resource to help guide the field toward more standardized approaches and rigorous evaluation required for the widespread adoption of assessment games.

Project description:BACKGROUND:Dialog agents (chatbots) have a long history of application in health care, where they have been used for tasks such as supporting patient self-management and providing counseling. Their use is expected to grow with increasing demands on health systems and improving artificial intelligence (AI) capability. Approaches to the evaluation of health care chatbots, however, appear to be diverse and haphazard, resulting in a potential barrier to the advancement of the field. OBJECTIVE:This study aims to identify the technical (nonclinical) metrics used by previous studies to evaluate health care chatbots. METHODS:Studies were identified by searching 7 bibliographic databases (eg, MEDLINE and PsycINFO) in addition to conducting backward and forward reference list checking of the included studies and relevant reviews. The studies were independently selected by two reviewers who then extracted data from the included studies. Extracted data were synthesized narratively by grouping the identified metrics into categories based on the aspect of chatbots that the metrics evaluated. RESULTS:Of the 1498 citations retrieved, 65 studies were included in this review. Chatbots were evaluated using 27 technical metrics, which were related to chatbots as a whole (eg, usability, classifier performance, speed), response generation (eg, comprehensibility, realism, repetitiveness), response understanding (eg, chatbot understanding as assessed by users, word error rate, concept error rate), and esthetics (eg, appearance of the virtual agent, background color, and content). CONCLUSIONS:The technical metrics of health chatbot studies were diverse, with survey designs and global usability metrics dominating. The lack of standardization and paucity of objective measures make it difficult to compare the performance of health chatbots and could inhibit advancement of the field. We suggest that researchers more frequently include metrics computed from conversation logs. In addition, we recommend the development of a framework of technical metrics with recommendations for specific circumstances for their inclusion in chatbot studies.

Project description:IntroductionTobacco control mass media campaigns (MMCs) can be effective generally, but little is known about their effects among people with mental illness. The objectives of this study were to systematically review: (1) Whether tobacco control MMCs affect smoking-related outcomes among people with mental illness. (2) Cost-effectiveness.Aims and methodsData sources: MEDLINE, Embase, PsycInfo, Web of Science, CINAHL, the Cochrane Library (searched March 2021), reference lists of included articles and relevant systematic reviews. Study eligibility criteria: Population: Adults with mental illness and experience of smoking tobacco and/or using other nicotine-containing products. Intervention/exposure: Tobacco control MMC messages. Comparator: No exposure, other tobacco control intervention(s), no comparator. Primary outcome: Changes in quitting behaviors. Study design: All primary research. Quantitative data were appraised using the EPHPP tool, qualitative data using CASP's Studies Checklist. Data were synthesized narratively.ResultsEight studies were included, seven were at high risk of bias. There was inconclusive evidence of the effect of MMCs on quit attempts and intentions to quit among people with mental illness. Increasing advertisement exposure did not increase quit attempts or intentions to quit among those with mental illness, however, increased exposure to an advertisement that addressed smoking and mental health did. None of the studies assessed cost-effectiveness.ConclusionsFindings should be interpreted with caution as data are limited and of low or moderate quality. There is evidence to suggest that tobacco control MMCs have limited impact on those with mental illness, although campaigns that are specific to smoking and mental health may be effective.ImplicationsThere is a paucity of good-quality evidence of the effect of tobacco control MMC messages among people with mental illness. Careful consideration should be given to the design of future studies that evaluate MMCs in order to minimize the risk of bias, establish causality, and ensure the findings reflect real-world implementation. Further research should examine the need for MMC messages that address mental health.

Dataset Information

Review of Evaluation Metrics Used in Digital and Traditional Tobacco Control Campaigns.

Publications

Review of Evaluation Metrics Used in Digital and Traditional Tobacco Control Campaigns.

Similar Datasets

OmicsDI is part of the ELIXIR infrastructure

Tweets