Dataset Information

A Framework for Analyzing and Measuring Usage and Engagement Data (AMUsED) in Digital Interventions: Viewpoint.

ABSTRACT: Trials of digital interventions can yield extensive, in-depth usage data, yet usage analyses tend to focus on broad descriptive summaries of how an intervention has been used by the whole sample. This paper proposes a novel framework to guide systematic, fine-grained usage analyses that better enables understanding of how an intervention works, when, and for whom. The framework comprises three stages to assist in the following: (1) familiarization with the intervention and its relationship to the captured data, (2) identification of meaningful measures of usage and specifying research questions to guide systematic analyses of usage data, and (3) preparation of datasheets and consideration of available analytical methods with which to examine the data. The framework can be applied to inform data capture during the development of a digital intervention and/or in the analysis of data after the completion of an evaluation trial. We will demonstrate how the framework shaped preparation and aided efficient data capture for a digital intervention to lower transmission of cold and flu viruses in the home, as well as how it informed a systematic, in-depth analysis of usage data collected from a separate digital intervention designed to promote self-management of colds and flu. The Analyzing and Measuring Usage and Engagement Data (AMUsED) framework guides systematic and efficient in-depth usage analyses that will support standardized reporting with transparent and replicable findings. These detailed findings may also enable examination of what constitutes effective engagement with particular interventions.

SUBMITTER: Miller S

PROVIDER: S-EPMC6396072 | biostudies-literature | 2019 Feb

REPOSITORIES: biostudies-literature

ACCESS DATA

Publications

A Framework for Analyzing and Measuring Usage and Engagement Data (AMUsED) in Digital Interventions: Viewpoint.

Miller Sascha S Ainsworth Ben B Yardley Lucy L Milton Alex A Weal Mark M Smith Peter P Morrison Leanne L

Journal of medical Internet research 20190215 2

Trials of digital interventions can yield extensive, in-depth usage data, yet usage analyses tend to focus on broad descriptive summaries of how an intervention has been used by the whole sample. This paper proposes a novel framework to guide systematic, fine-grained usage analyses that better enables understanding of how an intervention works, when, and for whom. The framework comprises three stages to assist in the following: (1) familiarization with the intervention and its relationship to th ...[more]

PMID: 30767905

Similar Datasets

Project description:BackgroundEvaluating engagement with an intervention is a key component of understanding its efficacy. With an increasing interest in developing behavioral interventions in the mobile health (mHealth) space, appropriate methods for evaluating engagement in this context are necessary. Data collected to evaluate mHealth interventions are often collected much more frequently than those for clinic-based interventions. Additionally, missing data on engagement is closely linked to level of engagement resulting in the potential for informative missingness. Thus, models that can accommodate intensively collected data and can account for informative missingness are required for unbiased inference when analyzing engagement with an mHealth intervention.ObjectiveThe objectives of this paper are to discuss the utility of the joint modeling approach in the analysis of longitudinal engagement data in mHealth research and to illustrate the application of this approach using data from an mHealth intervention designed to support illness management among people with schizophrenia.MethodsEngagement data from an evaluation of an mHealth intervention designed to support illness management among people with schizophrenia is analyzed. A joint model is applied to the longitudinal engagement outcome and time-to-dropout to allow unbiased inference on the engagement outcome. Results are compared to a naïve model that does not account for the relationship between dropout and engagement.ResultsThe joint model shows a strong relationship between engagement and reduced risk of dropout. Using the mHealth app 1 day more per week was associated with a 23% decreased risk of dropout (P<.001). The decline in engagement over time was steeper when the joint model was used in comparison with the naïve model.ConclusionsNaïve longitudinal models that do not account for informative missingness in mHealth data may produce biased results. Joint models provide a way to model intensively collected engagement outcomes while simultaneously accounting for the relationship between engagement and missing data in mHealth intervention research.

Project description:BACKGROUND:Digital interventions for weight management provide a unique opportunity to target daily lifestyle choices and eating behaviors over a sustained period of time. However, recent evidence has demonstrated a lack of user engagement with digital health interventions, impacting on the levels of intervention effectiveness. Thus, it is critical to identify the factors that may facilitate user engagement with digital health interventions to encourage behavior change and weight management. OBJECTIVE:The aim of this study was to identify and synthesize the available evidence to gain insights about users' perspectives on factors that affect engagement with digital interventions for weight management. METHODS:A rapid review methodology was adopted. The search strategy was executed in the following databases: Web of Science, PsycINFO, and PubMed. Studies were eligible for inclusion if they investigated users' engagement with a digital weight management intervention and were published from 2000 onwards. A narrative synthesis of data was performed on all included studies. RESULTS:A total of 11 studies were included in the review. The studies were qualitative, mixed-methods, or randomized controlled trials. Some of the studies explored features influencing engagement when using a Web-based digital intervention, others specifically explored engagement when accessing a mobile phone app, and some looked at engagement after text message (short message service, SMS) reminders. Factors influencing engagement with digital weight management interventions were found to be both user-related (eg, perceived health benefits) and digital intervention-related (eg, ease of use and the provision of personalized information). CONCLUSIONS:The findings highlight the importance of incorporating user perspectives during the digital intervention development process to encourage engagement. The review contributes to our understanding of what facilitates user engagement and points toward a coproduction approach for developing digital interventions for weight management. Particularly, it highlights the importance of thinking about user-related and digital tool-related factors from the very early stages of the intervention development process.

Project description:BACKGROUND:Multimedia interventions can provide a cost-effective solution to public health needs; however, user engagement is low. Multimedia use within specific populations such as those affected by cancer differs from that of the general population. To our knowledge, there are no frameworks on how to accurately assess usage within this population to ensure that interventions are appropriate for the end users. Therefore, a framework was developed to improve the accuracy of determining data usage. Formative work included creating a data usage framework during target audience testing for smartphone app development and analysis in a pilot study. OBJECTIVE:The purpose of this study was to develop a framework for assessing smartphone app usage among people living with cancer and their caregivers. METHODS:The frequency and duration of use were compared based on manual data extraction from two previous studies and the newly developed Assessment of Data Usage of Cancer e-Interventions (ADUCI) Framework. RESULTS:Manual extraction demonstrated that 279 logins occurred compared with 241 when the ADUCI Framework was applied. The frequency of use in each section of the app also decreased when the ADUCI Framework was used. The total duration of use was 91,256 seconds (25.3 hours) compared with 53,074 seconds (14.7 hours) when using the ADUCI Framework. The ADUCI Framework identified 38 logins with no navigation, and there were 15 discrepancies in the data where time on a specific page of the app exceeded the login time. Practice recommendations to improve user engagement and capturing usage data include tracking data use in external websites, having a login function on apps, creating a five-star page rating functionality, using the ADUCI Framework to thoroughly clean usage data, and validating the Framework between expected and observed use. CONCLUSIONS:Applying the ADUCI Framework may eliminate errors and allow for more accurate analysis of usage data in e-research projects. The Framework can also improve the process of capturing usage data by providing a guide for usage data analysis to facilitate evidence-based assessment of user engagement with apps.

Project description:BackgroundWhile many digital mental health interventions (DMHIs) have been found to be efficacious, patient engagement with DMHIs has increasingly emerged as a concern for implementation in real-world clinical settings. To address engagement, we must first understand what standard engagement levels are in the context of randomized controlled trials (RCTs) and how these compare with other treatments.ObjectiveThis scoping review aims to examine the state of reporting on intervention engagement in RCTs of mobile app-based interventions intended to treat symptoms of depression. We sought to identify what engagement metrics are and are not routinely reported as well as what the metrics that are reported reflect about standard engagement levels.MethodsWe conducted a systematic search of 7 databases to identify studies meeting our eligibility criteria, namely, RCTs that evaluated use of a mobile app-based intervention in adults, for which depressive symptoms were a primary outcome of interest. We then extracted 2 kinds of information from each article: intervention details and indices of DMHI engagement. A 5-element framework of minimum necessary DMHI engagement reporting was derived by our team and guided our data extraction. This framework included (1) recommended app use as communicated to participants at enrollment and, when reported, app adherence criteria; (2) rate of intervention uptake among those assigned to the intervention; (3) level of app use metrics reported, specifically number of uses and time spent using the app; (4) duration of app use metrics (ie, weekly use patterns); and (5) number of intervention completers.ResultsDatabase searching yielded 2083 unique records. Of these, 22 studies were eligible for inclusion. Only 64% (14/22) of studies included in this review specified rate of intervention uptake. Level of use metrics was only reported in 59% (13/22) of the studies reviewed. Approximately one-quarter of the studies (5/22, 23%) reported duration of use metrics. Only half (11/22, 50%) of the studies reported the number of participants who completed the app-based components of the intervention as intended or other metrics related to completion. Findings in those studies reporting metrics related to intervention completion indicated that between 14.4% and 93.0% of participants randomized to a DMHI condition completed the intervention as intended or according to a specified adherence criteria.ConclusionsFindings suggest that engagement was underreported and widely varied. It was not uncommon to see completion rates at or below 50% (11/22) of those participants randomized to a treatment condition or to simply see completion rates not reported at all. This variability in reporting suggests a failure to establish sufficient reporting standards and limits the conclusions that can be drawn about level of engagement with DMHIs. Based on these findings, the 5-element framework applied in this review may be useful as a minimum necessary standard for DMHI engagement reporting.

Project description:BackgroundDigital interventions offer a solution to address the high demand for mental health promotion, especially when facing physical contact restrictions or lacking accessibility. Engagement with digital interventions is critical for their effectiveness; however, retaining users' engagement throughout the intervention is challenging. It remains unclear what strategies facilitate engagement with digital interventions that target mental health promotion.ObjectiveOur aim is to conduct a scoping review to investigate user engagement strategies and methods to evaluate engagement with digital interventions that target mental health promotion in adults.MethodsThis scoping review adheres to the PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) guidelines for scoping reviews. The search was conducted in 7 electronic databases from inception to April 2020. The inclusion criteria for studies were as follows: adult (aged ≥18 years) users of digital interventions for mental health promotion from the general population; any digital intervention for mental health promotion; and user engagement strategies described in the intervention design. We extracted the following data items: study characteristics, digital intervention (type and engagement strategy), evaluation of engagement strategy (method and result specifying whether the strategy was effective at facilitating engagement), and features of engagement (extent of use and subjective experience of users).ResultsA total of 2766 studies were identified, of which 16 (0.58%) met the inclusion criteria. The 16 studies included randomized controlled trials (6/16, 37%), studies analyzing process data (5/16, 31%), observational studies (3/16, 19%), and qualitative studies (2/16, 13%). The digital interventions for mental health promotion were either web based (12/16, 75%) or mobile app based (4/16, 25%). The engagement strategies included personalized feedback about intervention content or users' mental health status; guidance regarding content and progress through e-coaching; social forums, and interactivity with peers; content gamification; reminders; and flexibility and ease of use. These engagement strategies were deemed effective based on qualitative user feedback or responses on questionnaires or tools (4/16, 25%), usability data (5/16, 31%), or both (7/16, 44%). Most studies identified personalized support in the form of e-coaching, peer support through a social platform, personalized feedback, or joint videoconference sessions as an engaging feature.ConclusionsPersonalized support during the intervention, access to social support, and personalized feedback seem to promote engagement with digital interventions for mental health promotion. These findings need to be interpreted with caution because the included studies were heterogeneous, had small sample sizes, and typically did not address engagement as the primary outcome. Despite the importance of user engagement for the effectiveness of digital interventions, this field has not yet received much attention. Further research is needed on the effectiveness of different strategies required to facilitate user engagement in digital interventions for mental health promotion.

Dataset Information

A Framework for Analyzing and Measuring Usage and Engagement Data (AMUsED) in Digital Interventions: Viewpoint.

Publications

A Framework for Analyzing and Measuring Usage and Engagement Data (AMUsED) in Digital Interventions: Viewpoint.

Similar Datasets

OmicsDI is part of the ELIXIR infrastructure

Tweets