Dataset Information

Mobile app validation: a digital health scorecard approach.

ABSTRACT: While digital health solutions continue to grow in number and in complexity, the ability for stakeholders in healthcare to easily discern quality lags far behind. This challenge is in part due to the lack of a transparent and standardized approach to validation. Evaluation of mobile health applications (apps) is further burdened by low barriers to development and direct-to-user marketing, leading to a crowded and confusing landscape. In this context, we investigated the pragmatic application of a previously described framework for digital health validation, the Digital Health Scorecard, in a cohort of 22 popular mobile health oncology apps. The apps evaluated using this framework performed poorly, scoring 49.4% across all evaluation criteria as a group. Performance across component domains varied considerably with cost scoring highest at 100%, usability at 56.7%, technical at 37.3%, and clinical at 15.9%. satisfaction of prospectively determined end-user requirements derived from patient, family, and clinician consensus scored 37.2%. While cost outperformed consistently and usability was adequate, the results also suggested that apps suffered from significant technical limitations, were of limited clinical value, and generally did not do what end users wanted. These large gaps further support the need for transparent and standardized evaluation to help all stakeholders in healthcare improve the quality of mobile health.

SUBMITTER: Sedhom R

PROVIDER: S-EPMC8282811 | biostudies-literature |

REPOSITORIES: biostudies-literature

ACCESS DATA

Similar Datasets

Project description:BackgroundThe classic Marshmallow Test, where children were offered a choice between one small but immediate reward (eg, one marshmallow) or a larger reward (eg, two marshmallows) if they waited for a period of time, instigated a wealth of research on the relationships among impulsive responding, self-regulation, and clinical and life outcomes. Impulsivity is a hallmark feature of self-regulation failures that lead to poor health decisions and outcomes, making understanding and treating impulsivity one of the most important constructs to tackle in building a culture of health. Despite a large literature base, impulsivity measurement remains difficult due to the multidimensional nature of the construct and limited methods of assessment in daily life. Mobile devices and the rise of mobile health (mHealth) have changed our ability to assess and intervene with individuals remotely, providing an avenue for ambulatory diagnostic testing and interventions. Longitudinal studies with mobile devices can further help to understand impulsive behaviors and variation in state impulsivity in daily life.ObjectiveThe aim of this study was to develop and validate an impulsivity mHealth diagnostics and monitoring app called Digital Marshmallow Test (DMT) using both the Apple and Android platforms for widespread dissemination to researchers, clinicians, and the general public.MethodsThe DMT app was developed using Apple's ResearchKit (iOS) and Android's ResearchStack open source frameworks for developing health research study apps. The DMT app consists of three main modules: self-report, ecological momentary assessment, and active behavioral and cognitive tasks. We conducted a study with a 21-day assessment period (N=116 participants) to validate the novel measures of the DMT app.ResultsWe used a semantic differential scale to develop self-report trait and momentary state measures of impulsivity as part of the DMT app. We identified three state factors (inefficient, thrill seeking, and intentional) that correlated highly with established measures of impulsivity. We further leveraged momentary semantic differential questions to examine intraindividual variability, the effect of daily life, and the contextual effect of mood on state impulsivity and daily impulsive behaviors. Our results indicated validation of the self-report sematic differential and related results, and of the mobile behavioral tasks, including the Balloon Analogue Risk Task and Go-No-Go task, with relatively low validity of the mobile Delay Discounting task. We discuss the design implications of these results to mHealth research.ConclusionsThis study demonstrates the potential for assessing different facets of trait and state impulsivity during everyday life and in clinical settings using the DMT mobile app. The DMT app can be further used to enhance our understanding of the individual facets that underlie impulsive behaviors, as well as providing a promising avenue for digital interventions.Trial registrationClinicalTrials.gov NCT03006653; https://www.clinicaltrials.gov/ct2/show/NCT03006653.

Project description:BackgroundThe usability of mobile health (mHealth) apps needs to be effectively evaluated before they are officially approved to be used to deliver health interventions. To this end, the mHealth App Usability Questionnaire (MAUQ) has been designed and proved valid and reliable in assessing the usability of mHealth apps. However, this English questionnaire needs to be translated into other languages, adapted, and validated before being utilized to evaluate the usability of mHealth apps.ObjectiveThis study aims to improve, further adapt, and validate the Chinese version of the MAUQ (C-MAUQ; interactive for patients) on Left-handed Doctor, one of the most popular "reaching out to patients" interactive mHealth apps with chatbot function in China, to test the reliability and cross-cultural adaptability of the questionnaire.MethodsThe MAUQ (interactive for patients) has been translated into Chinese and validated for its reliability on Good Doctor, one of the most influential "reaching out to patients" mHealth apps without chatbot function in China. After asking for the researchers' approval to use this Chinese version, we adjusted and further adapted the C-MAUQ by checking it against the original English version and improving its comprehensibility, readability, idiomaticity, and cross-cultural adaptability. Following a trial survey completed by 50 respondents on wenjuanxing, the most popular online questionnaire platform in China, the improved version of the C-MAUQ (I-C-MAUQ) was finally used to evaluate the usability of Left-handed Doctor through an online questionnaire survey (answered by 322 participants) on wenjuanxing, to test its internal consistency, reliability, and validity.ResultsThe I-C-MAUQ still retained the 21 items and 3 dimensions of the original MAUQ: 8 items for usability and satisfaction, 6 items for system information arrangement, and 7 items for efficiency. The translation problems in the C-MAUQ, including (1) redundancy, (2) incompleteness, (3) misuse of parts of speech, (4) choice of inappropriate words, (5) incomprehensibility, and (6) cultural difference-induced improper translation, were improved. As shown in the analysis of data obtained through the online survey, the I-C-MAUQ had a better internal consistency (ie, the correlation coefficient between the score of each item and the total score of the questionnaire determined within the range of 0.861-0.938; P<.01), reliability (Cronbach α=.988), and validity (Kaiser-Meyer-Olkin=0.973), compared with the C-MAUQ. It was effectively used to test the usability of Left-handed Doctor, eliciting over 80% of informants' positive attitudes toward this mHealth app.ConclusionsThe I-C-MAUQ is highly reliable and valid for Left-handed Doctor, and suitable for testing the usability of interactive mHealth apps used by patients in China. This finding further confirms the cross-cultural validity, reliability, and adaptability of the MAUQ. We identified certain factors influencing the perceived usability of mHealth apps, including users' age, gender, education, profession, and possibly previous experience with mHealth apps and the chatbot function of such apps. Most notably, we found a wider acceptance of this new technology among young Chinese female college students who were more engaged in the interaction with health care chatbots. The age-, gender-, and profession-induced preference for new digital health interventions in China aligns with the findings in other similar studies in America and Malaysia. This preference identifies areas for further research on the social, cultural, and gender adaptation of health technologies.

Project description:BackgroundAlmost a quarter or more than a fifth of children in the United Kingdom are overweight or obese by the time they start school. The UK Department of Health and Social Care's national policy for combating childhood obesity has critical outcomes centered on sugar and caloric consumption reduction. Health Education England has developed two digital apps for families with children up to 15 years and for their associated health care professionals to provide a digital learning resource and tool aimed at encouraging healthy lifestyles to prevent obesity.ObjectiveThis feasibility study assesses the usability and acceptability of Health Education England's NoObesity app for undertaking activities to improve families' diet and physical activity. The purpose of the study is to evaluate the app's influence on self-efficacy and goal setting and to determine what can be learnt to improve its design for future studies, if there is evidence of adoption and sustainability.MethodsThe study population will include 20 to 40 families and their linked health care professionals. Considering issues related to digital access associated with socioeconomic status and the impact on information technology use, study recruitment will be regionally focused in a low socioeconomic status area. The study will last for 9 months (3-month intervention period and 6-month follow-up). The evaluations of feasibility, acceptability, and usability will be conducted using the following scales and theoretical frameworks: (1) system usability scale; (2) Reach Effectiveness Adoption Implementation Maintenance framework; (3) Bandura model of health promotion; and (4) Nonadoption, Abandonment, and Challenges to the Scale-up, Spread, and Suitability framework. App use will be captured and quantitatively analyzed for net use patterns (eg, number of screens viewed, number of logins, cumulative minutes using the app, number of plans made, and number of times goals met) and to triangulate qualitative feedback from study participants.ResultsThis study was funded in March 2019 by Health Education England and received University of Oxford Medical Sciences Interdivisional Research Ethics Committee approval on January 31, 2020 (R62092/RE001). At manuscript submission, study recruitment is pending, and expected results will be published in 2021.ConclusionsThis study will provide evidence on the NoObesity app's influence on self-efficacy and goal-setting and determine what can be learnt to improve its design for future studies, if there is evidence of adoption and sustainability.International registered report identifier (irrid)PRR1-10.2196/18068.

Project description:BackgroundPhysical inactivity is a global issue that affects people's health and productivity. With the advancement of mobile technologies, many apps have been developed to facilitate health self-management. However, few studies have examined the effectiveness of these mobile health (mHealth) apps in motivating exercise adherence.ObjectiveThis study aims to demonstrate the enhanced action design research (ADR) process and improve the design of mHealth apps for exercise self-management. Specifically, we investigate whether sending motivational messages improves adherence to exercise plans, whether the motivational effect is affected by personality, the impact of message type and repetition, and the process of involving a field experiment in the design process and learning new design principles from the results.MethodsThis formative research was conducted by proposing an enhanced ADR process. We incorporated a field experiment into the process to iteratively refine and evaluate the design until it converges into a final mHealth app. We used the Apple ResearchKit to develop the mHealth app and promoted it via trainers at their gyms. We targeted users who used the app for at least two months. Participants were randomly assigned to 1 of the 12 groups in a 2×3×2 factorial design and remained blinded to the assigned intervention. The groups were defined based on personality type (thinking or feeling), message type (emotional, logical, or none), and repetition (none or once). Participants with different personality types received tailored and repeated messages. Finally, we used the self-reported completion rate to measure participants' adherence level to exercise plans. By analyzing users' usage patterns, we could verify, correct, and enhance the mHealth app design principles.ResultsIn total, 160 users downloaded the app, and 89 active participants remained during the 2-month period. The results suggest a significant main effect of personality type and repetition and a significant interaction effect between personality type and repetition. The adherence rate of people with feeling personality types was 18.15% higher than that of people with thinking types. Emotional messages were more effective than logical messages in motivating exercise adherence. Although people received repeated messages, they were more likely to adhere to exercise plans. With repeated reminders, the adherence rates of people with thinking personality types were significantly improved by 27.34% (P<.001).ConclusionsThis study contributes to the literature on mHealth apps. By incorporating a field experiment into the ADR process, we demonstrate the benefit of combining design science and field experiments. This study also contributes to the research on mHealth apps. The principles learned from this study can be applied to improve the effectiveness of mHealth apps. The app design can be considered a foundation for the development of more advanced apps for specific diseases, such as diabetes and asthma, in future research.

Project description:BackgroundPregnant women are often unaware of the potential risks that working conditions can cause to them and their unborn child. A mobile health (mHealth) app, the Pregnancy and Work (P and W) app, developed by a multidisciplinary team and based on an evidence-based guideline for occupational physicians, aims to provide advice on work adjustment during pregnancy.ObjectiveThis study evaluates the usability of the mHealth P and W app and the perceived usefulness of the work advice, the main goal of the app, by potential end users.MethodsA total of 12 working pregnant women participated in think aloud usability sessions and performed 9 tasks. All think aloud sessions were recorded, transcribed, and coanalyzed. The usability problems were rated for their severity in accordance with Nielsen severity scale. The completion rates and time taken for completion of tasks were registered. In addition, participants were questioned on demographics and user characteristics and were asked to evaluate the value of the app by filling in the Intrinsic Motivation Inventory (IMI) score and the System Usability Scale (SUS) questionnaire.ResultsIn total, 82 usability problems with a severity ≥1 were identified, of which 40 had severity ≥3. The main usability problems concerned the interpretation of terminology used in the app's questionnaires and difficulties in finding and understanding the work advice. Furthermore, 10 out of 12 participants were able to open the work advice page in the app. Only 7 out of these 10 participants understood and intended to follow the work advice. The overall mean IMI score was relatively high (5 out of 7), indicating that the participants did indeed value the use of the app. This IMI score corresponded to the overall mean SUS score (68 out of 100) and the mean grade given to the P and W app (7 out of 10).ConclusionsThis think aloud usability study showed that the information provided in the P and W app was considered valuable by the end users, working pregnant women, and it meets their needs; however, usability issues severely impacted the perceived usefulness of the work advice given in the app.

Project description:BackgroundClinical governance of medical mobile apps is challenging, and there is currently no standard method for assessing the quality of such apps. In 2018, the National Institute for Health and Care Excellence (NICE) developed a framework for assessing the required level of evidence for digital health technologies (DHTs), as determined by their clinical function. The framework can potentially be used to assess mobile apps, which are a subset of DHTs. To be used reliably in this context, the framework must allow unambiguous classification of an app's clinical function.ObjectiveThe objective of this study was to determine whether mobile health apps could be reliably classified using the NICE evidence standards framework for DHTs.MethodsWe manually extracted app titles, screenshots, and content descriptions for all apps listed on the National Health Service (NHS) Apps Library website on July 12, 2019; none of the apps were downloaded. Using this information, 2 mobile health (mHealth) researchers independently classified each app to one of the 4 functional tiers (ie, 1, 2, 3a, and 3b) described in the NICE digital technologies evaluation framework. Coders also answered contextual questions from the framework to identify whether apps were deemed to be higher risk. Agreement between coders was assessed using Cohen ? statistic.ResultsIn total, we assessed 76 apps from the NHS Apps Library. There was classification agreement for 42 apps. Of these, 0 apps were unanimously classified into Tier 1; 24, into Tier 2; 15, into Tier 3a; and 3, into Tier 3b. There was disagreement between coders in 34/76 cases (45%); interrater agreement was poor (Cohen ?=0.32, 95% CI 0.16-0.47). Further investigation of disagreements highlighted 5 main explanatory themes: apps that did not correspond to any tier, apps that corresponded to multiple tiers, ambiguous tier descriptions, ambiguous app descriptions, and coder error.ConclusionsThe current iteration of the NICE evidence standards framework for DHTs did not allow mHealth researchers to consistently and unambiguously classify digital health mobile apps listed on the NHS app library according to their functional tier.

Dataset Information

Mobile app validation: a digital health scorecard approach.

Similar Datasets

OmicsDI is part of the ELIXIR infrastructure

Tweets