Dataset Information

Scoping review: Development and assessment of evaluation frameworks of mobile health apps for recommendations to consumers.

ABSTRACT:

Objective

The study sought to review the different assessment items that have been used within existing health app evaluation frameworks aimed at individual, clinician, or organizational users, and to analyze the scoring and evaluation methods used in these frameworks.

Materials and methods

We searched multiple bibliographic databases and conducted backward searches of reference lists, using search terms that were synonyms of "health apps," "evaluation," and "frameworks." The review covered publications from 2011 to April 2020. Studies on health app evaluation frameworks and studies that elaborated on the scaling and scoring mechanisms applied in such frameworks were included.

Results

Ten common domains were identified across general health app evaluation frameworks. A list of 430 assessment criteria was compiled across 97 identified studies. The most frequently used scaling mechanism was a 5-point Likert scale. Most studies have adopted summary statistics to generate the total scoring of each app, and the most popular approach taken was the calculation of mean or average scores. Other frameworks did not use any scaling or scoring mechanism and adopted criteria-based, pictorial, or descriptive approaches, or "threshold" filter.

Discussion

There is wide variance in the approaches to evaluating health apps within published frameworks, and this variance leads to ongoing uncertainty in how to evaluate health apps.

Conclusions

A new evaluation framework is needed that can integrate the full range of evaluative criteria within one structure, and provide summative guidance on health app rating, to support individual app users, clinicians, and health organizations in choosing or recommending the best health app.

SUBMITTER: Hensher M

PROVIDER: S-EPMC8263081 | biostudies-literature |

REPOSITORIES: biostudies-literature

ACCESS DATA

Similar Datasets

Project description:BackgroundPrivacy has always been a concern, especially in the health domain. The proliferation of mobile health (mHealth) apps has led to a large amount of sensitive data being generated. Some authors have performed privacy assessments of mHealth apps. They have evaluated diverse privacy components; however, different authors have used different criteria for their assessments.ObjectiveThis scoping review aims to understand how privacy is assessed for mHealth apps, focusing on the components, scales, criteria, and scoring methods used. A simple taxonomy to categorize the privacy assessments of mHealth apps based on component evaluation is also proposed.MethodsWe followed the methodology defined by Arksey and O'Malley to conduct a scoping review. Included studies were categorized based on the privacy component, which was assessed using the proposed taxonomy.ResultsThe database searches retrieved a total of 710 citations-24 of them met the defined selection criteria, and data were extracted from them. Even though the inclusion criteria considered articles published since 2009, all the studies that were ultimately included were published from 2014 onward. Although 12 papers out of 24 (50%) analyzed only privacy, 8 (33%) analyzed both privacy and security. Moreover, 4 papers (17%) analyzed full apps, with privacy being just part of the assessment. The evaluation criteria used by authors were heterogeneous and were based on their experience, the literature, and/or existing legal frameworks. Regarding the set of items used for the assessments, each article defined a different one. Items included app permissions, analysis of the destination, analysis of the content of communications, study of the privacy policy, use of remote storage, and existence of a password to access the app, among many others. Most of the included studies provided a scoring method that enables the comparison of privacy among apps.ConclusionsThe privacy assessment of mHealth apps is a complex task, as the criteria used by different authors for their evaluations are very heterogeneous. Although some studies about privacy assessment have been conducted, a very large set of items to evaluate privacy has been used up until now. In-app information and privacy policies are primarily utilized by the scientific community to extract privacy information from mHealth apps. The creation of a scale based on more objective criteria is a desirable step forward for privacy assessment in the future.

Project description:ObjectivesDespite an estimated 300 000 mobile health apps on the market, there remains no consensus around helping patients and clinicians select safe and effective apps. In 2018, our team drew on existing evaluation frameworks to identify salient categories and create a new framework endorsed by the American Psychiatric Association (APA). We have since created a more expanded and operational framework Mhealth Index and Navigation Database (MIND) that aligns with the APA categories but includes objective and auditable questions (105). We sought to survey the existing space, conducting a review of all mobile health app evaluation frameworks published since 2018, and demonstrate the comprehensiveness of this new model by comparing it to existing and emerging frameworks.DesignWe conducted a scoping review of mobile health app evaluation frameworks.Data sourcesReferences were identified through searches of PubMed, EMBASE and PsychINFO with publication date between January 2018 and October 2020.Eligibility criteriaPapers were selected for inclusion if they meet the predetermined eligibility criteria-presenting an evaluation framework for mobile health apps with patient, clinician or end user-facing questions.Data extraction and synthesisTwo reviewers screened the literature separately and applied the inclusion criteria. The data extracted from the papers included: author and dates of publication, source affiliation, country of origin, name of framework, study design, description of framework, intended audience/user and framework scoring system. We then compiled a collection of more than 1701 questions across 79 frameworks. We compared and grouped these questions using the MIND framework as a reference. We sought to identify the most common domains of evaluation while assessing the comprehensiveness and flexibility-as well as any potential gaps-of MIND.ResultsNew app evaluation frameworks continue to emerge and expand. Since our 2019 review of the app evaluation framework space, more frameworks include questions around privacy (43) and clinical foundation (57), reflecting an increased focus on issues of app security and evidence base. The majority of mapped frameworks overlapped with at least half of the MIND categories. The results of this search have informed a database (apps.digitalpsych.org) that users can access today.ConclusionAs the number of app evaluation frameworks continues to rise, it is becoming difficult for users to select both an appropriate evaluation tool and to find an appropriate health app. This review provides a comparison of what different app evaluation frameworks are offering, where the field is converging and new priorities for improving clinical guidance.

Project description:BackgroundThe importance of mobile health (mHealth) apps is growing. Independent of the technologies used, mHealth apps bring more functionality into the hands of users. In the health context, mHealth apps play an important role in providing information and services to patients, offering health care professionals ways to monitor vital parameters or consult patients remotely. The importance of confidentiality in health care and the opaqueness of transport security in apps make the latter an important research subject.ObjectiveThis study aimed to (1) identify relevant security concerns on the server side of mHealth apps, (2) test a subset of mHealth apps regarding their vulnerability to those concerns, and (3) compare the servers used by mHealth apps with servers used in all domains.MethodsServer security characteristics relevant to the security of mHealth apps were assessed, presented, and discussed. To evaluate servers, appropriate tools were selected. Apps from the Android and iOS app stores were selected and tested, and the results for functional and other backend servers were evaluated.ResultsThe 60 apps tested communicate with 823 servers. Of these, 291 were categorized as functional backend servers, and 44 (44/291, 15.1%) of these received a rating below the A range (A+, A, and A-) by Qualys SSL Labs. A chi-square test was conducted against the number of servers receiving such ratings from SSL Pulse by Qualys SSL Labs. It was found that the tested servers from mHealth apps received significantly fewer ratings below the A range (P<.001). The internationally available apps from the test set performed significantly better than those only available in the German stores (alpha=.05; P=.03). Of the 60 apps, 28 (28/60, 47%) were found using at least one functional backend server that received a rating below the A range from Qualys SSL Labs, endangering confidentiality, authenticity, and integrity of the data displayed. The number of apps that used at least one entirely unsecured connection was 20 (20/60, 33%) when communicating with functional backend servers. It was also found that a majority of apps used advertising, tracking, or external content provider servers. When looking at all nonfunctional backend servers, 48 (48/60, 80%) apps used at least one server that received a rating below the A range.ConclusionsThe results show that although servers in the mHealth domain perform significantly better regarding their security, there are still problems with the configuration of some. The most severe problems observed can expose patient communication with health care professionals, be exploited to display false or harmful information, or used to send data to an app facilitating further damage on the device. Following the recommendations for mHealth app developers, the most regularly observed security issues can be avoided or mitigated.

Project description:BACKGROUND:Mobile health (mHealth) apps for pediatric chronic conditions are growing in availability and challenge investigators to conduct rigorous evaluations that keep pace with mHealth innovation. Traditional research methods are poorly suited to operationalize the agile, iterative trials required to evidence and optimize these digitally mediated interventions. OBJECTIVE:We sought to contribute a resource to support the quantification, analysis, and visualization of analytic indicators of effective engagement with mHealth apps for chronic conditions. METHODS:We applied user-centered design methods to design and develop an Analytics Platform to Evaluate Effective Engagement (APEEE) with consumer mHealth apps for chronic conditions and implemented the platform to analyze both retrospective and prospective data generated from a smartphone-based pain self-management app called iCanCope for young people with chronic pain. RESULTS:Through APEEE, we were able to automate the process of defining, operationalizing, and evaluating effective engagement with iCanCope. Configuring the platform to integrate with the app was feasible and provided investigators with a resource to consolidate, analyze, and visualize engagement data generated by participants in real time. Preliminary efforts to evaluate APEEE showed that investigators perceived the platform to be an acceptable evaluative resource and were satisfied with its design, functionality, and performance. Investigators saw potential in APEEE to accelerate and augment evidence generation and expressed enthusiasm for adopting the platform to support their evaluative practice once fully implemented. CONCLUSIONS:Dynamic, real-time analytic platforms may provide investigators with a powerful means to characterize the breadth and depth of mHealth app engagement required to achieve intended health outcomes. Successful implementation of APEEE into evaluative practice may contribute to the realization of effective and evidence-based mHealth care.

Project description:BackgroundDespite the emergence of app evaluation tools, there remains no well-defined process receptive to diverse local needs, rigorous standards, and current content. The need for such a process to assist in the implementation of app evaluation across all medical fields is evident. Such a process has the potential to increase stakeholder engagement and catalyze interest and engagement with present-day app evaluation models.ObjectiveThis study aimed to develop and pilot test the Technology Evaluation and Assessment Criteria for Health apps (TEACH-apps).MethodsTailoring a well-known implementation framework, Replicating Effective Programs, we present a new process to approach the challenges faced in implementing app evaluation tools today. As a culmination of our experience implementing this process and feedback from stakeholders, we present the four-part process to aid the implementation of mobile health technology. This paper outlines the theory, evidence, and initial versions of the process.ResultsThe TEACH-apps process is designed to be broadly usable and widely applicable across all fields of health. The process comprises four parts: (1) preconditions (eg, gathering apps and considering local needs), (2) preimplementation (eg, customizing criteria and offering digital skills training), (3) implementation (eg, evaluating apps and creating educational handouts), and (4) maintenance and evolution (eg, repeating the process every 90 days and updating content). TEACH-apps has been tested internally at our hospital, and there is growing interest in partnering health care facilities to test the system at their sites.ConclusionsThis implementation framework introduces a process that equips stakeholders, clinicians, and users with the foundational tools to make informed decisions around app use and increase app evaluation engagement. The application of this process may lead to the selection of more culturally appropriate and clinically relevant tools in health care.

Project description:BACKGROUND:Modern smartphones contain sophisticated high-end hardware features, offering high computational capabilities at extremely manageable costs and have undoubtedly become an integral part in users' daily life. Additionally, smartphones offer a well-established ecosystem that is easily discoverable and accessible via the marketplaces of differing mobile platforms, thus encouraging the development of many smartphone apps. Such apps are not exclusively used for entertainment purposes but are also commonplace in health care and medical use. A variety of those health and medical apps exist within the context of tinnitus, a phantom sound perception in the absence of any physical external source. OBJECTIVE:In this paper, we shed light on existing smartphone apps addressing tinnitus by providing an up-to-date overview. METHODS:Based on PRISMA guidelines, we systematically searched and identified existing smartphone apps on the most prominent app markets, namely Google Play Store and Apple App Store. In addition, we applied the Mobile App Rating Scale (MARS) to evaluate and assess the apps in terms of their general quality and in-depth user experience. RESULTS:Our systematic search and screening of smartphone apps yielded a total of 34 apps (34 Android apps, 26 iOS apps). The mean MARS scores (out of 5) ranged between 2.65-4.60. The Tinnitus Peace smartphone app had the lowest score (mean 2.65, SD 0.20), and Sanvello-Stress and Anxiety Help had the highest MARS score (mean 4.60, SD 0.10). The interrater agreement was substantial (Fleiss ?=0.74), the internal consistency was excellent (Cronbach ?=.95), and the interrater reliability was found to be both high and excellent-Guttman ?6=0.94 and intraclass correlation, ICC(2,k) 0.94 (95% CI 0.91-0.97), respectively. CONCLUSIONS:This work demonstrated that there exists a plethora of smartphone apps for tinnitus. All of the apps received MARS scores higher than 2, suggesting that they all have some technical functional value. However, nearly all identified apps were lacking in terms of scientific evidence, suggesting the need for stringent clinical validation of smartphone apps in future. To the best of our knowledge, this work is the first to systematically identify and evaluate smartphone apps within the context of tinnitus.

Project description:For most women of reproductive age, assessing menstrual health and fertility typically involves regular visits to a gynecologist or another clinician. While these evaluations provide critical information on an individual's reproductive health status, they typically rely on memory-based self-reports, and the results are rarely, if ever, assessed at the population level. In recent years, mobile apps for menstrual tracking have become very popular, allowing us to evaluate the reliability and tracking frequency of millions of self-observations, thereby providing an unparalleled view, both in detail and scale, on menstrual health and its evolution for large populations. In particular, the primary aim of this study was to describe the tracking behavior of the app users and their overall observation patterns in an effort to understand if they were consistent with previous small-scale medical studies. The secondary aim was to investigate whether their precision allowed the detection and estimation of ovulation timing, which is critical for reproductive and menstrual health. Retrospective self-observation data were acquired from two mobile apps dedicated to the application of the sympto-thermal fertility awareness method, resulting in a dataset of more than 30 million days of observations from over 2.7 million cycles for two hundred thousand users. The analysis of the data showed that up to 40% of the cycles in which users were seeking pregnancy had recordings every single day. With a modeling approach using Hidden Markov Models to describe the collected data and estimate ovulation timing, it was found that follicular phases average duration and range were larger than previously reported, with only 24% of ovulations occurring at cycle days 14 to 15, while the luteal phase duration and range were in line with previous reports, although short luteal phases (10 days or less) were more frequently observed (in up to 20% of cycles). The digital epidemiology approach presented here can help to lead to a better understanding of menstrual health and its connection to women's health overall, which has historically been severely understudied.