Scoping review: Development and assessment of evaluation frameworks of mobile health apps for recommendations to consumers.
Ontology highlight
ABSTRACT: Objective
The study sought to review the different assessment items that have been used within existing health app evaluation frameworks aimed at individual, clinician, or organizational users, and to analyze the scoring and evaluation methods used in these frameworks.Materials and methods
We searched multiple bibliographic databases and conducted backward searches of reference lists, using search terms that were synonyms of "health apps," "evaluation," and "frameworks." The review covered publications from 2011 to April 2020. Studies on health app evaluation frameworks and studies that elaborated on the scaling and scoring mechanisms applied in such frameworks were included.Results
Ten common domains were identified across general health app evaluation frameworks. A list of 430 assessment criteria was compiled across 97 identified studies. The most frequently used scaling mechanism was a 5-point Likert scale. Most studies have adopted summary statistics to generate the total scoring of each app, and the most popular approach taken was the calculation of mean or average scores. Other frameworks did not use any scaling or scoring mechanism and adopted criteria-based, pictorial, or descriptive approaches, or "threshold" filter.Discussion
There is wide variance in the approaches to evaluating health apps within published frameworks, and this variance leads to ongoing uncertainty in how to evaluate health apps.Conclusions
A new evaluation framework is needed that can integrate the full range of evaluative criteria within one structure, and provide summative guidance on health app rating, to support individual app users, clinicians, and health organizations in choosing or recommending the best health app.
SUBMITTER: Hensher M
PROVIDER: S-EPMC8263081 | biostudies-literature |
REPOSITORIES: biostudies-literature
Similar Datasets
Project description:BackgroundPrivacy has always been a concern, especially in the health domain. The proliferation of mobile health (mHealth) apps has led to a large amount of sensitive data being generated. Some authors have performed privacy assessments of mHealth apps. They have evaluated diverse privacy components; however, different authors have used different criteria for their assessments.ObjectiveThis scoping review aims to understand how privacy is assessed for mHealth apps, focusing on the components, scales, criteria, and scoring methods used. A simple taxonomy to categorize the privacy assessments of mHealth apps based on component evaluation is also proposed.MethodsWe followed the methodology defined by Arksey and O'Malley to conduct a scoping review. Included studies were categorized based on the privacy component, which was assessed using the proposed taxonomy.ResultsThe database searches retrieved a total of 710 citations-24 of them met the defined selection criteria, and data were extracted from them. Even though the inclusion criteria considered articles published since 2009, all the studies that were ultimately included were published from 2014 onward. Although 12 papers out of 24 (50%) analyzed only privacy, 8 (33%) analyzed both privacy and security. Moreover, 4 papers (17%) analyzed full apps, with privacy being just part of the assessment. The evaluation criteria used by authors were heterogeneous and were based on their experience, the literature, and/or existing legal frameworks. Regarding the set of items used for the assessments, each article defined a different one. Items included app permissions, analysis of the destination, analysis of the content of communications, study of the privacy policy, use of remote storage, and existence of a password to access the app, among many others. Most of the included studies provided a scoring method that enables the comparison of privacy among apps.ConclusionsThe privacy assessment of mHealth apps is a complex task, as the criteria used by different authors for their evaluations are very heterogeneous. Although some studies about privacy assessment have been conducted, a very large set of items to evaluate privacy has been used up until now. In-app information and privacy policies are primarily utilized by the scientific community to extract privacy information from mHealth apps. The creation of a scale based on more objective criteria is a desirable step forward for privacy assessment in the future.
| S-EPMC7367524 | biostudies-literature
Project description:ObjectivesDespite an estimated 300 000 mobile health apps on the market, there remains no consensus around helping patients and clinicians select safe and effective apps. In 2018, our team drew on existing evaluation frameworks to identify salient categories and create a new framework endorsed by the American Psychiatric Association (APA). We have since created a more expanded and operational framework Mhealth Index and Navigation Database (MIND) that aligns with the APA categories but includes objective and auditable questions (105). We sought to survey the existing space, conducting a review of all mobile health app evaluation frameworks published since 2018, and demonstrate the comprehensiveness of this new model by comparing it to existing and emerging frameworks.DesignWe conducted a scoping review of mobile health app evaluation frameworks.Data sourcesReferences were identified through searches of PubMed, EMBASE and PsychINFO with publication date between January 2018 and October 2020.Eligibility criteriaPapers were selected for inclusion if they meet the predetermined eligibility criteria-presenting an evaluation framework for mobile health apps with patient, clinician or end user-facing questions.Data extraction and synthesisTwo reviewers screened the literature separately and applied the inclusion criteria. The data extracted from the papers included: author and dates of publication, source affiliation, country of origin, name of framework, study design, description of framework, intended audience/user and framework scoring system. We then compiled a collection of more than 1701 questions across 79 frameworks. We compared and grouped these questions using the MIND framework as a reference. We sought to identify the most common domains of evaluation while assessing the comprehensiveness and flexibility-as well as any potential gaps-of MIND.ResultsNew app evaluation frameworks continue to emerge and expand. Since our 2019 review of the app evaluation framework space, more frameworks include questions around privacy (43) and clinical foundation (57), reflecting an increased focus on issues of app security and evidence base. The majority of mapped frameworks overlapped with at least half of the MIND categories. The results of this search have informed a database (apps.digitalpsych.org) that users can access today.ConclusionAs the number of app evaluation frameworks continues to rise, it is becoming difficult for users to select both an appropriate evaluation tool and to find an appropriate health app. This review provides a comparison of what different app evaluation frameworks are offering, where the field is converging and new priorities for improving clinical guidance.
| S-EPMC7986656 | biostudies-literature
Project description:BackgroundCancer patients are increasingly using mobile health (mHealth) apps to take control of their health. Many studies have explored their efficiency, content, usability, and adherence; however, these apps have created a new set of privacy challenges, as they store personal and sensitive data.ObjectiveThe purpose of this study was to refine and evaluate a scale based on the General Data Protection Regulation and assess the fairness of privacy policies of mHealth apps.MethodsBased on the experience gained from our previous work, we redefined some of the items and scores of our privacy scale. Using the new version of our scale, we conducted a case study in which we analyzed the privacy policies of cancer Android apps. A systematic search of cancer mobile apps was performed in the Spanish version of the Google Play website.ResultsThe redefinition of certain items reduced discrepancies between reviewers. Thus, use of the scale was made easier, not only for the reviewers but also for any other potential users of our scale. Assessment of the privacy policies revealed that 29% (9/31) of the apps included in the study did not have a privacy policy, 32% (10/31) had a score over 50 out of a maximum of 100 points, and 39% (12/31) scored fewer than 50 points.ConclusionsIn this paper, we present a scale for the assessment of mHealth apps that is an improved version of our previous scale with adjusted scores. The results showed a lack of fairness in the mHealth app privacy policies that we examined, and the scale provides developers with a tool to evaluate their privacy policies.
| S-EPMC7420637 | biostudies-literature
Project description:BackgroundMobile health applications (mHealth apps) are increasingly being used to perform tasks that are conventionally performed by general practitioners (GPs), such as those involved in promoting health, preventing disease, diagnosis, treatment, monitoring, and support for health services. This raises an important question: can mobile apps replace GPs? This study aimed to systematically search for and identify mobile apps that can perform GP tasks.MethodsA scoping review was carried out. The Google Play Store and Apple App Store were searched for mobile apps, using search terms derived from the UK Royal College of General Practitioners (RCGP) guideline on GPs' core capabilities and competencies. A manual search was also performed to identify additional apps.ResultsThe final analysis included 17 apps from the Google Play Store and Apple App Store, and 21 apps identified by the manual search. mHealth apps were found to have the potential to replace GPs for tasks such as recording medical history and making diagnoses; performing some physical examinations; supporting clinical decision making and management; assisting in urgent, long-term, and disease-specific care; and health promotion. In contrast, mHealth apps were unable to perform medical procedures, appropriately utilise other professionals, and coordinate a team-based approach.ConclusionsThis scoping review highlights the functions of mHealth apps that can potentially replace GP tasks. Future research should focus on assessing the performance and quality of mHealth apps in comparison with that of real doctors.
| S-EPMC6945711 | biostudies-literature
Project description:Digital health and mobile medical apps (MMAs) have shown great promise in transforming health care, but their adoption in clinical care has been unsatisfactory, and regulatory guidance and coverage decisions have been lacking or incomplete. A multidimensional assessment framework for regulatory, policymaking, health technology assessment, and coverage purposes based on the MMA lifecycle is needed. A targeted review of relevant policy documents from international sources was conducted to map current MMA assessment frameworks, to formulate 10 recommendations, subsequently shared amongst an expert panel of key stakeholders. Recommendations go beyond economic dimensions such as cost and economic evaluation and also include MMA development and update, classification and evidentiary requirements, performance and maintenance monitoring, usability testing, clinical evidence requirements, safety and security, equity considerations, organizational assessment, and additional outcome domains (patient empowerment and environmental impact). The COVID-19 pandemic greatly expanded the use of MMAs, but temporary policies governing their use and oversight need consolidation through well-developed frameworks to support decision-makers, producers and introduction into clinical care processes, especially in light of the strong international, cross-border character of MMAs, the new EU medical device and health technology assessment regulations, and the Next Generation EU funding earmarked for health digitalization.
| S-EPMC9545972 | biostudies-literature
Project description:BackgroundThe importance of mobile health (mHealth) apps is growing. Independent of the technologies used, mHealth apps bring more functionality into the hands of users. In the health context, mHealth apps play an important role in providing information and services to patients, offering health care professionals ways to monitor vital parameters or consult patients remotely. The importance of confidentiality in health care and the opaqueness of transport security in apps make the latter an important research subject.ObjectiveThis study aimed to (1) identify relevant security concerns on the server side of mHealth apps, (2) test a subset of mHealth apps regarding their vulnerability to those concerns, and (3) compare the servers used by mHealth apps with servers used in all domains.MethodsServer security characteristics relevant to the security of mHealth apps were assessed, presented, and discussed. To evaluate servers, appropriate tools were selected. Apps from the Android and iOS app stores were selected and tested, and the results for functional and other backend servers were evaluated.ResultsThe 60 apps tested communicate with 823 servers. Of these, 291 were categorized as functional backend servers, and 44 (44/291, 15.1%) of these received a rating below the A range (A+, A, and A-) by Qualys SSL Labs. A chi-square test was conducted against the number of servers receiving such ratings from SSL Pulse by Qualys SSL Labs. It was found that the tested servers from mHealth apps received significantly fewer ratings below the A range (P<.001). The internationally available apps from the test set performed significantly better than those only available in the German stores (alpha=.05; P=.03). Of the 60 apps, 28 (28/60, 47%) were found using at least one functional backend server that received a rating below the A range from Qualys SSL Labs, endangering confidentiality, authenticity, and integrity of the data displayed. The number of apps that used at least one entirely unsecured connection was 20 (20/60, 33%) when communicating with functional backend servers. It was also found that a majority of apps used advertising, tracking, or external content provider servers. When looking at all nonfunctional backend servers, 48 (48/60, 80%) apps used at least one server that received a rating below the A range.ConclusionsThe results show that although servers in the mHealth domain perform significantly better regarding their security, there are still problems with the configuration of some. The most severe problems observed can expose patient communication with health care professionals, be exploited to display false or harmful information, or used to send data to an app facilitating further damage on the device. Following the recommendations for mHealth app developers, the most regularly observed security issues can be avoided or mitigated.
| S-EPMC6364205 | biostudies-literature
Project description:BACKGROUND:Mobile health (mHealth) apps for pediatric chronic conditions are growing in availability and challenge investigators to conduct rigorous evaluations that keep pace with mHealth innovation. Traditional research methods are poorly suited to operationalize the agile, iterative trials required to evidence and optimize these digitally mediated interventions. OBJECTIVE:We sought to contribute a resource to support the quantification, analysis, and visualization of analytic indicators of effective engagement with mHealth apps for chronic conditions. METHODS:We applied user-centered design methods to design and develop an Analytics Platform to Evaluate Effective Engagement (APEEE) with consumer mHealth apps for chronic conditions and implemented the platform to analyze both retrospective and prospective data generated from a smartphone-based pain self-management app called iCanCope for young people with chronic pain. RESULTS:Through APEEE, we were able to automate the process of defining, operationalizing, and evaluating effective engagement with iCanCope. Configuring the platform to integrate with the app was feasible and provided investigators with a resource to consolidate, analyze, and visualize engagement data generated by participants in real time. Preliminary efforts to evaluate APEEE showed that investigators perceived the platform to be an acceptable evaluative resource and were satisfied with its design, functionality, and performance. Investigators saw potential in APEEE to accelerate and augment evidence generation and expressed enthusiasm for adopting the platform to support their evaluative practice once fully implemented. CONCLUSIONS:Dynamic, real-time analytic platforms may provide investigators with a powerful means to characterize the breadth and depth of mHealth app engagement required to achieve intended health outcomes. Successful implementation of APEEE into evaluative practice may contribute to the realization of effective and evidence-based mHealth care.
| S-EPMC6320392 | biostudies-other
Project description:BackgroundThe number of mobile health apps is rapidly increasing. This means that consumers are faced with a bewildering array of choices, and finding the benefit of such apps may be challenging. The significant international burden of breast cancer (BC) and the potential of mobile health apps to improve medical and public health practices mean that such apps will likely be important because of their functionalities in daily life. As the app market has grown exponentially, several review studies have scrutinized cancer- or BC-related apps. However, those reviews concentrated on the availability of the apps and relied on user ratings to decide on app quality. To minimize subjectivity in quality assessment, quantitative methods to assess BC-related apps are required.ObjectiveThe purpose of this study is to analyze the content and quality of BC-related apps to provide useful information for end users and clinicians.MethodsBased on a stepwise systematic approach, we analyzed apps related to BC, including those related to prevention, detection, treatment, and survivor support. We used the keywords "breast cancer" in English and Korean to identify commercially available apps in the Google Play and App Store. The apps were then independently evaluated by 2 investigators to determine their eligibility for inclusion. The content and quality of the apps were analyzed using objective frameworks and the Mobile App Rating Scale (MARS), respectively.ResultsThe initial search identified 1148 apps, 69 (6%) of which were included. Most BC-related apps provided information, and some recorded patient-generated health data, provided psychological support, and assisted with medication management. The Kendall coefficient of concordance between the raters was 0.91 (P<.001). The mean MARS score (range: 1-5) of the apps was 3.31 (SD 0.67; range: 1.94-4.53). Among the 5 individual dimensions, functionality had the highest mean score (4.37, SD 0.42) followed by aesthetics (3.74, SD 1.14). Apps that only provided information on BC prevention or management of its risk factors had lower MARS scores than those that recorded medical data or patient-generated health data. Apps that were developed >2 years ago, or by individuals, had significantly lower MARS scores compared to other apps (P<.001).ConclusionsThe quality of BC-related apps was generally acceptable according to the MARS, but the gaps between the highest- and lowest-rated apps were large. In addition, apps using personalized data were of higher quality than those merely giving related information, especially after treatment in the cancer care continuum. We also found that apps that had been updated within 1 year and developed by private companies had higher MARS scores. This may imply that there are criteria for end users and clinicians to help choose the right apps for better clinical outcomes.
| S-EPMC9999256 | biostudies-literature
Project description:BackgroundDespite the emergence of app evaluation tools, there remains no well-defined process receptive to diverse local needs, rigorous standards, and current content. The need for such a process to assist in the implementation of app evaluation across all medical fields is evident. Such a process has the potential to increase stakeholder engagement and catalyze interest and engagement with present-day app evaluation models.ObjectiveThis study aimed to develop and pilot test the Technology Evaluation and Assessment Criteria for Health apps (TEACH-apps).MethodsTailoring a well-known implementation framework, Replicating Effective Programs, we present a new process to approach the challenges faced in implementing app evaluation tools today. As a culmination of our experience implementing this process and feedback from stakeholders, we present the four-part process to aid the implementation of mobile health technology. This paper outlines the theory, evidence, and initial versions of the process.ResultsThe TEACH-apps process is designed to be broadly usable and widely applicable across all fields of health. The process comprises four parts: (1) preconditions (eg, gathering apps and considering local needs), (2) preimplementation (eg, customizing criteria and offering digital skills training), (3) implementation (eg, evaluating apps and creating educational handouts), and (4) maintenance and evolution (eg, repeating the process every 90 days and updating content). TEACH-apps has been tested internally at our hospital, and there is growing interest in partnering health care facilities to test the system at their sites.ConclusionsThis implementation framework introduces a process that equips stakeholders, clinicians, and users with the foundational tools to make informed decisions around app use and increase app evaluation engagement. The application of this process may lead to the selection of more culturally appropriate and clinically relevant tools in health care.
| S-EPMC7484774 | biostudies-literature
Project description:Mental health conditions can have significant negative impacts on wellbeing and healthcare systems. Despite their high prevalence worldwide, there is still insufficient recognition and accessible treatments. Many mobile apps are available to the general population that aim to support mental health needs; however, there is limited evidence of their effectiveness. Mobile apps for mental health are beginning to incorporate artificial intelligence and there is a need for an overview of the state of the literature on these apps. The purpose of this scoping review is to provide an overview of the current research landscape and knowledge gaps regarding the use of artificial intelligence in mobile health apps for mental health. The Preferred Reporting Items for Systematic Reviews and Meta-Analyses extension for Scoping Reviews (PRISMA-ScR) and Population, Intervention, Comparator, Outcome, and Study types (PICOS) frameworks were used to structure the review and the search. PubMed was systematically searched for randomised controlled trials and cohort studies published in English since 2014 that evaluate artificial intelligence- or machine learning-enabled mobile apps for mental health support. Two reviewers collaboratively screened references (MMI and EM), selected studies for inclusion based on the eligibility criteria and extracted the data (MMI and CL), which were synthesised in a descriptive analysis. 1,022 studies were identified in the initial search and 4 were included in the final review. The mobile apps investigated incorporated different artificial intelligence and machine learning techniques for a variety of purposes (risk prediction, classification, and personalisation) and aimed to address a wide range of mental health needs (depression, stress, and suicide risk). The studies' characteristics also varied in terms of methods, sample size, and study duration. Overall, the studies demonstrated the feasibility of using artificial intelligence to support mental health apps, but the early stages of the research and weaknesses in the study designs highlight the need for more research into artificial intelligence- and machine learning-enabled mental health apps and stronger evidence of their effectiveness. This research is essential and urgent, considering the easy availability of these apps to a large population.
| S-EPMC9931284 | biostudies-literature