Dataset Information

Evaluating evaluation frameworks: a scoping review of frameworks for assessing health apps.

ABSTRACT:

Objectives

Despite an estimated 300 000 mobile health apps on the market, there remains no consensus around helping patients and clinicians select safe and effective apps. In 2018, our team drew on existing evaluation frameworks to identify salient categories and create a new framework endorsed by the American Psychiatric Association (APA). We have since created a more expanded and operational framework Mhealth Index and Navigation Database (MIND) that aligns with the APA categories but includes objective and auditable questions (105). We sought to survey the existing space, conducting a review of all mobile health app evaluation frameworks published since 2018, and demonstrate the comprehensiveness of this new model by comparing it to existing and emerging frameworks.

Design

We conducted a scoping review of mobile health app evaluation frameworks.

Data sources

References were identified through searches of PubMed, EMBASE and PsychINFO with publication date between January 2018 and October 2020.

Eligibility criteria

Papers were selected for inclusion if they meet the predetermined eligibility criteria-presenting an evaluation framework for mobile health apps with patient, clinician or end user-facing questions.

Data extraction and synthesis

Two reviewers screened the literature separately and applied the inclusion criteria. The data extracted from the papers included: author and dates of publication, source affiliation, country of origin, name of framework, study design, description of framework, intended audience/user and framework scoring system. We then compiled a collection of more than 1701 questions across 79 frameworks. We compared and grouped these questions using the MIND framework as a reference. We sought to identify the most common domains of evaluation while assessing the comprehensiveness and flexibility-as well as any potential gaps-of MIND.

Results

New app evaluation frameworks continue to emerge and expand. Since our 2019 review of the app evaluation framework space, more frameworks include questions around privacy (43) and clinical foundation (57), reflecting an increased focus on issues of app security and evidence base. The majority of mapped frameworks overlapped with at least half of the MIND categories. The results of this search have informed a database (apps.digitalpsych.org) that users can access today.

Conclusion

As the number of app evaluation frameworks continues to rise, it is becoming difficult for users to select both an appropriate evaluation tool and to find an appropriate health app. This review provides a comparison of what different app evaluation frameworks are offering, where the field is converging and new priorities for improving clinical guidance.

SUBMITTER: Lagan S

PROVIDER: S-EPMC7986656 | biostudies-literature |

REPOSITORIES: biostudies-literature

ACCESS DATA

Similar Datasets

Project description:BackgroundPrivacy has always been a concern, especially in the health domain. The proliferation of mobile health (mHealth) apps has led to a large amount of sensitive data being generated. Some authors have performed privacy assessments of mHealth apps. They have evaluated diverse privacy components; however, different authors have used different criteria for their assessments.ObjectiveThis scoping review aims to understand how privacy is assessed for mHealth apps, focusing on the components, scales, criteria, and scoring methods used. A simple taxonomy to categorize the privacy assessments of mHealth apps based on component evaluation is also proposed.MethodsWe followed the methodology defined by Arksey and O'Malley to conduct a scoping review. Included studies were categorized based on the privacy component, which was assessed using the proposed taxonomy.ResultsThe database searches retrieved a total of 710 citations-24 of them met the defined selection criteria, and data were extracted from them. Even though the inclusion criteria considered articles published since 2009, all the studies that were ultimately included were published from 2014 onward. Although 12 papers out of 24 (50%) analyzed only privacy, 8 (33%) analyzed both privacy and security. Moreover, 4 papers (17%) analyzed full apps, with privacy being just part of the assessment. The evaluation criteria used by authors were heterogeneous and were based on their experience, the literature, and/or existing legal frameworks. Regarding the set of items used for the assessments, each article defined a different one. Items included app permissions, analysis of the destination, analysis of the content of communications, study of the privacy policy, use of remote storage, and existence of a password to access the app, among many others. Most of the included studies provided a scoring method that enables the comparison of privacy among apps.ConclusionsThe privacy assessment of mHealth apps is a complex task, as the criteria used by different authors for their evaluations are very heterogeneous. Although some studies about privacy assessment have been conducted, a very large set of items to evaluate privacy has been used up until now. In-app information and privacy policies are primarily utilized by the scientific community to extract privacy information from mHealth apps. The creation of a scale based on more objective criteria is a desirable step forward for privacy assessment in the future.

Project description:IntroductionMany regions in the world are using the population health approach and require a means to measure the health of their population of interest. Population health frameworks provide a theoretical grounding for conceptualization of population health and therefore a logical basis for selection of indicators. The aim of this scoping review was to provide an overview and summary of the characteristics of existing population health frameworks that have been used to conceptualize the measurement of population health.MethodsWe used the Population, Concept and Context (PCC) framework to define eligibility criteria of frameworks. We were interested in frameworks applicable for general populations, that contained components of measurement of health with or without its antecedents and applied at the population level or used a population health approach. Eligible reports of eligible frameworks should include at least domains and subdomains, purpose, or indicators. We searched 5 databases (Pubmed, EMBASE, Web of Science, NYAM Grey Literature Report, and OpenGrey), governmental and organizational sites on Google and websites of selected organizations using keywords from the PCC framework. Characteristics of the frameworks were summarized descriptively and narratively.ResultsFifty-seven frameworks were included. The majority originated from the US (46%), Europe (23%) and Canada (19%). Apart from 1 framework developed for rural populations and 2 for indigenous populations, the rest were for general urban populations. The numbers of domains, subdomains and indicators were highly variable. Health status and social determinants of health were the most common domains across all frameworks. Different frameworks had different priorities and therefore focus on different domains.ConclusionKey domains common across frameworks other than health status were social determinants of health, health behaviours and healthcare system performance. The results in this review serve as a useful resource for governments and healthcare organizations for informing their population health measurement efforts.

Project description:BackgroundUsability is a key factor affecting the acceptance of mobile health applications (mHealth apps) for elderly individuals, but traditional usability evaluation methods may not be suitable for use in this population because of aging barriers. The objectives of this study were to identify, explore, and summarize the current state of the literature on the usability evaluation of mHealth apps for older adults and to incorporate these methods into the appropriate evaluation stage.MethodsElectronic searches were conducted in 10 databases. Inclusion criteria were articles focused on the usability evaluation of mHealth apps designed for older adults. The included studies were classified according to the mHealth app usability evaluation framework, and the suitability of evaluation methods for use among the elderly was analyzed.ResultsNinety-six articles met the inclusion criteria. Research activity increased steeply after 2013 (n = 92). Satisfaction (n = 74) and learnability (n = 60) were the most frequently evaluated critical measures, while memorability (n = 13) was the least evaluated. The ratios of satisfaction, learnability, operability, and understandability measures were significantly related to the different stages of evaluation (P < 0.05). The methods used for usability evaluation were questionnaire (n = 68), interview (n = 36), concurrent thinking aloud (n = 25), performance metrics (n = 25), behavioral observation log (n = 14), screen recording (n = 3), eye tracking (n = 1), retrospective thinking aloud (n = 1), and feedback log (n = 1). Thirty-two studies developed their own evaluation tool to assess unique design features for elderly individuals.ConclusionIn the past five years, the number of studies in the field of usability evaluation of mHealth apps for the elderly has increased rapidly. The mHealth apps are often used as an auxiliary means of self-management to help the elderly manage their wellness and disease. According to the three stages of the mHealth app usability evaluation framework, the critical measures and evaluation methods are inconsistent. Future research should focus on selecting specific critical measures relevant to aging characteristics and adapting usability evaluation methods to elderly individuals by improving traditional tools, introducing automated evaluation tools and optimizing evaluation processes.

Project description:BackgroundDigital health technologies can be key to improving health outcomes, provided health care workers are adequately trained to use these technologies. There have been efforts to identify digital competencies for different health care worker groups; however, an overview of these efforts has yet to be consolidated and analyzed.ObjectiveThe review aims to identify and study existing digital health competency frameworks for health care workers and provide recommendations for future digital health training initiatives and framework development.MethodsA literature search was performed to collate digital health competency frameworks published from 2000. A total of 6 databases including gray literature sources such as OpenGrey, ResearchGate, Google Scholar, Google, and websites of relevant associations were searched in November 2019. Screening and data extraction were performed in parallel by the reviewers. The included evidence is narratively described in terms of characteristics, evolution, and structural composition of frameworks. A thematic analysis was also performed to identify common themes across the included frameworks.ResultsIn total, 30 frameworks were included in this review, a majority of which aimed at nurses, originated from high-income countries, were published since 2016, and were developed via literature reviews, followed by expert consultations. The thematic analysis uncovered 28 digital health competency domains across the included frameworks. The most prevalent domains pertained to basic information technology literacy, health information management, digital communication, ethical, legal, or regulatory requirements, and data privacy and security. The Health Information Technology Competencies framework was found to be the most comprehensive framework, as it presented 21 out of the 28 identified domains, had the highest number of competencies, and targeted a wide variety of health care workers.ConclusionsDigital health training initiatives should focus on competencies relevant to a particular health care worker group, role, level of seniority, and setting. The findings from this review can inform and guide digital health training initiatives. The most prevalent competency domains identified represent essential interprofessional competencies to be incorporated into health care workers' training. Digital health frameworks should be regularly updated with novel digital health technologies, be applicable to low- and middle-income countries, and include overlooked health care worker groups such as allied health professionals.

Project description:BackgroundThe strategic importance of monitoring social determinants of health (SDH) and health equity and inequity has been a central focus in global discussions around the 2011 Rio Political Declaration on SDH and the Millennium Development Goals. This study is part of the World Health Organization (WHO) equity-oriented analysis of linkages between health and other sectors (EQuAL) project, which aims to define a framework for monitoring SDH and health equity.ObjectivesThis review provides a global summary and analysis of the domains and indicators that have been used in recent studies covering the SDH. These studies are considered here within the context of indicators proposed by the WHO EQuAL project. The objectives are as follows: to describe the range of international and national studies and the types of indicators most frequently used; report how they are used in causal explanation of the SDH; and identify key priorities and challenges reported in current research for national monitoring of the SDH.DesignWe conducted a scoping review of published SDH studies in the PubMed(®) database to obtain evidence of socio-economic indicators. We evaluated, selected, and extracted data from national scale studies published from 2004 to 2014. The research included papers published in English, Italian, French, Portuguese, and Spanish.ResultsThe final sample consisted of 96 articles. SDH monitoring is well reported in the scientific literature independent of the economic level of the country and magnitude of deprivation in population groups. The research methods were mostly quantitative and many papers used multilevel and multivariable statistical analyses and indexes to measure health inequalities and SDH. In addition to the usual economic indicators, a high number of socio-economic indicators were used. The indicators covered a broad range of social dimensions, which were given consideration within and across different social groups. Many indicators included in the WHO EQuAL framework were not common in the studies in this review due to their intersectoral and interdisciplinary nature.ConclusionsOur review illustrates that the attention to SDH monitoring has grown in terms of its importance and complexity within the scientific health literature. We identified a need to make indicators more wide-ranging in order to include a broader range of social conditions. The WHO EQuAL framework can provide intersectoral and interdisciplinary means of building a more comprehensive standardised approach to monitoring the SDH and improving equity in health.

Project description:BACKGROUND:The use of eHealth has increased tremendously in recent years. eHealth is generally considered to have a positive effect on health care quality and to be a promising alternative to face-to-face health care contacts. Surprisingly little is known about possible adverse effects of eHealth apps. OBJECTIVE:We conducted a scoping review on empirical research into adverse effects of eHealth apps that aim to deliver health care at a distance. We investigated whether adverse effects were reported and the nature and quality of research into these possible adverse effects. METHODS:For this scoping review, we followed the five steps of Arksey and O'Malley's scoping review methodology. We searched specifically for studies into eHealth apps that replaced or complemented the face-to-face contact between a health professional and a patient in the context of treatment, health monitoring, or supporting self-management. Studies were included when eHealth and adverse effects were mentioned in the title or abstract and when empirical data on adverse effects were provided. All health conditions, with the exception of mental health conditions, all ages, and all sample sizes were included. We examined the literature published between December 2012 and August 2017 in the following databases: PubMed, Cumulative Index to Nursing and Allied Health Literature (CINAHL), Web of Science, and the Cochrane Library. The methodological quality of the studies was assessed using the Critical Appraisal Skills Programme (CASP) checklists. RESULTS:Our search identified 79 papers that were potentially relevant; 11 studies met our inclusion criteria after screening. These studies differed in many ways and the majority were characterized by small research populations and low study quality. Adverse effects are rarely subject to systematic scientific research. So far, information on real adverse effects is mainly limited to incidental reporting or as a bycatch from qualitative pilot studies. Despite the shortage of solid research, we found some indications of possible negative impact on patient-centeredness and efficiency, such as less transparency in the relationship between health professionals and patients and time-consuming work routines. CONCLUSIONS:There is a lack of high-quality empirical research on adverse effects of eHealth apps that replace or complement face-to-face care. While the development of eHealth apps is ongoing, the knowledge with regard to possible adverse effects is limited. The available research often focuses on efficacy, added value, implementation issues, use, and satisfaction, whereas adverse effects are underexplored. A better understanding of possible adverse effects could be a starting point in improving the positive impact of eHealth-based health care delivery.

Project description:BackgroundSignificant efforts have been made to develop artificial intelligence (AI) solutions for health care improvement. Despite the enthusiasm, health care professionals still struggle to implement AI in their daily practice.ObjectiveThis paper aims to identify the implementation frameworks used to understand the application of AI in health care practice.MethodsA scoping review was conducted using the Cochrane, Evidence Based Medicine Reviews, Embase, MEDLINE, and PsycINFO databases to identify publications that reported frameworks, models, and theories concerning AI implementation in health care. This review focused on studies published in English and investigating AI implementation in health care since 2000. A total of 2541 unique publications were retrieved from the databases and screened on titles and abstracts by 2 independent reviewers. Selected articles were thematically analyzed against the Nilsen taxonomy of implementation frameworks, and the Greenhalgh framework for the nonadoption, abandonment, scale-up, spread, and sustainability (NASSS) of health care technologies.ResultsIn total, 7 articles met all eligibility criteria for inclusion in the review, and 2 articles included formal frameworks that directly addressed AI implementation, whereas the other articles provided limited descriptions of elements influencing implementation. Collectively, the 7 articles identified elements that aligned with all the NASSS domains, but no single article comprehensively considered the factors known to influence technology implementation. New domains were identified, including dependency on data input and existing processes, shared decision-making, the role of human oversight, and ethics of population impact and inequality, suggesting that existing frameworks do not fully consider the unique needs of AI implementation.ConclusionsThis literature review demonstrates that understanding how to implement AI in health care practice is still in its early stages of development. Our findings suggest that further research is needed to provide the knowledge necessary to develop implementation frameworks to guide the future implementation of AI in clinical practice and highlight the opportunity to draw on existing knowledge from the field of implementation science.

Project description:BackgroundRecent surveys have revealed many adults have basic or below basic health literacy, which is linked to medical errors, increased illness, and compromised public health. Health literacy as a concept is multi-faceted extending beyond the individual to include social structures and the context in which health information is being accessed. Delivering health information via mobile devices (mHealth) expands the amount of information available while presenting challenges to ensuring these materials are suitable for a variety of literacy needs. The aims of this study are to discover how health literacy is addressed and evaluated in mHealth app development.MethodsA scoping review of 5 peer-reviewed databases was conducted. Eligible articles were written in English, addressed general literacy or mHealth/digital/eHealth literacy, and collected literacy information in order to incorporate literacy into the design and/or modification of an app or collected literacy information to describe the population being studied. The "Health Literacy Online" (HLO) United States (U.S.) government guide was used as a framework.ResultsThirty-two articles were reviewed. Articles included health literacy recommendations for all HLO categories and some recommendations not aligned with these categories. Most articles addressed health literacy using specific HLO categories though none incorporated every HLO category. The most common categories addressed engagement and testing of mHealth content. Though several studies addressed health literacy through a formal assessment tool, most did not. Evaluation of health literacy in mHealth was end-user focused and did not extensively evaluate content for fit to a variety of individuals with limited health literacy.ConclusionsThe recommendations seen consistently in our results in conjunction with formal HLO categories can act as beginning steps towards development of a health literacy evaluation tool for mHealth apps themselves. It is clear efforts are being made to reduce barriers to using mHealth for those with literacy deficits, however, it was also clear that this space has room to be more pragmatic in evaluation of mHealth tools for literacy. End user engagement in design and testing is necessary in future mHealth literacy tool development.