Dataset Information

Medical Mobile App Classification Using the National Institute for Health and Care Excellence Evidence Standards Framework for Digital Health Technologies: Interrater Reliability Study.

ABSTRACT:

Background

Clinical governance of medical mobile apps is challenging, and there is currently no standard method for assessing the quality of such apps. In 2018, the National Institute for Health and Care Excellence (NICE) developed a framework for assessing the required level of evidence for digital health technologies (DHTs), as determined by their clinical function. The framework can potentially be used to assess mobile apps, which are a subset of DHTs. To be used reliably in this context, the framework must allow unambiguous classification of an app's clinical function.

Objective

The objective of this study was to determine whether mobile health apps could be reliably classified using the NICE evidence standards framework for DHTs.

Methods

We manually extracted app titles, screenshots, and content descriptions for all apps listed on the National Health Service (NHS) Apps Library website on July 12, 2019; none of the apps were downloaded. Using this information, 2 mobile health (mHealth) researchers independently classified each app to one of the 4 functional tiers (ie, 1, 2, 3a, and 3b) described in the NICE digital technologies evaluation framework. Coders also answered contextual questions from the framework to identify whether apps were deemed to be higher risk. Agreement between coders was assessed using Cohen ? statistic.

Results

In total, we assessed 76 apps from the NHS Apps Library. There was classification agreement for 42 apps. Of these, 0 apps were unanimously classified into Tier 1; 24, into Tier 2; 15, into Tier 3a; and 3, into Tier 3b. There was disagreement between coders in 34/76 cases (45%); interrater agreement was poor (Cohen ?=0.32, 95% CI 0.16-0.47). Further investigation of disagreements highlighted 5 main explanatory themes: apps that did not correspond to any tier, apps that corresponded to multiple tiers, ambiguous tier descriptions, ambiguous app descriptions, and coder error.

Conclusions

The current iteration of the NICE evidence standards framework for DHTs did not allow mHealth researchers to consistently and unambiguously classify digital health mobile apps listed on the NHS app library according to their functional tier.

SUBMITTER: Nwe K

PROVIDER: S-EPMC7305556 | biostudies-literature | 2020 Jun

REPOSITORIES: biostudies-literature

ACCESS DATA

Publications

Medical Mobile App Classification Using the National Institute for Health and Care Excellence Evidence Standards Framework for Digital Health Technologies: Interrater Reliability Study.

Nwe Khine K Larsen Mark Erik ME Nelissen Natalie N Wong David Chi-Wai DC

Journal of medical Internet research 20200605 6

<h4>Background</h4>Clinical governance of medical mobile apps is challenging, and there is currently no standard method for assessing the quality of such apps. In 2018, the National Institute for Health and Care Excellence (NICE) developed a framework for assessing the required level of evidence for digital health technologies (DHTs), as determined by their clinical function. The framework can potentially be used to assess mobile apps, which are a subset of DHTs. To be used reliably in this cont ...[more]

PMID: 32501271

Similar Datasets

Project description:BackgroundThere is a growing role of digital health technologies (DHTs) in the management of chronic health conditions, specifically type 2 diabetes. It is increasingly important that health technologies meet the evidence standards for health care settings. In 2019, the National Institute for Health and Care Excellence (NICE) published the NICE Evidence Standards Framework for DHTs. This provides guidance for evaluating the effectiveness and economic value of DHTs in health care settings in the United Kingdom.ObjectiveThe aim of this study is to assess whether scientific articles on DHTs for the self-management of type 2 diabetes mellitus report the evidence suggested for implementation in clinical practice, as described in the NICE Evidence Standards Framework for DHTs.MethodsWe performed a scoping review of published articles and searched 5 databases to identify systematic reviews and primary studies of mobile device-delivered DHTs that provide self-management support for adults with type 2 diabetes mellitus. The evidence reported within articles was assessed against standards described in the NICE framework.ResultsThe database search yielded 715 systematic reviews, of which, 45 were relevant and together included 59 eligible primary studies. Within these, there were 39 unique technologies. Using the NICE framework, 13 technologies met best practice standards, 3 met minimum standards only, and 23 technologies did not meet minimum standards.ConclusionsOn the assessment of peer-reviewed publications, over half of the identified DHTs did not appear to meet the minimum evidence standards recommended by the NICE framework. The most common reasons for studies of DHTs not meeting these evidence standards included the absence of a comparator group, no previous justification of sample size, no measurable improvement in condition-related outcomes, and a lack of statistical data analysis. This report provides information that will enable researchers and digital health developers to address these limitations when designing, delivering, and reporting digital health technology research in the future.

Project description:There is currently a lack of clear and accepted standards for the development (planning, requirement analysis and research, design and application testing) of apps for medical and healthcare use which poses different risks to developers, providers, patients and the public. The aim of this work is to provide an overview of the current standards, frameworks, best practices and guidelines for the development of digital health apps. This review is a critical 'stepping stone' for further work on producing appropriate standards that can help mitigate risks (eg, clinical, privacy and economic risks).A systematic review identifying criteria from applicable standards, guidelines, frameworks and best practices for the development of health apps. We will draw from standards for software for medical devices, clinical information systems and medicine because of their relatedness and hope to apply lessons learnt to apps. We will exclude other types of publications, and those published in languages other than English. We will search websites of relevant regulatory and professional organisations. For health apps, we will also search electronic research databases (eg, MEDLINE, Embase, SCOPUS, ProQuest Technology Collection and Engineering Index) because relevant publications may not be found on other websites. We will hand-search reference lists of included publications. The review will focus on international, USA, European and UK standards because these are the markets of primary interest to the majority of app developers currently. We will provide a narrative overview of findings and tabular summaries of extracted data. Also, we will examine the relationship between different standards and compare USA and European Union standards.No ethics approval is required. The review will be disseminated through peer-reviewed publications, conference presentations and inform efforts that aim to improve the quality of health apps through existing links with relevant organisations.

Project description:BackgroundThere is a huge number of health-related apps available, and the numbers are growing fast. However, many of them have been developed without any kind of quality control. In an attempt to contribute to the development of high-quality apps and enable existing apps to be assessed, several guides have been developed.ObjectiveThe main aim of this study was to study the interrater reliability of a new guide - the Mobile App Development and Assessment Guide (MAG) - and compare it with one of the most used guides in the field, the Mobile App Rating Scale (MARS). Moreover, we also focused on whether the interrater reliability of the measures is consistent across multiple types of apps and stakeholders.MethodsIn order to study the interrater reliability of the MAG and MARS, we evaluated the 4 most downloaded health apps for chronic health conditions in the medical category of IOS and Android devices (ie, App Store and Google Play). A group of 8 reviewers, representative of individuals that would be most knowledgeable and interested in the use and development of health-related apps and including different types of stakeholders such as clinical researchers, engineers, health care professionals, and end users as potential patients, independently evaluated the quality of the apps using the MAG and MARS. We calculated the Krippendorff alpha for every category in the 2 guides, for each type of reviewer and every app, separately and combined, to study the interrater reliability.ResultsOnly a few categories of the MAG and MARS demonstrated a high interrater reliability. Although the MAG was found to be superior, there was considerable variation in the scores between the different types of reviewers. The categories with the highest interrater reliability in MAG were "Security" (α=0.78) and "Privacy" (α=0.73). In addition, 2 other categories, "Usability" and "Safety," were very close to compliance (health care professionals: α=0.62 and 0.61, respectively). The total interrater reliability of the MAG (ie, for all categories) was 0.45, whereas the total interrater reliability of the MARS was 0.29.ConclusionsThis study shows that some categories of MAG have significant interrater reliability. Importantly, the data show that the MAG scores are better than the ones provided by the MARS, which is the most commonly used guide in the area. However, there is great variability in the responses, which seems to be associated with subjective interpretation by the reviewers.

Project description:BackgroundThe classic Marshmallow Test, where children were offered a choice between one small but immediate reward (eg, one marshmallow) or a larger reward (eg, two marshmallows) if they waited for a period of time, instigated a wealth of research on the relationships among impulsive responding, self-regulation, and clinical and life outcomes. Impulsivity is a hallmark feature of self-regulation failures that lead to poor health decisions and outcomes, making understanding and treating impulsivity one of the most important constructs to tackle in building a culture of health. Despite a large literature base, impulsivity measurement remains difficult due to the multidimensional nature of the construct and limited methods of assessment in daily life. Mobile devices and the rise of mobile health (mHealth) have changed our ability to assess and intervene with individuals remotely, providing an avenue for ambulatory diagnostic testing and interventions. Longitudinal studies with mobile devices can further help to understand impulsive behaviors and variation in state impulsivity in daily life.ObjectiveThe aim of this study was to develop and validate an impulsivity mHealth diagnostics and monitoring app called Digital Marshmallow Test (DMT) using both the Apple and Android platforms for widespread dissemination to researchers, clinicians, and the general public.MethodsThe DMT app was developed using Apple's ResearchKit (iOS) and Android's ResearchStack open source frameworks for developing health research study apps. The DMT app consists of three main modules: self-report, ecological momentary assessment, and active behavioral and cognitive tasks. We conducted a study with a 21-day assessment period (N=116 participants) to validate the novel measures of the DMT app.ResultsWe used a semantic differential scale to develop self-report trait and momentary state measures of impulsivity as part of the DMT app. We identified three state factors (inefficient, thrill seeking, and intentional) that correlated highly with established measures of impulsivity. We further leveraged momentary semantic differential questions to examine intraindividual variability, the effect of daily life, and the contextual effect of mood on state impulsivity and daily impulsive behaviors. Our results indicated validation of the self-report sematic differential and related results, and of the mobile behavioral tasks, including the Balloon Analogue Risk Task and Go-No-Go task, with relatively low validity of the mobile Delay Discounting task. We discuss the design implications of these results to mHealth research.ConclusionsThis study demonstrates the potential for assessing different facets of trait and state impulsivity during everyday life and in clinical settings using the DMT mobile app. The DMT app can be further used to enhance our understanding of the individual facets that underlie impulsive behaviors, as well as providing a promising avenue for digital interventions.Trial registrationClinicalTrials.gov NCT03006653; https://www.clinicaltrials.gov/ct2/show/NCT03006653.

Project description:BACKGROUND:Almost a quarter or more than a fifth of children in the United Kingdom are overweight or obese by the time they start school. The UK Department of Health and Social Care's national policy for combating childhood obesity has critical outcomes centered on sugar and caloric consumption reduction. Health Education England has developed two digital apps for families with children up to 15 years and for their associated health care professionals to provide a digital learning resource and tool aimed at encouraging healthy lifestyles to prevent obesity. OBJECTIVE:This feasibility study assesses the usability and acceptability of Health Education England's NoObesity app for undertaking activities to improve families' diet and physical activity. The purpose of the study is to evaluate the app's influence on self-efficacy and goal setting and to determine what can be learnt to improve its design for future studies, if there is evidence of adoption and sustainability. METHODS:The study population will include 20 to 40 families and their linked health care professionals. Considering issues related to digital access associated with socioeconomic status and the impact on information technology use, study recruitment will be regionally focused in a low socioeconomic status area. The study will last for 9 months (3-month intervention period and 6-month follow-up). The evaluations of feasibility, acceptability, and usability will be conducted using the following scales and theoretical frameworks: (1) system usability scale; (2) Reach Effectiveness Adoption Implementation Maintenance framework; (3) Bandura model of health promotion; and (4) Nonadoption, Abandonment, and Challenges to the Scale-up, Spread, and Suitability framework. App use will be captured and quantitatively analyzed for net use patterns (eg, number of screens viewed, number of logins, cumulative minutes using the app, number of plans made, and number of times goals met) and to triangulate qualitative feedback from study participants. RESULTS:This study was funded in March 2019 by Health Education England and received University of Oxford Medical Sciences Interdivisional Research Ethics Committee approval on January 31, 2020 (R62092/RE001). At manuscript submission, study recruitment is pending, and expected results will be published in 2021. CONCLUSIONS:This study will provide evidence on the NoObesity app's influence on self-efficacy and goal-setting and determine what can be learnt to improve its design for future studies, if there is evidence of adoption and sustainability. INTERNATIONAL REGISTERED REPORT IDENTIFIER (IRRID):PRR1-10.2196/18068.

Project description:The majority of breast cancer survivors do not meet recommendations in terms of diet and physical activity. To address this problem, we developed a mobile health (mHealth) app for assessing and monitoring healthy lifestyles in breast cancer survivors, called the Energy Balance on Cancer (BENECA) mHealth system. The BENECA mHealth system is a novel and interactive mHealth app, which allows breast cancer survivors to engage themselves in their energy balance monitoring. BENECA was designed to facilitate adherence to healthy lifestyles in an easy and intuitive way.The objective of the study was to assess the concurrent validity and test-retest reliability between the BENECA mHealth system and the gold standard assessment methods for diet and physical activity.A reliability study was conducted with 20 breast cancer survivors. In the study, tri-axial accelerometers (ActiGraphGT3X+) were used as gold standard for 8 consecutive days, in addition to 2, 24-hour dietary recalls, 4 dietary records, and sociodemographic questionnaires. Two-way random effect intraclass correlation coefficients, a linear regression-analysis, and a Passing-Bablok regression were calculated.The reliability estimates were very high for all variables (alpha?.90). The lowest reliability was found in fruit and vegetable intakes (alpha=.94). The reliability between the accelerometer and the dietary assessment instruments against the BENECA system was very high (intraclass correlation coefficient=.90). We found a mean match rate of 93.51% between instruments and a mean phantom rate of 3.35%. The Passing-Bablok regression analysis did not show considerable bias in fat percentage, portions of fruits and vegetables, or minutes of moderate to vigorous physical activity.The BENECA mHealth app could be a new tool to measure energy balance in breast cancer survivors in a reliable and simple way. Our results support the use of this technology to not only to encourage changes in breast cancer survivors' lifestyles, but also to remotely monitor energy balance.ClinicalTrials.gov NCT02817724; https://clinicaltrials.gov/ct2/show/NCT02817724 (Archived by WebCite at http://www.webcitation.org/6xVY1buCc).

Dataset Information

Medical Mobile App Classification Using the National Institute for Health and Care Excellence Evidence Standards Framework for Digital Health Technologies: Interrater Reliability Study.

Background

Objective

Methods

Results

Conclusions

Publications

Medical Mobile App Classification Using the National Institute for Health and Care Excellence Evidence Standards Framework for Digital Health Technologies: Interrater Reliability Study.

Similar Datasets

OmicsDI is part of the ELIXIR infrastructure