Dataset Information

Tools for Evaluating the Content, Efficacy, and Usability of Mobile Health Apps According to the Consensus-Based Standards for the Selection of Health Measurement Instruments: Systematic Review.

ABSTRACT:

Background

There are several mobile health (mHealth) apps in mobile app stores. These apps enter the business-to-customer market with limited controls. Both, apps that users use autonomously and those designed to be recommended by practitioners require an end-user validation to minimize the risk of using apps that are ineffective or harmful. Prior studies have reviewed the most relevant aspects in a tool designed for assessing mHealth app quality, and different options have been developed for this purpose. However, the psychometric properties of the mHealth quality measurement tools, that is, the validity and reliability of the tools for their purpose, also need to be studied. The Consensus-based Standards for the Selection of Health Measurement Instruments (COSMIN) initiative has developed tools for selecting the most suitable measurement instrument for health outcomes, and one of the main fields of study was their psychometric properties.

Objective

This study aims to address and psychometrically analyze, following the COSMIN guideline, the quality of the tools that are used to measure the quality of mHealth apps.

Methods

From February 1, 2019, to December 31, 2019, 2 reviewers searched PubMed and Embase databases, identifying mHealth app quality measurement tools and all the validation studies associated with each of them. For inclusion, the studies had to be meant to validate a tool designed to assess mHealth apps. Studies that used these tools for the assessment of mHealth apps but did not include any psychometric validation were excluded. The measurement tools were analyzed according to the 10 psychometric properties described in the COSMIN guideline. The dimensions and items analyzed in each tool were also analyzed.

Results

The initial search showed 3372 articles. Only 10 finally met the inclusion criteria and were chosen for analysis in this review, analyzing 8 measurement tools. Of these tools, 4 validated ≥5 psychometric properties defined in the COSMIN guideline. Although some of the tools only measure the usability dimension, other tools provide information such as engagement, esthetics, or functionality. Furthermore, 2 measurement tools, Mobile App Rating Scale and mHealth Apps Usability Questionnaire, have a user version, as well as a professional version.

Conclusions

The Health Information Technology Usability Evaluation Scale and the Measurement Scales for Perceived Usefulness and Perceived Ease of Use were the most validated tools, but they were very focused on usability. The Mobile App Rating Scale showed a moderate number of validated psychometric properties, measures a significant number of quality dimensions, and has been validated in a large number of mHealth apps, and its use is widespread. It is suggested that the continuation of the validation of this tool in other psychometric properties could provide an appropriate option for evaluating the quality of mHealth apps.

SUBMITTER: Muro-Culebras A

PROVIDER: S-EPMC8686474 | biostudies-literature |

REPOSITORIES: biostudies-literature

ACCESS DATA

Similar Datasets

Project description:BackgroundMobile devices can provide extendable learning environments in higher education and motivate students to engage in adaptive and collaborative learning. Developers must design mobile apps that are practical, effective, and easy to use, and usability testing is essential for understanding how mobile apps meet users' needs. No previous reviews have investigated the usability of mobile apps developed for health care education.ObjectiveThe aim of this scoping review is to identify usability methods and attributes in usability studies of mobile apps for health care education.MethodsA comprehensive search was carried out in 10 databases, reference lists, and gray literature. Studies were included if they dealt with health care students and usability of mobile apps for learning. Frequencies and percentages were used to present the nominal data, together with tables and graphical illustrations. Examples include a figure of the study selection process, an illustration of the frequency of inquiry usability evaluation and data collection methods, and an overview of the distribution of the identified usability attributes. We followed the Arksey and O'Malley framework for scoping reviews.ResultsOur scoping review collated 88 articles involving 98 studies, mainly related to medical and nursing students. The studies were conducted from 22 countries and were published between 2008 and 2021. Field testing was the main usability experiment used, and the usability evaluation methods were either inquiry-based or based on user testing. Inquiry methods were predominantly used: 1-group design (46/98, 47%), control group design (12/98, 12%), randomized controlled trials (12/98, 12%), mixed methods (12/98, 12%), and qualitative methods (11/98, 11%). User testing methods applied were all think aloud (5/98, 5%). A total of 17 usability attributes were identified; of these, satisfaction, usefulness, ease of use, learning performance, and learnability were reported most frequently. The most frequently used data collection method was questionnaires (83/98, 85%), but only 19% (19/98) of studies used a psychometrically tested usability questionnaire. Other data collection methods included focus group interviews, knowledge and task performance testing, and user data collected from apps, interviews, written qualitative reflections, and observations. Most of the included studies used more than one data collection method.ConclusionsExperimental designs were the most commonly used methods for evaluating usability, and most studies used field testing. Questionnaires were frequently used for data collection, although few studies used psychometrically tested questionnaires. The usability attributes identified most often were satisfaction, usefulness, and ease of use. The results indicate that combining different usability evaluation methods, incorporating both subjective and objective usability measures, and specifying which usability attributes to test seem advantageous. The results can support the planning and conduct of future usability studies for the advancement of mobile learning apps in health care education.International registered report identifier (irrid)RR2-10.2196/19072.

Project description:BackgroundIn recent years, the considerable increase in the number of mobile health (mHealth) apps has made health care more accessible and affordable for all. However, the exponential growth in mHealth solutions has occurred with almost no control or regulation of any kind. Despite some recent initiatives, there is still no specific regulation procedure, accreditation system, or standards to help the development of the apps, mitigate risks, or guarantee quality.ObjectiveThe main aim of this study was to propose a set of criteria for mHealth-related apps on the basis of what is available from published studies, guidelines, and standards in the various areas that are related to health app development.MethodsWe used three sources of information to identify the most important criteria. First, we conducted a systematic review of all the studies published on pain-related apps. Second, we searched for health app recommendations on the websites of professional organizations. Third, we looked for standards governing the development of software for medical devices on the specialized websites of regulatory organizations. Then, we grouped and subsumed the criteria we had identified on the basis of their shared characteristics. Finally, the comprehensibility and perceived importance of the resulting criteria were evaluated for face validity with a group of 18 stakeholders.ResultsWe identified a total of 503 criteria from all sources, which, after close analysis, were grouped into eight different categories, including 36 important criteria for health apps. The resulting categories were usability, privacy, security, appropriateness and suitability, transparency and content, safety, technical support and updates, and technology. The results of the preliminary analysis showed that the criteria were mostly understood by the group of stakeholders. In addition, they perceived all of them as important.ConclusionsThis set of criteria can help health care providers, developers, patients, and other stakeholders to guide the development of mHealth-related apps and, potentially, to measure the quality of an mHealth app.

Project description:Social connection is important for long-term care (LTC) residents' quality of life and care. However, there is a lack of consensus on how to measure it and this limits ability to find what improves and impairs social connection in LTC homes. We therefore aimed to systematically review and evaluate the measurement properties of existing measures of social connection for LTC residents, to identify which, if any, measures can be recommended. We searched eight electronic databases from inception to April 2022 for studies which reported on psychometric properties of a measure of any aspect(s) of social connection (including social networks, interaction, engagement, support, isolation, connectedness, and loneliness) for LTC residents. We used COnsensus-based Standards for the selection of health Measurement INstruments (COSMIN) guidelines to evaluate the measurement properties reported for each identified measure and make recommendations. We identified 62 studies reporting on 38 measures; 21 measured quality of life, well-being or life satisfaction and included a social connection subscale or standalone items and 17 measures specifically targeted social connection. We found there was little high-quality evidence on psychometric properties such as sufficient content validity (n = 0), structural validity (n = 3), internal consistency (n = 3), reliability (n = 1), measurement error (n = 0), construct validity (n = 4), criterion validity (n = 0) and responsiveness (n = 0). No measures demonstrated satisfactory psychometric properties on all these aspects, so none could be recommended for use. Thirty-four measures have the potential to be recommended but require further research to assess their quality and the remaining four are not recommended for use. Our review therefore found that no existing measures have sufficient evidence to be recommended for assessment of social connection in residents of LTC homes. Further validation and reliability studies of existing instruments or the development of new measures are needed to enable accurate measurement of social connection in LTC residents for future observational and interventional studies.HighlightsSocial connection is fundamental to person-centered care in long-term care homes.There is insufficient evidence for the reliability and validity of existing measures.No current measures can be recommended for use based on existing evidence.A reliable and valid measure of social connection is needed for future research.

Project description:BACKGROUND:E-learning technologies, including mobile apps, are used to a large extent in health care education. Mobile apps can provide extendable learning environments and motivate students for adaptive and collaborative learning outside the classroom context. Developers should design practical, effective, and easy-to-use mobile apps. Usability testing is an important part of app development in order to understand if apps meet the needs of users. OBJECTIVE:The aim of this study is to perform a scoping review of usability methods and attributes reported in usability studies of mobile apps for health care education. METHODS:The scoping review is guided by the methodological framework developed by Arksey & O'Malley and further developed by Levac et al and Kahlil et al. The stages we will follow are as follows: (1) identifying the research question; (2) identifying relevant studies; (3) selecting studies; (4) charting the data; and (5) summarizing and reporting the results. We have developed two research questions to meet the aim of the study, which are as follows: (1) What usability methods are used to evaluate the usability of mobile apps for health care education? and (2) What usability attributes are reported in the usability studies of mobile apps for health care education? We will apply a comprehensive search of the literature, including 10 databases, a reference search, and a search for grey literature. Two review authors will independently screen articles for eligibility. RESULTS:The initial electronic database searches were completed in March 2019. The literature search identified 14,297 unique references. Following title and abstract screening, the full texts of 369 records were obtained. The scoping review is expected to be completed in spring 2021. CONCLUSIONS:We expect the overview of usability methods and attributes reported in usability studies of mobile apps for health care education to contribute to the knowledge base for researchers and developers. It will give an overview of the research field and provide researchers and developers with relevant and important information on the usability research area, including highlighting possible research gaps. INTERNATIONAL REGISTERED REPORT IDENTIFIER (IRRID):DERR1-10.2196/19072.

Project description:ImportanceMaternal depression is frequently reported in the postpartum period, with an estimated prevalence of approximately 15% during the first postpartum year. Despite the high prevalence of postpartum depression, there is no consensus regarding which patient-reported outcome measure (PROM) should be used to screen for this complex, multidimensional construct.ObjectiveTo evaluate psychometric measurement properties of existing PROMs of maternal postpartum depression using the Consensus-Based Standards for the Selection of Health Measurement Instruments (COSMIN) guideline and identify the best available patient-reported screening measure.Evidence reviewThis systematic review followed the Preferred Reporting Items for Systematic Reviews and Meta-analyses (PRISMA) reporting guideline. PubMed, CINAHL, Embase, and Web of Science were searched on July 1, 2019, for validated PROMs of postpartum depression, and an additional search including a hand search of references from eligible studies was conducted in June 2021. Included studies evaluated 1 or more psychometric measurement properties of the identified PROMs. A risk-of-bias assessment was performed to evaluate methods of each included study. Psychometric measurement properties of each PROM were rated according to COSMIN criteria. A modified Grading of Recommendations Assessment, Development, and Evaluation approach was used to assess the level of evidence supporting each rating, and a recommendation class (A, recommended for use; B, further research required; or C, not recommended) was given based on the overall quality of each included PROM.FindingsAmong 10 264 postpartum recovery studies, 27 PROMs were identified. Ten PROMs (37.0%) met the inclusion criteria and were used in 43 studies (0.4%) involving 22 095 postpartum women. At least 1 psychometric measurement property was assessed for each of the 10 validated PROMs identified. Content validity was sufficient in all PROMs. The Edinburgh Postnatal Depression Scale (EPDS) demonstrated adequate content validity and a moderate level of evidence for sufficient internal consistency (with sufficient structural validity), resulting in a recommendation of class A. The other 9 PROMs evaluated received a recommendation of class B.Conclusions and relevanceThe findings of this systematic review suggest that the EPDS is the best available patient-reported screening measure of maternal postpartum depression. Future studies should focus on evaluating the cross-cultural validity, reliability, and measurement error of the EPDS to improve understanding of its psychometric properties and utility.

Project description:Patient-reported outcome measures (PROMs) are important tools used to understand patient-focused outcomes from care. Various PROMs have been developed for patients with bladder cancer (BC), although the disease's heterogeneity makes selection difficult. Accurate measurement of health-related quality of life (HRQL) can only be achieved if the PROM chosen is 'fit for purpose' (i.e. psychometrically sound). Systematic reviews of psychometric properties are useful for selecting the best PROM for a specific purpose. The COnsensus-based Standards for the selection of health Measurement INstruments (COSMIN) developed a checklist to improve the selection of health measurement instruments as part of a review process. Our aims were to undertake a systematic review, using the COSMIN criteria, to assess the quality of studies that report the psychometric properties of PROMs used with people with BC and determine the psychometric quality of these PROMs. An electronic search of seven databases including PubMed, MEDLINE and EMBASE (PROSPERO reference CRD42016051974) was undertaken to identify English language publications, published between January 1990 and September 2017 that evaluated psychometric properties of PROMs used in BC research. Two researchers independently screened abstracts and selected full-text papers. Studies were rated on methodological quality using the COSMIN checklist. Overall, 4663 records were screened and 23 studies, reporting outcomes in 3568 patients, were evaluated using the COSMIN checklist. Most PROMs had limited information reported about their psychometric properties. Studies reporting on the Bladder Cancer Index (BCI) and Functional Assessment of Cancer Therapy Vanderbilt Cystectomy Index (FACT-VCI) provided the most detail and these PROMs could be evaluated on the most COSMIN properties. Based on the available evidence, no existing PROM stands out as the most appropriate to measure HRQL in BC populations. This is due to two factors; (i) the heterogeneity of BC and its treatments (ii) no PROM was evaluated on all COSMIN measurement properties due to a lack of validation studies. We suggest future evaluation of generic, cancer generic and BC-specific PROMs to better understand their application with BC populations and propose strategies to help clinicians and researchers.

Project description:A growing number of mobile health (mHealth) technology-based apps are being developed for personal lifestyle and medical health care support, of which several apps are related to pregnancy. Evidence on usability and effectiveness is limited but crucial for successful implementation.This study aimed to evaluate the usability, that is, feasibility and acceptability, as well as effectiveness of mHealth lifestyle and medical apps to support health care during pregnancy in high-income countries. Feasibility was defined as the actual use, interest, intention, and continued use; perceived suitability; and ability of users to carry out the activities of the app. Acceptability was assessed by user satisfaction, appreciation, and the recommendation of the app to others.We performed a systematic review searching the following electronic databases for studies on mHealth technology-based apps in maternal health care in developed countries: EMBASE, MEDLINE Epub (Ovid), Cochrane Library, Web of Science, and Google Scholar. All included studies were scored on quality, using the ErasmusAGE Quality Score or the consolidated criteria for reporting qualitative research. Main outcome measures were usability and effectiveness of mHealth lifestyle and medical health care support apps related to pregnancy. All studies were screened by 2 reviewers individually, and the guidelines of the Preferred Reporting Items for Systematic Reviews and Meta-Analyses statement were followed.Our search identified 4204 titles and abstracts, of which 2487 original studies remained after removing duplicates. We performed full-text screening of 217 studies, of which 29 were included in our study. In total, 19 out of 29 studies reported on mHealth apps to adopt healthy lifestyles and 10 out of 29 studies to support medical care. The lifestyle apps evaluated in 19 studies reported on usability and effectiveness: 10 studies reported positive on acceptability, and 14 studies reported on feasibility with positive results except one study. In total, 4 out of 19 studies evaluating effectiveness showed significant results on weight gain restriction during pregnancy, intake of vegetables and fruits, and smoking cessation. The 10 studies on medical mHealth apps involved asthma care, diabetic treatment, and encouraging vaccination. Only one study on diabetic treatment reported on acceptability with a positive user satisfaction. In total, 9 out of 10 studies reported on effectiveness. Moreover, the power of most studies was inadequate to show significant effects.Most studies on mHealth apps to support lifestyle and medical care for high-income countries reveal the usability of these apps to reduce gestational weight gain, increase intakes of vegetables and fruit, to quit smoking cessation, and to support health care for prevention of asthma and infections during pregnancy. In general, the evidence on effectiveness of these apps is limited and needs further investigation before implementation in medical health care.

Project description:BackgroundMobile health (mHealth) apps offer great opportunities to deliver large-scale, cost-efficient digital solutions for implementing lifestyle changes. Furthermore, many mHealth apps act as medical devices. Yet, there is little research on how to assess user satisfaction with an mHealth solution.ObjectiveThis study presents the development of the mHealth Satisfaction Questionnaire and evaluates its measurement properties.MethodsRespondents who took part in the Health Integrator Study and were randomized to use the Health Integrator smartphone app for lifestyle changes (n=112), with and without additional telephone coaching, rated their satisfaction with the app using the new 14-item mHealth Satisfaction Questionnaire. The ratings were given on a 5-point Likert scale and measurement properties were evaluated using Rasch measurement theory (RMT).ResultsOptimal scoring was reached when response options 2, 3, and 4 were collapsed, giving three response categories. After omitting two items that did not fit into the scale, fit residuals were within, or close to, the recommended range of ±2.5. There was no differential item functioning between intervention group, age group, or sex. The Person Separation Index was 0.79, indicating that the scale's ability to discriminate correctly between person leniency was acceptable for group comparisons but not for individual evaluations. The scale did not meet the criterion of unidimensionality; 16.1% (18/112) of the respondents were outside the desired range of -1.96 to 1.96. In addition, several items showed local dependency and three underlying dimensions emerged: negative experiences, positive experiences, and lifestyle consequences of using the mHealth solution.ConclusionsIn times where mHealth apps and digital solutions are given more attention, the mHealth Satisfaction Questionnaire provides a new possibility to measure user satisfaction to ensure usability and improve development of new apps. Our study is one of only a few cases where RMT has been used to evaluate the usability of such an instrument. There is, though, a need for further development of the mHealth Satisfaction Questionnaire, including the addition of more items and consideration of further response options. The mHealth Satisfaction Questionnaire should also be evaluated in a larger sample and with other mHealth apps and in other contexts.Trial registrationClinicalTrials.gov NCT03579342; http://clinicaltrials.gov/ct2/show/NCT03579342.