Dataset Information

An Assessment Framework for e-Mental Health Apps in Canada: Results of a Modified Delphi Process.

ABSTRACT:

Background

The number of e-mental health apps is increasing rapidly. Studies have shown that the use of some apps is beneficial, whereas others are ineffective or do not meet users' privacy expectations. Individuals and organizations that curate, recommend, host, use, or pay for apps have an interest in categorizing apps according to the consensus criteria of usability and effectiveness. Others have previously published recommendations for assessing health-related apps; however, the extent to which these recommendations can be generalized across different population groups (eg, culture, gender, and language) remains unclear. This study describes an attempt by Canadian stakeholders to develop an e-mental health assessment framework that responds to the unique needs of people living in Canada in an evidence-based manner.

Objective

The objective of our study was to achieve consensus from a broad group of Canadian stakeholders on guiding principles and criteria for a framework to assess e-mental health apps in Canada.

Methods

We developed an initial set of guiding principles and criteria from a rapid review and environmental scan of pre-existing app assessment frameworks. The initial list was refined through a two-round modified Delphi process. Participants (N=25) included app developers and users, health care providers, mental health advocates, people with lived experience of a mental health problem or mental illness, policy makers, and researchers. Consensus on each guideline or criterion was defined a priori as at least 70% agreement. The first round of voting was conducted electronically. Prior to Round 2 voting, in-person presentations from experts and a persona empathy mapping process were used to explore the perspectives of diverse stakeholders.

Results

Of all respondents, 68% (17/25) in Round 1 and 100% (13/13) in Round 2 agreed that a framework for evaluating health apps is needed to help Canadian consumers identify high-quality apps. Consensus was reached on 9 guiding principles: evidence based, gender responsive, culturally appropriate, user centered, risk based, internationally aligned, enabling innovation, transparent and fair, and based on ethical norms. In addition, 15 informative and evaluative criteria were defined to assess the effectiveness, functionality, clinical applicability, interoperability, usability, transparency regarding security and privacy, security or privacy standards, supported platforms, targeted users, developers' transparency, funding transparency, price, user desirability, user inclusion, and meaningful inclusion of a diverse range of communities.

Conclusions

Canadian mental health stakeholders reached the consensus on a framework of 9 guiding principles and 15 criteria important in assessing e-mental health apps. What differentiates the Canadian framework from other scales is explicit attention to user inclusion at all stages of the development, gender responsiveness, and cultural appropriateness. Furthermore, an empathy mapping process markedly influenced the development of the framework. This framework may be used to inform future mental health policies and programs.

SUBMITTER: Zelmer J

PROVIDER: S-EPMC6056739 | biostudies-literature | 2018 Jul

REPOSITORIES: biostudies-literature

ACCESS DATA

Publications

An Assessment Framework for e-Mental Health Apps in Canada: Results of a Modified Delphi Process.

Zelmer Jennifer J van Hoof Krystle K Notarianni MaryAnn M van Mierlo Trevor T Schellenberg Megan M Tannenbaum Cara C

JMIR mHealth and uHealth 20180709 7

<h4>Background</h4>The number of e-mental health apps is increasing rapidly. Studies have shown that the use of some apps is beneficial, whereas others are ineffective or do not meet users' privacy expectations. Individuals and organizations that curate, recommend, host, use, or pay for apps have an interest in categorizing apps according to the consensus criteria of usability and effectiveness. Others have previously published recommendations for assessing health-related apps; however, the exte ...[more]

PMID: 29986846

Similar Datasets

Project description:BackgroundThe use of apps for weight management has increased over recent years; however, there is a lack of evidence regarding the efficacy and safety of these apps. The EVALAPPS project will develop and validate an assessment instrument to specifically assess the safety and efficacy of weight management apps.ObjectiveThe aim of this study was to reach a consensus among stakeholders on a comprehensive set of criteria to guide development of the EVALAPPS assessment instrument. A modified Delphi process was used in order to verify the robustness of the criteria that had been identified through a literature review and to prioritize a set of the identified criteria.MethodsStakeholders (n=31) were invited to participate in a 2-round Delphi process with 114 initial criteria that had been identified from the literature. In round 1, participants rated criteria according to relevance on a scale from 0 ("I suggest this criterion is excluded") to 5 ("This criterion is extremely relevant"). A criterion was accepted if the median rating was 4 or higher and if the relative intraquartile range was equal to 0.67 or lower. In round 2, participants were asked about criteria that had been discarded in round 1. A prioritization strategy was used to identify crucial criteria according to (1) the importance attributed by participants (criteria with a mean rating of 4.00 or higher), (2) the level of consensus (criteria with a score of 4 or 5 by at least 80% of the participants).ResultsThe response rate was 83.9% (26/31) in round 1 and 90.3% (28/31) in round 2. A total of 107 out of 114 criteria (93.9%) were accepted by consensus-105 criteria in round 1 and 2 criteria in round 2. After prioritization, 53 criteria were deemed crucial. These related mainly to the dimensions of security and privacy (13/53, 24.5%) and usability (9/53, 17.0%), followed by activity data (5/53, 9.4%), clinical effectiveness (5/53, 9.4%), and reliability (5/53, 9.4%).ConclusionsResults confirmed the robustness of the criteria that were identified, with those relating to security and privacy being deemed most relevant by stakeholders. Additionally, a specific set of criteria based on health indicators (activity data, physical state data, and personal data) was also prioritized.

Project description:ObjectivesTo develop a consensus statement to provide advice on designing, implementing and evaluating crowdsourcing challenge contests in public health and medical contexts.DesignModified Delphi using three rounds of survey questionnaires and one consensus workshop.SettingUganda for face-to-face consensus activities, global for online survey questionnaires.ParticipantsA multidisciplinary expert panel was convened at a consensus-development conference in Uganda and included 21 researchers with experience leading challenge contests, five public health sector workers, and nine Ugandan end users. An online survey was sent to 140 corresponding authors of previously published articles that had used crowdsourcing methods.ResultsA subgroup of expert panel members developed the initial statement and survey. We received responses from 120 (85.7%) survey participants, which were presented at an in-person workshop of all 21 panel members. Panelists discussed each of the sections, revised the statement, and participated in a second round of the survey questionnaire. Based on this second survey round, we held detailed discussions of each subsection with workshop participants and further revised the consensus statement. We then conducted the third round of the questionnaire among the 21 expert panelists and used the results to finalize the statement. This iterative process resulted in 23 final statement items, all with greater than 80% consensus. Statement items are organised into the seven stages of a challenge contest, including the following: considering the appropriateness, organising a community steering committee, promoting the contest, assessing contributions, recognising contributors, sharing ideas and evaluating the contest (COPARSE).ConclusionsThere is high agreement among crowdsourcing experts and stakeholders on the design and implementation of crowdsourcing challenge contests. The COPARSE consensus statement can be used to organise crowdsourcing challenge contests, improve the rigour and reproducibility of crowdsourcing research and enable large-scale collaboration.

Project description:PurposeLocal treatment for bone metastases is becoming increasingly complex. National guidelines traditionally focus only on radiation therapy (RT), leaving a gap in clinical decision support resources available to clinicians. The objective of this study was to reach expert consensus regarding multidisciplinary management of non-spine bone metastases, which would facilitate standardizing treatment within an academic-community partnership.Methods and materialsA multidisciplinary panel of physicians treating metastatic disease across the Memorial Sloan Kettering (MSK) Cancer Alliance, including community-based partner sites, was convened. Clinical questions rated of high importance in the management of non-spine bone metastases were identified via survey. A literature review was conducted, and panel physicians drafted initial recommendation statements. Consensus was gathered on recommendation statements through a modified Delphi process from a full panel of 17 physicians from radiation oncology, orthopaedic surgery, medical oncology, interventional radiology, and anesthesia pain. Consensus was defined a priori as 75% of respondents indicating "agree" or "strongly agree" with the consensus statement. Strength of Recommendation Taxonomy was employed to assign evidence strength for each statement.ResultsSeventeen clinical questions were identified, of which 11 (65%) were selected for the consensus process. Consensus was reached for 16 of 17 answer statements (94%), of which 12 were approved after Round 1 and additional 4 approved after Round 2 of the modified Delphi voting process. Topics included indications for referral to surgery or interventional radiology, radiation fractionation and appropriate use of stereotactic approaches, and the handling of systemic therapies during radiation. Evidence strength was most commonly C (n = 7), followed by B (n = 5) and A (n = 3).ConclusionsConsensus among a multidisciplinary panel of community and academic physicians treating non-spine bone metastases was feasible. Recommendations will assist clinicians and potentially provide measures to reduce variation across diverse practice settings. Findings highlight areas for further research such as pathologic fracture risk estimation, pre-operative radiation, and percutaneous ablation.

Project description:BackgroundSocial distancing policies work in different ways and at different levels. In addition, various forms of monitoring systems have been implemented in different countries. However, there is an almost complete lack of specific monitoring system in Republic of Korea to effectively monitor social distancing measures compliance and outcome. This study aims to develop a monitoring system for social distancing measures compliance and outcome in Korea to evaluate and improve the implemented policy.MethodsA draft monitoring system was developed after reviewing Korea's social distancing measures (central and local government briefings) and checking available data about social distancing behavior. The modified Delphi process was used to evaluate the draft of the social distancing monitoring system. In total, 27 experts participated in the evaluation. The round 1 evaluation includes (1) commenting on the composition of the monitoring fields (open response), (2) monitoring indicators for each monitoring field (10-point Likert scale), and (3) commenting on the source of data used to develop the monitoring system (open response). In the round 2 evaluation, 55 indicators, excepting open responses, were re-evaluated.ResultsThe response rate for the Delphi survey was 100% in both the first and second rounds. Of the 55 indicators, 1 indicator, which did not satisfy the quantitative criteria, was excluded. According to the experts' open response comments, 15 indicators were excluded, as these indicators overlapped with other indicators or had little relevance to social distancing. Instead, 23 new indicators were added. Finally, 62 indicators were included with 12 available data sources. The monitoring system domain was divided into 'social distancing measures state, social distancing measures compliance, social distancing outcome'.ConclusionsThis study is significant in that it is the first in Korea to develop a comprehensive monitoring system for checking if social distancing measures are being followed well, and is applicable to estimates utilizing data that are immediately available for each indicator. Furthermore, the developed monitoring system could be a reference for other countries that require the development of such systems to monitor social distancing.

Project description:ObjectiveGuidelines for a structured assessment in community paramedicine home visit programmes have not been established and evidence to inform their creation is lacking. We sought to investigate the relevance of assessment items to the practice of community paramedics according to a pre-established clarity-utility matrix.DesignWe designed a modified-Delphi study consisting of predetermined thresholds for achieving consensus, number of rounds of for scoring items, a defined meeting and discussion process, and a sample of participants that was purposefully representative.Setting and participantsWe established a panel of 26 community paramedics representing 20 municipal paramedic services in Ontario, Canada. The sample represented a majority of paramedic services within the province that were operating a community paramedicine home visit programme.MeasuresDrawing from a bank of standardised assessment items grouped according to domains aligned with the International Classification on Functioning, Disability, and Health taxonomy, 64 previously pilot-tested assessment items were scored according to their clarity (being free from ambiguity and easy to understand) and utility (being valued in care planning or case management activities). Assessment items covered a broad range of health, social and environmental domains. To conclude scoring rounds, assessment items that did not achieve consensus for relevance to assessment practices were discussed among participants with opportunities to modify assessment items for subsequent rounds of scoring.ResultsResulting from the first round of scoring, 54 assessment items were identified as being relevant to assessment practices and 3 assessment items were removed from subsequent rounds. The remaining 7 assessment items were modified, with some parts removed from the final items that achieved consensus in the final rounds of scoring.ConclusionA broadly representative panel of community paramedics identified consensus for 61 assessment items that could be included in a structured, multidomain, assessment instrument for guiding practice in community paramedicine home visit programmes.Trail registration numberISRCTN58273216.

Project description:BackgroundApproximately 800 million people, representing 11% of the world's population, are affected by mental health problems. The COVID-19 pandemic exacerbated problems and triggered a decline in well-being, with drastic increase in the incidence of conditions such as anxiety, depression, and stress. Approximately 20,000 mental health apps are listed in mobile app stores. However, no significant evaluation of mental health apps in French, spoken by approximately 300 million people, has been identified in the literature yet.ObjectiveThis study aims to review the mental health mobile apps currently available on the French Apple App Store and Google Play Store and to evaluate their quality using Mobile App Rating Scale-French (MARS-F).MethodsScreening of mental health apps was conducted from June 10, 2022, to June 17, 2022, on the French Apple App Store and Google Play Store. A shortlist of 12 apps was identified using the criteria of selection and assessed using MARS-F by 9 mental health professionals. Intraclass correlation was used to evaluate interrater agreement. Mean (SD) scores and their distributions for each section and item were calculated.ResultsThe highest scores for MARS-F quality were obtained by Soutien psy avec Mon Sherpa (mean 3.85, SD 0.48), Evoluno (mean 3.54, SD 0.72), and Teale (mean 3.53, SD 0.87). Mean engagement scores (section A) ranged from 2.33 (SD 0.69) for Reflexe reussite to 3.80 (SD 0.61) for Soutien psy avec Mon Sherpa. Mean aesthetics scores (section C) ranged from 2.52 (SD 0.62) for Mental Booster to 3.89 (SD 0.69) for Soutien psy avec Mon Sherpa. Mean information scores (section D) ranged from 2.00 (SD 0.75) for Mental Booster to 3.46 (SD 0.77) for Soutien psy avec Mon Sherpa. Mean Mobile App Rating Scale subjective quality (section E) score varied from 1.22 (SD 0.26) for VOS - journal de l'humeur to 2.69 (SD 0.84) for Soutien psy avec Mon Sherpa. Mean app specificity (section F) score varied from 1.56 (SD 0.97) for Mental Booster to 3.31 (SD 1.22) for Evoluno. For all the mental health apps studied, except Soutien psy avec Mon Sherpa (11/12, 92%), the subjective quality score was always lower than the app specificity score, which was always lower than the MARS-F quality score, and that was lower than the rating score from the iPhone Operating System or Android app stores.ConclusionsMental health professionals assessed that, despite the lack of scientific evidence, the mental health mobile apps available on the French Apple App Store and Google Play Store were of good quality. However, they are reluctant to use them in their professional practice. Additional investigations are needed to assess their compliance with recommendations and their long-term impact on users.

Dataset Information

An Assessment Framework for e-Mental Health Apps in Canada: Results of a Modified Delphi Process.

Background

Objective

Methods

Results

Conclusions

Publications

An Assessment Framework for e-Mental Health Apps in Canada: Results of a Modified Delphi Process.

Similar Datasets

OmicsDI is part of the ELIXIR infrastructure

Tweets