Dataset Information

Procedures of User-Centered Usability Assessment for Digital Solutions: Scoping Review of Reviews Reporting on Digital Solutions Relevant for Older Adults.

ABSTRACT:

Background

The assessment of usability is a complex process that involves several steps and procedures. It is important to standardize the evaluation and reporting of usability procedures across studies to guide researchers, facilitate comparisons across studies, and promote high-quality usability studies. The first step to standardizing is to have an overview of how usability study procedures are reported across the literature.

Objective

This scoping review of reviews aims to synthesize the procedures reported for the different steps of the process of conducting a user-centered usability assessment of digital solutions relevant for older adults and to identify potential gaps in the present reporting of procedures. The secondary aim is to identify any principles or frameworks guiding this assessment in view of a standardized approach.

Methods

This is a scoping review of reviews. A 5-stage scoping review methodology was used to identify and describe relevant literature published between 2009 and 2020 as follows: identify the research question, identify relevant studies, select studies for review, chart data from selected literature, and summarize and report results. The research was conducted on 5 electronic databases: PubMed, ACM Digital Library, IEEE, Scopus, and Web of Science. Reviews that met the inclusion criteria (reporting on user-centered usability evaluation procedures for any digital solution that could be relevant for older adults and were published in English) were identified, and data were extracted for further analysis regarding study evaluators, study participants, methods and techniques, tasks, and test environment.

Results

A total of 3958 articles were identified. After a detailed screening, 20 reviews matched the eligibility criteria. The characteristics of the study evaluators and participants and task procedures were only briefly and differently reported. The methods and techniques used for the assessment of usability are the topics that were most commonly and comprehensively reported in the reviews, whereas the test environment was seldom and poorly characterized.

Conclusions

A lack of a detailed description of several steps of the process of assessing usability and no evidence on good practices of performing it suggests that there is a need for a consensus framework on the assessment of user-centered usability evaluation. Such a consensus would inform researchers and allow standardization of procedures, which are likely to result in improved study quality and reporting, increased sensitivity of the usability assessment, and improved comparability across studies and digital solutions. Our findings also highlight the need to investigate whether different ways of assessing usability are more sensitive than others. These findings need to be considered in light of review limitations.

SUBMITTER: Silva AG

PROVIDER: S-EPMC7840284 | biostudies-literature | 2021 Jan

REPOSITORIES: biostudies-literature

ACCESS DATA

Publications

Procedures of User-Centered Usability Assessment for Digital Solutions: Scoping Review of Reviews Reporting on Digital Solutions Relevant for Older Adults.

Silva Anabela G AG Caravau Hilma H Martins Ana A Almeida Ana Margarida Pisco AMP Silva Telmo T Ribeiro Óscar Ó Santinha Gonçalo G Rocha Nelson P NP

JMIR human factors 20210113 1

<h4>Background</h4>The assessment of usability is a complex process that involves several steps and procedures. It is important to standardize the evaluation and reporting of usability procedures across studies to guide researchers, facilitate comparisons across studies, and promote high-quality usability studies. The first step to standardizing is to have an overview of how usability study procedures are reported across the literature.<h4>Objective</h4>This scoping review of reviews aims to syn ...[more]

PMID: 33439128

Similar Datasets

Project description:BackgroundThe goal of user-centered design (UCD) is to understand the users' perspective and to use that knowledge to shape more effective solutions. The UCD approach provides insight into users' needs and requirements and thereby improves the design of the developed services. However, involving users in the development process does not guarantee that feedback from different subgroups of users will shape the development in ways that will make the solutions more useful for the entire target user population.ObjectiveThe aim of this study was to describe a protocol for systematic analysis and prioritization of feedback from user subgroups in the usability testing of a digital motivation support for fall-preventive physical activity (PA) interventions in seniors (aged 65 years and older). This protocol can help researchers and developers to systematically exploit feedback from relevant user subgroups in UCD.MethodsGender, PA level, and level of technology experience have been identified in the literature to influence users' experience and use of digital support systems for fall-preventive PA interventions in seniors. These 3 key user characteristics were dichotomized and used to define 8 (ie, 23) possible user subgroups. The presented method enables systematic tracking of the user subgroups' contributions in iterative development. The method comprises (1) compilation of difficulties and deficiencies in the digital applications identified in usability testing, (2) clustering of the identified difficulties and deficiencies, and (3) prioritization of deficiencies to be rectified. Tracking user subgroup representation in the user feedback ensures that the development process is prioritized according to the needs of different subgroups. Mainly qualitative data collection methods are used.ResultsA protocol was developed to ensure that feedback from users representing all possible variants of 3 selected key user characteristics (gender, PA level, and level of technology experience) is considered in the iterative usability testing of a digital support for seniors' PA. The method was applied in iterative usability testing of two digital applications during spring/summer 2018. Results from the study on the users' experiences and the iterative modification of the digital applications are expected to be published during 2021.ConclusionsMethods for systematic collection, analysis, and prioritization of feedback from user subgroups might be particularly important in heterogenous user groups (eg, seniors). This study can contribute to identifying and improving the understanding of potential differences between user subgroups of seniors in their use and experiences of digital support for fall-preventive PA interventions. This knowledge may be relevant for developing digital support systems that are appropriate, useful, and attractive to users and for enabling the design of digital support systems that target specific user subgroups (ie, tailoring of the support). The protocol needs to be further used and investigated in order to validate its potential value.International registered report identifier (irrid)RR1-10.2196/20061.

Project description:IntroductionOver 25% of Dutch young people are psychologically unhealthy. Individual and societal consequences that follow from having mental health complaints at this age are substantial. Young people need care which is often unavailable. ENgage YOung people earlY (ENYOY) is a moderated digital social therapy-platform that aims to help youngsters with emerging mental health complaints. Comprehensive research is being conducted into the effects and to optimize and implement the ENYOY-platform throughout the Netherlands. The aim of this study is to explore the usability and user experience of the ENYOY-platform.MethodsA user-centered mixed-method design was chosen. 26 young people aged 16-25 with emerging mental health complaints participated. Semi-structured interviews were conducted to explore usability, user-friendliness, impact, accessibility, inclusivity, and connection (Phase 1). Phase 2 assessed usability problems using the concurrent and retrospective Think Aloud-method. User experience and perceived helpfulness were assessed using a 10-point rating scale and semi-structured interviews (Phase 3). The Health Information Technology Usability Evaluation Scale (Health-ITUES; Phase 1) and System Usability Scale (SUS; Phase 2 and 3) were administered. Qualitative data was analyzed using thematic analysis. Task completion rate and time were tracked and usability problems were categorized using the Nielsen's rating scale (Phase 2).ResultsAdequate to high usability was found (Phase 1 Health-ITUES 4.0(0.34); Phase 2 SUS 69,5(13,70); Phase 3 SUS 71,6(5,63)). Findings from Phase 1 (N = 10) indicated that users viewed ENYOY as a user-friendly, safe, accessible, and inclusive initiative which helped them reduce their mental health complaints and improve quality of life. Phase 2 (N = 10) uncovered 18 usability problems of which 5 of major severity (e.g. troubles accessing the platform). Findings from Phase 3 (N = 6) suggested that users perceived the coaching calls the most helpful [9(0.71)] followed by the therapy content [6.25(1.41)]. Users liked the social networking aspect but rated it least helpful [6(2.1)] due to inactivity.ConclusionThe ENYOY-platform has been found to have adequate to high usability and positive user experiences were reported. All findings will be transferred to the developmental team to improve the platform. Other evaluation methods and paring these with quantitative outcomes could provide additional insight in future research.

Project description:BackgroundSensors and digital devices have revolutionized the measurement, collection, and storage of behavioral and physiological data, leading to the new term digital biomarkers.ObjectiveThis study aimed to investigate the scope of clinical evidence covered by systematic reviews (SRs) of randomized controlled trials involving digital biomarkers.MethodsThis scoping review was organized using the PRISMA-ScR (Preferred Reporting Items for Systematic Reviews and Meta-Analyses extension for Scoping Reviews) guidelines. With the search limited to English publications, full-text SRs of digital biomarkers included randomized controlled trials that involved a human population and reported changes in participants' health status. PubMed and the Cochrane Library were searched with time frames limited to 2019 and 2020. The World Health Organization's classification systems for diseases (International Classification of Diseases, Eleventh Revision), health interventions (International Classification of Health Interventions), and bodily functions (International Classification of Functioning, Disability, and Health [ICF]) were used to classify populations, interventions, and outcomes, respectively.ResultsA total of 31 SRs met the inclusion criteria. The majority of SRs studied patients with circulatory system diseases (19/31, 61%) and respiratory system diseases (9/31, 29%). Most of the prevalent interventions focused on physical activity behavior (16/31, 52%) and conversion of cardiac rhythm (4/31, 13%). Looking after one's health (physical activity; 15/31, 48%), walking (12/31, 39%), heart rhythm functions (8/31, 26%), and mortality (7/31, 23%) were the most commonly reported outcomes. In total, 16 physiological and behavioral data groups were identified using the ICF tool, such as looking after one's health (physical activity; 14/31, 45%), walking (11/31, 36%), heart rhythm (7/31, 23%), and weight maintenance functions (7/31, 23%). Various digital devices were also studied to collect these data in the included reviews, such as smart glasses, smartwatches, smart bracelets, smart shoes, and smart socks for measuring heart functions, gait pattern functions, and temperature. A substantial number (24/31, 77%) of digital biomarkers were used as interventions. Moreover, wearables (22/31, 71%) were the most common types of digital devices. Position sensors (21/31, 68%) and heart rate sensors and pulse rate sensors (12/31, 39%) were the most prevalent types of sensors used to acquire behavioral and physiological data in the SRs.ConclusionsIn recent years, the clinical evidence concerning digital biomarkers has been systematically reviewed in a wide range of study populations, interventions, digital devices, and sensor technologies, with the dominance of physical activity and cardiac monitors. We used the World Health Organization's ICF tool for classifying behavioral and physiological data, which seemed to be an applicable tool to categorize the broad scope of digital biomarkers identified in this review. To understand the clinical value of digital biomarkers, the strength and quality of the evidence on their health consequences need to be systematically evaluated.

Project description:BackgroundScoping reviews have emerged as a valuable method for synthesizing emerging evidence, providing a comprehensive contextual overview, and influencing policy and practice developments. The objective of this study is to provide an overview of scoping reviews conducted in Chinese academic institutions over the last decades.MethodWe conducted a comprehensive search of nine databases and six grey literature databases for scoping reviews conducted in Chinese academic institutions. The reporting quality of the included reviews was assessed using the Preferred Reporting Items for PRISMA-ScR checklist. We performed both quantitative and qualitative analyses, examining the conduct of the scoping reviews and exploring the breadth of research topics covered. We used Chi-squared and Wilcoxon rank-sum tests to compare methodological issues and reporting quality in English and Chinese-language reviews.ResultsA total of 392 reviews published between 2013 and 2022 were included, 238 English-reported reviews and 154 Chinese-reported reviews, respectively. The primary purposes of these reviews were to map and summarize the evidence, with a particular focus on health and nursing topics. 98.7% of reviews explicitly used the term "scoping review", and the Arksey and O'Malley framework was the most frequently cited framework. Thirty-five English-reported scoping reviews provided a protocol for scoping review. PubMed was the most common source in English-reported reviews and CNKI in Chinese-reported reviews. Reviews published in English were more likely to search the grey literature (P = 0.005), consult information specialists (P < 0.001) and conduct an updated search (P = 0.012) than those in Chinese. Reviews published in English had a significantly high score compared to those published in Chinese (16 vs. 14; P < 0.001). The reporting rates in English-reported reviews were higher than those in Chinese reviews for seven items, but lower for structured summary (P < 0.001), eligibility criteria (P < 0.001), data charting process (P = 0.009) and data items (P = 0.015).ConclusionThere has been a significant increase in the number of scoping reviews conducted in Chinese academic institutions each year since 2020. While the research topics covered are diverse, the overall reporting quality of these reviews is need to be improved. And there is a need for greater standardization in the conduct of scoping reviews in Chinese academic institutions.

Project description:BackgroundWhile many digital mental health interventions (DMHIs) have been found to be efficacious, patient engagement with DMHIs has increasingly emerged as a concern for implementation in real-world clinical settings. To address engagement, we must first understand what standard engagement levels are in the context of randomized controlled trials (RCTs) and how these compare with other treatments.ObjectiveThis scoping review aims to examine the state of reporting on intervention engagement in RCTs of mobile app-based interventions intended to treat symptoms of depression. We sought to identify what engagement metrics are and are not routinely reported as well as what the metrics that are reported reflect about standard engagement levels.MethodsWe conducted a systematic search of 7 databases to identify studies meeting our eligibility criteria, namely, RCTs that evaluated use of a mobile app-based intervention in adults, for which depressive symptoms were a primary outcome of interest. We then extracted 2 kinds of information from each article: intervention details and indices of DMHI engagement. A 5-element framework of minimum necessary DMHI engagement reporting was derived by our team and guided our data extraction. This framework included (1) recommended app use as communicated to participants at enrollment and, when reported, app adherence criteria; (2) rate of intervention uptake among those assigned to the intervention; (3) level of app use metrics reported, specifically number of uses and time spent using the app; (4) duration of app use metrics (ie, weekly use patterns); and (5) number of intervention completers.ResultsDatabase searching yielded 2083 unique records. Of these, 22 studies were eligible for inclusion. Only 64% (14/22) of studies included in this review specified rate of intervention uptake. Level of use metrics was only reported in 59% (13/22) of the studies reviewed. Approximately one-quarter of the studies (5/22, 23%) reported duration of use metrics. Only half (11/22, 50%) of the studies reported the number of participants who completed the app-based components of the intervention as intended or other metrics related to completion. Findings in those studies reporting metrics related to intervention completion indicated that between 14.4% and 93.0% of participants randomized to a DMHI condition completed the intervention as intended or according to a specified adherence criteria.ConclusionsFindings suggest that engagement was underreported and widely varied. It was not uncommon to see completion rates at or below 50% (11/22) of those participants randomized to a treatment condition or to simply see completion rates not reported at all. This variability in reporting suggests a failure to establish sufficient reporting standards and limits the conclusions that can be drawn about level of engagement with DMHIs. Based on these findings, the 5-element framework applied in this review may be useful as a minimum necessary standard for DMHI engagement reporting.

Project description:BackgroundOlder adults (OAs) residing in long-term care (LTC) homes are often unable to engage in adequate amounts of physical activity because of multiple comorbidities, including frailty and severe cognitive impairments. This level of physical inactivity is associated with declines in cognitive and functional abilities and can be further compounded by social isolation. Exergaming, defined as a combination of exercise and gaming, has the potential to engage OAs in exercise and encourage social interaction. However, previously used systems such as the Nintendo Wii are no longer commercially available, and the physical design of other exergames is not suitable for OAs (ie, fall risks, accessibility issues, and games geared toward a younger population) with diverse physical and cognitive impairments.ObjectiveThis study aims to design and develop a novel, user-centered, evidence-based exergaming system for use among OAs in LTC homes. In addition, we aim to identify facilitators and barriers to the implementation of our exergaming intervention, the MouvMat, into LTC homes according to staff input.MethodsThis study used a user-centered design (UCD) process that consisted of 4 rounds of usability testing. The exergame was developed and finalized based on existing evidence, end user and stakeholder input, and user testing. Semistructured interviews and standardized and validated scales were used iteratively to evaluate the acceptability, usability, and physical activity enjoyment of the MouvMat.ResultsA total of 28 participants, 13 LTC residents, and 15 staff and family members participated in the UCD process for over 18 months to design and develop the novel exergaming intervention, the MouvMat. The iterative use of validated scales (System Usability Scale, 8-item Physical Activity Enjoyment Scale, and modified Treatment Evaluation Inventory) indicated an upward trend in the acceptability, usability, and enjoyment scores of MouvMat over 4 rounds of usability testing, suggesting that identified areas for refinement and improvement were appropriately addressed by the team. A qualitative analysis of semistructured interview data found that residents enjoyed engaging with the prototype and appreciated the opportunity to increase their PA. In addition, staff and stakeholders were drawn to MouvMat's ability to increase residents' autonomous PA. The intended and perceived benefits of MouvMat use, that is, improved physical and cognitive health, were the most common facilitators of its use identified by study participants.ConclusionsThis study was successful in applying UCD to collaborate with LTC residents, despite the high number of physical and sensory impairments that this population experiences. By following a UCD process, an exergaming intervention that meets diverse requirements (ie, hardware design features and motivation) and considers environmental barriers and residents' physical and cognitive needs was developed. The effectiveness of MouvMat in improving physical and cognitive abilities should be explored in future multisite randomized controlled trials.

Dataset Information

Procedures of User-Centered Usability Assessment for Digital Solutions: Scoping Review of Reviews Reporting on Digital Solutions Relevant for Older Adults.

Background

Objective

Methods

Results

Conclusions

Publications

Procedures of User-Centered Usability Assessment for Digital Solutions: Scoping Review of Reviews Reporting on Digital Solutions Relevant for Older Adults.

Similar Datasets

OmicsDI is part of the ELIXIR infrastructure

Tweets