Dataset Information

From expert-derived user needs to user-perceived ease of use and usefulness: a two-phase mixed-methods evaluation framework.

ABSTRACT: Underspecified user needs and frequent lack of a gold standard reference are typical barriers to technology evaluation. To address this problem, this paper presents a two-phase evaluation framework involving usability experts (phase 1) and end-users (phase 2). In phase 1, a cross-system functionality alignment between expert-derived user needs and system functions was performed to inform the choice of "the best available" comparison system to enable a cognitive walkthrough in phase 1 and a comparative effectiveness evaluation in phase 2. During phase 2, five quantitative and qualitative evaluation methods are mixed to assess usability: time-motion analysis, software log, questionnaires - System Usability Scale and the Unified Theory of Acceptance of Use of Technology, think-aloud protocols, and unstructured interviews. Each method contributes data for a unique measure (e.g., time motion analysis contributes task-completion-time; software log contributes action transition frequency). The measures are triangulated to yield complementary insights regarding user-perceived ease-of-use, functionality integration, anxiety during use, and workflow impact. To illustrate its use, we applied this framework in a formative evaluation of a software called Integrated Model for Patient Care and Clinical Trials (IMPACT). We conclude that this mixed-methods evaluation framework enables an integrated assessment of user needs satisfaction and user-perceived usefulness and usability of a novel design. This evaluation framework effectively bridges the gap between co-evolving user needs and technology designs during iterative prototyping and is particularly useful when it is difficult for users to articulate their needs for technology support due to the lack of a baseline.

SUBMITTER: Boland MR

PROVIDER: S-EPMC4055529 | biostudies-literature | 2014 Dec

REPOSITORIES: biostudies-literature

ACCESS DATA

Publications

From expert-derived user needs to user-perceived ease of use and usefulness: a two-phase mixed-methods evaluation framework.

Boland Mary Regina MR Rusanov Alexander A So Yat Y Lopez-Jimenez Carlos C Busacca Linda L Steinman Richard C RC Bakken Suzanne S Bigger J Thomas JT Weng Chunhua C

Journal of biomedical informatics 20131212

Underspecified user needs and frequent lack of a gold standard reference are typical barriers to technology evaluation. To address this problem, this paper presents a two-phase evaluation framework involving usability experts (phase 1) and end-users (phase 2). In phase 1, a cross-system functionality alignment between expert-derived user needs and system functions was performed to inform the choice of "the best available" comparison system to enable a cognitive walkthrough in phase 1 and a compa ...[more]

PMID: 24333875

Similar Datasets

Project description:BackgroundInternet and social media platforms offer insights into the lived experiences of survivors of cancer and their caregivers; however, the volume of narrative data available is often cumbersome for thorough analysis. Survivors of gynecological cancer have unique needs, such as those related to a genetic predisposition to future cancers, impact of cancer on sexual health, the advanced stage at which many are diagnosed, and the influx of new therapeutic approaches.ObjectiveThis study aimed to present a unique methodology to leverage large amounts of data from internet-based platforms for mixed methods analysis. We analyzed discussion board posts made by survivors of gynecological cancer on the American Cancer Society website with a particular interest in evaluating the psychosocial aspects of survivorship.MethodsAll posts from the ovarian, uterine, and gynecological cancers (other than ovarian and uterine) discussion boards on the American Cancer Society Cancer Survivors Network were included. Posts were web scraped using Python and organized by psychosocial themes described in the Quality of Cancer Survivorship Care Framework. Keywords related to each theme were generated and verified. Keywords identified posts related to the predetermined psychosocial themes. Quantitative analysis was completed using Python and R Foundation for Statistical Computing packages. Qualitative analysis was completed on a subset of posts as a proof of concept. Themes discovered through latent Dirichlet allocation (LDA), an unsupervised topic modeling technique, were assessed and compared with the predetermined themes of interest.ResultsA total of 125,498 posts made by 6436 survivors of gynecological cancer and caregivers between July 2000 and February 2020 were evaluated. Of the 125,489 posts, 23,458 (18.69%) were related to the psychosocial experience of cancer and were included in the mixed methods psychosocial analysis. Quantitative analysis (23,458 posts) revealed that survivors across all gynecological cancer discussion boards most frequently discussed the role of friends and family in care, as well as fatigue, the effect of cancer on interpersonal relationships, and health insurance status. Words related to psychosocial aspects of survivorship most often used in posts included "family," "hope," and "help." Qualitative analysis (20 of the 23,458 posts) similarly demonstrated that survivors frequently discussed coping strategies, distress and worry, the role of family and caregivers in their cancer care, and the toll of managing financial and insurance concerns. Using LDA, we discovered 8 themes, none of which were directly related to psychosocial aspects of survivorship. Of the 56 keywords identified by LDA, 2 (4%), "sleep" and "work," were included in the keyword list that we independently devised.ConclusionsWeb-based discussion platforms offer a great opportunity to learn about patient experiences of survivorship. Our novel methodology expedites the quantitative and qualitative analyses of such robust data, which may be used for additional patient populations.

Project description:BACKGROUND:Data from countries that have implemented a complete phase out of dental amalgam following the Minamata agreement suggest increased costs and time related to the placement of alternatives with consumers absorbing the additional costs. This aim of this study was to investigate the impact of a complete phase out of dental amalgam on oral health inequalities in particular for countries dependent on state run oral health services. METHODS:A mixed methods component design quantitative and qualitative study in the United Kingdom. The quantitative study involved acquisition and analysis of datasets from NHS Scotland to compare trends in placement of dental amalgam and a survey of GDPs in Yorkshire, UK. The qualitative study involved analysis of the free text of the survey and a supplementary secondary analysis of semi-structured interviews and focus groups with GDPs (private and NHS), dental school teaching leads and NHS dental commissioners to understand the impact of amalgam phase down on oral health inequalities. RESULTS:Time-trends for amalgam placement showed that there was a significant (p?<?0.05) reduction in amalgam use compared with composites and glass ionomers. However dental amalgam still represented a large proportion (42%) of the restorations (circa 1.8 million) placed in the 2016-2017 financial year. Survey respondents suggest that direct impacts of a phase down were related to increased costs and time to place alternative restorations and reduced quality of care. This in turn would lead to increased tooth extractions, reduced access to care and privatisation of dental services with the greatest impact on deprived populations. CONCLUSION:Amalgam is still a widely placed material in state run oral health services. The complete phase down of dental amalgam poses a threat to such services and threatens to widen oral health inequalities. Our data suggest that a complete phase out is not currently feasible unless appropriate measures are in place to ensure cheaper, long-lasting and easy to use alternatives are available and can be readily adopted by primary care oral health providers.

Project description:BackgroundAs more people are surviving stroke, there is a growing need for services and programs that support the long-term needs of people living with the effects of stroke. Exercise has many benefits; however, most people with stroke do not have access to specialized exercise programs that meet their needs in their communities. To catalyze the implementation of these programs, our team developed the Stroke Recovery in Motion Implementation Planner, an evidence-informed implementation guide for teams planning a community-based exercise program for people with stroke.ObjectiveThis study aimed to conduct a user evaluation to elicit user perceptions of the usefulness and acceptability of the Planner to inform revisions.MethodsThis mixed methods study used a concurrent triangulation design. We used purposive sampling to enroll a diverse sample of end users (program managers and coordinators, rehabilitation health partners, and fitness professionals) from three main groups: those who are currently planning a program, those who intend to plan a program in the future, and those who had previously planned a program. Participants reviewed the Planner and completed a questionnaire and interviews to identify positive features, areas of improvement, value, and feasibility. We used descriptive statistics for quantitative data and content analysis for qualitative data. We triangulated the data sources to identify Planner modifications.ResultsA total of 39 people participated in this study. Overall, the feedback was positive, highlighting the value of the Planner's comprehensiveness, tools and templates, and real-world examples. The identified areas for improvement included clarifying the need for specific steps, refining navigation, and creating more action-oriented content. Most participants reported an increase in knowledge and confidence after reading the Planner and reported that using the resource would improve their planning approach.ConclusionsWe used a rigorous and user-centered process to develop and evaluate the Planner. End users indicated that it is a valuable resource and identified specific changes for improvement. The Planner was subsequently updated and is now publicly available for community planning teams to use in the planning and delivery of evidence-informed, sustainable, community-based exercise programs for people with stroke.

Project description:BackgroundThere is a need for medical education on health care transitions for youth with special health care needs. The Texas Transition Toolkit (the tool) supports providers through a one-stop shop for researching literature on care transitions, a catalog of care transition tools, and guides for developing care transition programs.ObjectiveThis study aims to assess the functionality and usability of the tool with providers working with transition-aged children and youth with special health care needs (representative users).MethodsThe tool was evaluated using a triangulated mixed methods case study approach consisting of a concurrent think-aloud phase, a satisfaction survey, and a survey of problem relevance and task performance to operationalize and capture functionality and usability. Our mixed methods deep dive into the functionality and usability of the tool focused on 10 representative users from one medical home in Texas and 5 website design experts.ResultsRepresentative users found the tool to be highly relevant, as demonstrated by the satisfaction score for relevance (138/150, 92%). According to the users, the tool provided comprehensive information related to health care transitions for youth with special health care needs, with a satisfaction score of 87.3% (131/150) for comprehensive. Overall satisfaction with the tool was high at 81.92% (1065/1300) with a cutoff score of 73.33% (953.4/1300) indicating high satisfaction, but users reported relatively lower satisfaction with search (114/150, 76%) and navigation (ease of use: 114/150, 76%; hyperlinks: 163/200, 81.5%; structure: 159/200, 79.5%). They experienced search- and navigation-related problems (total problems detected: 21/31, 68%) and, based on quality checks, had a relatively low task completion rate for tasks involving finding information (60/80, 75%), which required searching and navigation. The problems identified around search and navigation functionality were relevant (relevance scores ranging from 14.5 to 22, with a cutoff score of 11.7 indicating relevance).ConclusionsThe tool may help bridge the gaps in training on health care transitions for youth with special health care needs in US medical education. The tool can be used to create structured protocols to help improve provider knowledge, collaboration across pediatric and adult care providers, and the continuity of care as youth with special health care needs transition from pediatric to adult care. The results provided a road map for optimizing the tool and highlighted the importance of evaluating eHealth technologies with representative users.

Project description:BackgroundInformal carers play an increasingly vital role in supporting the older population and the sustainability of health care systems. Care Companion is a theory-based and coproduced Web-based intervention to help support informal carers' resilience. It aims to provide personalized access to information and resources that are responsive to individuals' caring needs and responsibilities and thereby reduce the burdens associated with caregiving roles. Following the development of a prototype, it was necessary to undertake user acceptability testing to assess its suitability for wider implementation.ObjectiveThis study aimed to undertake user acceptance testing to investigate the perceived usefulness and ease of use of Care Companion. The key objectives were to (1) explore how potential and actual users perceived its usefulness, (2) explore the barriers and facilitators to its uptake and use and (3) gather suggestions to inform plans for an area-wide implementation.MethodsWe conducted user acceptance testing underpinned by principles of rapid appraisal using a qualitative descriptive approach. Focus groups, observations, and semistructured interviews were used in two phases of data collection. Participants were adult carers who were recruited through local support groups. Within the first phase, think-aloud interviews and observations were undertaken while the carers familiarized themselves with and navigated through the platform. In the second phase, focus group discussions were undertaken. Interested participants were then invited to trial Care Companion for up to 4 weeks and were followed up through semistructured telephone interviews exploring their experiences of using the platform. Thematic analysis was applied to the data, and a coding framework was developed iteratively with each phase of the study, informing subsequent phases of data collection and analysis.ResultsOverall, Care Companion was perceived to be a useful tool to support caregiving activities. The key themes were related to its appearance and ease of use, the profile setup and log-in process, concerns related to the safety and confidentiality of personal information, potential barriers to use and uptake and suggestions for overcoming them, and suggestions for improving Care Companion. More specifically, these related to the need for personalized resources aimed specifically at the carers (instead of care recipients), the benefits of incorporating a Web-based journal, the importance of providing transparency about security and data usage, minimizing barriers to initial registration, offering demonstrations to support uptake by people with low technological literacy, and the need to develop a culturally sensitive approach.ConclusionsThe findings identified ways of improving the ease of use and usefulness of Care Companion and demonstrated the importance of undertaking detailed user acceptance testing when developing an intervention for a diverse population, such as informal carers of older people. These findings have informed the further refinement of Care Companion and the strategy for its full implementation.

Project description:BackgroundIdentifying feasible and effective implementation strategies that are contextually appropriate is a challenge for researchers and implementers, exacerbated by the lack of conceptual clarity surrounding terms and definitions for implementation strategies, as well as a literature that provides imperfect guidance regarding how one might select strategies for a given healthcare quality improvement effort. In this study, we will engage an Expert Panel comprising implementation scientists and mental health clinical managers to: establish consensus on a common nomenclature for implementation strategy terms, definitions and categories; and develop recommendations to enhance the match between implementation strategies selected to facilitate the use of evidence-based programs and the context of certain service settings, in this case the U.S. Department of Veterans Affairs (VA) mental health services.Methods/designThis study will use purposive sampling to recruit an Expert Panel comprising implementation science experts and VA mental health clinical managers. A novel, four-stage sequential mixed methods design will be employed. During Stage 1, the Expert Panel will participate in a modified Delphi process in which a published taxonomy of implementation strategies will be used to establish consensus on terms and definitions for implementation strategies. In Stage 2, the panelists will complete a concept mapping task, which will yield conceptually distinct categories of implementation strategies as well as ratings of the feasibility and effectiveness of each strategy. Utilizing the common nomenclature developed in Stages 1 and 2, panelists will complete an innovative menu-based choice task in Stage 3 that involves matching implementation strategies to hypothetical implementation scenarios with varying contexts. This allows for quantitative characterizations of the relative necessity of each implementation strategy for a given scenario. In Stage 4, a live web-based facilitated expert recommendation process will be employed to establish expert recommendations about which implementations strategies are essential for each phase of implementation in each scenario.DiscussionUsing a novel method of selecting implementation strategies for use within specific contexts, this study contributes to our understanding of implementation science and practice by sharpening conceptual distinctions among a comprehensive collection of implementation strategies.

Dataset Information

From expert-derived user needs to user-perceived ease of use and usefulness: a two-phase mixed-methods evaluation framework.

Publications

From expert-derived user needs to user-perceived ease of use and usefulness: a two-phase mixed-methods evaluation framework.

Similar Datasets

OmicsDI is part of the ELIXIR infrastructure

Tweets