Dataset Information

From expert-derived user needs to user-perceived ease of use and usefulness: a two-phase mixed-methods evaluation framework.

ABSTRACT: Underspecified user needs and frequent lack of a gold standard reference are typical barriers to technology evaluation. To address this problem, this paper presents a two-phase evaluation framework involving usability experts (phase 1) and end-users (phase 2). In phase 1, a cross-system functionality alignment between expert-derived user needs and system functions was performed to inform the choice of "the best available" comparison system to enable a cognitive walkthrough in phase 1 and a comparative effectiveness evaluation in phase 2. During phase 2, five quantitative and qualitative evaluation methods are mixed to assess usability: time-motion analysis, software log, questionnaires - System Usability Scale and the Unified Theory of Acceptance of Use of Technology, think-aloud protocols, and unstructured interviews. Each method contributes data for a unique measure (e.g., time motion analysis contributes task-completion-time; software log contributes action transition frequency). The measures are triangulated to yield complementary insights regarding user-perceived ease-of-use, functionality integration, anxiety during use, and workflow impact. To illustrate its use, we applied this framework in a formative evaluation of a software called Integrated Model for Patient Care and Clinical Trials (IMPACT). We conclude that this mixed-methods evaluation framework enables an integrated assessment of user needs satisfaction and user-perceived usefulness and usability of a novel design. This evaluation framework effectively bridges the gap between co-evolving user needs and technology designs during iterative prototyping and is particularly useful when it is difficult for users to articulate their needs for technology support due to the lack of a baseline.

SUBMITTER: Boland MR

PROVIDER: S-EPMC4055529 | biostudies-literature | 2014 Dec

REPOSITORIES: biostudies-literature

ACCESS DATA

Publications

From expert-derived user needs to user-perceived ease of use and usefulness: a two-phase mixed-methods evaluation framework.

Boland Mary Regina MR Rusanov Alexander A So Yat Y Lopez-Jimenez Carlos C Busacca Linda L Steinman Richard C RC Bakken Suzanne S Bigger J Thomas JT Weng Chunhua C

Journal of biomedical informatics 20131212

Underspecified user needs and frequent lack of a gold standard reference are typical barriers to technology evaluation. To address this problem, this paper presents a two-phase evaluation framework involving usability experts (phase 1) and end-users (phase 2). In phase 1, a cross-system functionality alignment between expert-derived user needs and system functions was performed to inform the choice of "the best available" comparison system to enable a cognitive walkthrough in phase 1 and a compa ...[more]

PMID: 24333875

Similar Datasets

Project description:BACKGROUND:Data from countries that have implemented a complete phase out of dental amalgam following the Minamata agreement suggest increased costs and time related to the placement of alternatives with consumers absorbing the additional costs. This aim of this study was to investigate the impact of a complete phase out of dental amalgam on oral health inequalities in particular for countries dependent on state run oral health services. METHODS:A mixed methods component design quantitative and qualitative study in the United Kingdom. The quantitative study involved acquisition and analysis of datasets from NHS Scotland to compare trends in placement of dental amalgam and a survey of GDPs in Yorkshire, UK. The qualitative study involved analysis of the free text of the survey and a supplementary secondary analysis of semi-structured interviews and focus groups with GDPs (private and NHS), dental school teaching leads and NHS dental commissioners to understand the impact of amalgam phase down on oral health inequalities. RESULTS:Time-trends for amalgam placement showed that there was a significant (p?<?0.05) reduction in amalgam use compared with composites and glass ionomers. However dental amalgam still represented a large proportion (42%) of the restorations (circa 1.8 million) placed in the 2016-2017 financial year. Survey respondents suggest that direct impacts of a phase down were related to increased costs and time to place alternative restorations and reduced quality of care. This in turn would lead to increased tooth extractions, reduced access to care and privatisation of dental services with the greatest impact on deprived populations. CONCLUSION:Amalgam is still a widely placed material in state run oral health services. The complete phase down of dental amalgam poses a threat to such services and threatens to widen oral health inequalities. Our data suggest that a complete phase out is not currently feasible unless appropriate measures are in place to ensure cheaper, long-lasting and easy to use alternatives are available and can be readily adopted by primary care oral health providers.

Project description:BackgroundThere is a need for medical education on health care transitions for youth with special health care needs. The Texas Transition Toolkit (the tool) supports providers through a one-stop shop for researching literature on care transitions, a catalog of care transition tools, and guides for developing care transition programs.ObjectiveThis study aims to assess the functionality and usability of the tool with providers working with transition-aged children and youth with special health care needs (representative users).MethodsThe tool was evaluated using a triangulated mixed methods case study approach consisting of a concurrent think-aloud phase, a satisfaction survey, and a survey of problem relevance and task performance to operationalize and capture functionality and usability. Our mixed methods deep dive into the functionality and usability of the tool focused on 10 representative users from one medical home in Texas and 5 website design experts.ResultsRepresentative users found the tool to be highly relevant, as demonstrated by the satisfaction score for relevance (138/150, 92%). According to the users, the tool provided comprehensive information related to health care transitions for youth with special health care needs, with a satisfaction score of 87.3% (131/150) for comprehensive. Overall satisfaction with the tool was high at 81.92% (1065/1300) with a cutoff score of 73.33% (953.4/1300) indicating high satisfaction, but users reported relatively lower satisfaction with search (114/150, 76%) and navigation (ease of use: 114/150, 76%; hyperlinks: 163/200, 81.5%; structure: 159/200, 79.5%). They experienced search- and navigation-related problems (total problems detected: 21/31, 68%) and, based on quality checks, had a relatively low task completion rate for tasks involving finding information (60/80, 75%), which required searching and navigation. The problems identified around search and navigation functionality were relevant (relevance scores ranging from 14.5 to 22, with a cutoff score of 11.7 indicating relevance).ConclusionsThe tool may help bridge the gaps in training on health care transitions for youth with special health care needs in US medical education. The tool can be used to create structured protocols to help improve provider knowledge, collaboration across pediatric and adult care providers, and the continuity of care as youth with special health care needs transition from pediatric to adult care. The results provided a road map for optimizing the tool and highlighted the importance of evaluating eHealth technologies with representative users.

Project description:BackgroundInformal carers play an increasingly vital role in supporting the older population and the sustainability of health care systems. Care Companion is a theory-based and coproduced Web-based intervention to help support informal carers' resilience. It aims to provide personalized access to information and resources that are responsive to individuals' caring needs and responsibilities and thereby reduce the burdens associated with caregiving roles. Following the development of a prototype, it was necessary to undertake user acceptability testing to assess its suitability for wider implementation.ObjectiveThis study aimed to undertake user acceptance testing to investigate the perceived usefulness and ease of use of Care Companion. The key objectives were to (1) explore how potential and actual users perceived its usefulness, (2) explore the barriers and facilitators to its uptake and use and (3) gather suggestions to inform plans for an area-wide implementation.MethodsWe conducted user acceptance testing underpinned by principles of rapid appraisal using a qualitative descriptive approach. Focus groups, observations, and semistructured interviews were used in two phases of data collection. Participants were adult carers who were recruited through local support groups. Within the first phase, think-aloud interviews and observations were undertaken while the carers familiarized themselves with and navigated through the platform. In the second phase, focus group discussions were undertaken. Interested participants were then invited to trial Care Companion for up to 4 weeks and were followed up through semistructured telephone interviews exploring their experiences of using the platform. Thematic analysis was applied to the data, and a coding framework was developed iteratively with each phase of the study, informing subsequent phases of data collection and analysis.ResultsOverall, Care Companion was perceived to be a useful tool to support caregiving activities. The key themes were related to its appearance and ease of use, the profile setup and log-in process, concerns related to the safety and confidentiality of personal information, potential barriers to use and uptake and suggestions for overcoming them, and suggestions for improving Care Companion. More specifically, these related to the need for personalized resources aimed specifically at the carers (instead of care recipients), the benefits of incorporating a Web-based journal, the importance of providing transparency about security and data usage, minimizing barriers to initial registration, offering demonstrations to support uptake by people with low technological literacy, and the need to develop a culturally sensitive approach.ConclusionsThe findings identified ways of improving the ease of use and usefulness of Care Companion and demonstrated the importance of undertaking detailed user acceptance testing when developing an intervention for a diverse population, such as informal carers of older people. These findings have informed the further refinement of Care Companion and the strategy for its full implementation.

Project description:BackgroundIdentifying feasible and effective implementation strategies that are contextually appropriate is a challenge for researchers and implementers, exacerbated by the lack of conceptual clarity surrounding terms and definitions for implementation strategies, as well as a literature that provides imperfect guidance regarding how one might select strategies for a given healthcare quality improvement effort. In this study, we will engage an Expert Panel comprising implementation scientists and mental health clinical managers to: establish consensus on a common nomenclature for implementation strategy terms, definitions and categories; and develop recommendations to enhance the match between implementation strategies selected to facilitate the use of evidence-based programs and the context of certain service settings, in this case the U.S. Department of Veterans Affairs (VA) mental health services.Methods/designThis study will use purposive sampling to recruit an Expert Panel comprising implementation science experts and VA mental health clinical managers. A novel, four-stage sequential mixed methods design will be employed. During Stage 1, the Expert Panel will participate in a modified Delphi process in which a published taxonomy of implementation strategies will be used to establish consensus on terms and definitions for implementation strategies. In Stage 2, the panelists will complete a concept mapping task, which will yield conceptually distinct categories of implementation strategies as well as ratings of the feasibility and effectiveness of each strategy. Utilizing the common nomenclature developed in Stages 1 and 2, panelists will complete an innovative menu-based choice task in Stage 3 that involves matching implementation strategies to hypothetical implementation scenarios with varying contexts. This allows for quantitative characterizations of the relative necessity of each implementation strategy for a given scenario. In Stage 4, a live web-based facilitated expert recommendation process will be employed to establish expert recommendations about which implementations strategies are essential for each phase of implementation in each scenario.DiscussionUsing a novel method of selecting implementation strategies for use within specific contexts, this study contributes to our understanding of implementation science and practice by sharpening conceptual distinctions among a comprehensive collection of implementation strategies.

Project description:BackgroundWhile overactive bladder is often managed in the primary care setting, a number of barriers including embarrassment, poor communication, and low patient adherence contribute to the under-treatment of patients with burdensome urinary symptoms. In order to address these challenges, it is crucial to have a fundamental understanding of patient perspectives toward OAB and urinary symptoms. To meet this aim, researchers designed and conducted a study to identify patients' knowledge, experiences and attitudes, barriers to treatment adherence, and desires and tendencies regarding patient/clinician communications.MethodsA mixed-methods qualitative/quantitative needs assessment of patients with overactive bladder and/or urinary symptoms. Researchers conducted in-depth qualitative interviews via telephone with 40 patients. Interview results informed the design and dissemination of a quantitative survey, which was completed by 200 self-selected respondents who had previously identified themselves as having overactive bladder or bladder problems. Statistical and qualitative analysis of results were conducted.ResultsAmong survey respondents, an average of 3.5 years elapsed between symptom onset and seeking diagnosis by a physician. In the long term most patients do not experience improvement in symptoms. Medication non-adherence is common and is related to therapy effectiveness and adverse effects. Patients clearly indicate that communication and patient/physician relationships are important to them and they would prefer the clinician initiate the conversation on overactive bladder. Patient experiences, perspectives, and attitudes toward their bladder symptoms differ in many ways from clinicians' assumptions.ConclusionsThe significant time gap between symptom onset and diagnosis indicates ongoing need for screening and diagnosis of overactive bladder. Contrary to guideline recommendations, urinalysis and physical examination are not widely used in clinical practice. Many patients experience no improvement in symptoms over time. Patients indicate that clinician/patient relationships and communication regarding their condition are important.

Project description:ObjectiveTo describe patients' perceived value and use of quality measures in evaluating and choosing community pharmacies.DesignFocus group methodology was combined with a survey tool. During the focus groups, participants assessed the value of the Pharmacy Quality Alliance's quality measures in evaluating and choosing a pharmacy. Also, participants completed questionnaires rating their perceived value of quality measures in evaluating a pharmacy (1 being low value and 5 being high) or choosing a pharmacy (yes/no). Thematic analysis and descriptive statistics were used to analyse the focus groups and surveys, respectively.SettingSemistructured focus groups were conducted in a private meeting space of an urban and a rural area of a Mid-western State in the USA.ParticipantsThirty-four adults who filled prescription medications in community pharmacies for a chronic illness were recruited in community pharmacies, senior centres and public libraries.ResultsWhile comments indicated that all measures were important, medication safety measures (eg, drug-drug interactions) were valued more highly than others. Rating of quality measure utility in evaluating a pharmacy ranged from a mean of 4.88 ('drug-drug interactions') to a mean of 4.0 ('absence of controller therapy for patients with asthma'). Patients were hesitant to use quality information in choosing a pharmacy (depending on the participant's location) but might consider if moving to a new area or having had a negative pharmacy experience. Use of select quality measures to choose a pharmacy ranged from 97.1% of participants using 'drug-drug interactions' (medication safety measure) to 55.9% using 'absence of controller therapy for patients with asthma'.ConclusionsThe study participants valued quality measures in evaluating and selecting a community pharmacy, with medication safety measures valued highest. The participants reported that the quality measures would not typically cause a switch in pharmacy but might influence their selection in certain situations.

Dataset Information

From expert-derived user needs to user-perceived ease of use and usefulness: a two-phase mixed-methods evaluation framework.

Publications

From expert-derived user needs to user-perceived ease of use and usefulness: a two-phase mixed-methods evaluation framework.

Similar Datasets

OmicsDI is part of the ELIXIR infrastructure

Tweets