Dataset Information

Usability Testing of a Digital Assessment Routing Tool: Protocol for an Iterative Convergent Mixed Methods Study.

ABSTRACT:

Background

Musculoskeletal conditions account for 16% of global disability, resulting in a negative effect on millions of patients and an increasing burden on health care utilization. Digital technologies that improve health care outcomes and efficiency are considered a priority; however, innovations are often inadequately developed and poorly adopted. Further, they are rarely tested with sufficient rigor in clinical trials-the gold standard for clinical proof of efficacy. We have developed a new musculoskeletal Digital Assessment Routing Tool (DART) that allows users to self-assess and be directed to the right care. DART requires usability testing in preparation for clinical trials.

Objective

This study will use the iterative convergent mixed methods design to assess and mitigate all serious usability issues to optimize user experience and adoption. Using this methodology, we will provide justifiable confidence to progress to full-scale randomized controlled trials when DART is integrated into clinical management pathways. This study protocol will provide a blueprint for future usability studies of mobile health solutions.

Methods

We will collect qualitative and quantitative data from 20-30 participants aged 18 years and older for 4 months. The exact number of participants recruited will be dependent on the number of iterative cycles required to reach the study end points. Building on previous internal testing and stakeholder involvement, quantitative data collection is defined by the constructs within the ISO 9241-210-2019 standard and the system usability scale, providing a usability score for DART. Guided by the participant responses to quantitative questioning, the researcher will focus the qualitative data collection on specific usability problems. These will then be graded to provide the rationale for further DART system improvements throughout the iterative cycles.

Results

This study received approval from the Queen Mary University of London Ethics of Research Committee (QMREC2018/48/048) on June 4, 2020. At manuscript submission, study recruitment was on-going, with data collection to be completed and results published in 2021.

Conclusions

This study will provide evidence concerning mobile health DART system usability and acceptance determining system improvements required to support user adoption and minimize suboptimal system usability as a potential confounder within subsequent noninferiority clinical trials. Success should produce a safe effective system with excellent usability, facilitating quicker and easier patient access to appropriate care while reducing the burden on primary and secondary care musculoskeletal services. This deliberately rigorous approach to mobile health innovation could be used as a guide for other developers of similar apps.

International registered report identifier (irrid)

DERR1-10.2196/27205.

SUBMITTER: Lowe C

PROVIDER: S-EPMC8170557 | biostudies-literature |

REPOSITORIES: biostudies-literature

ACCESS DATA

Similar Datasets

Project description:BackgroundCholera poses a significant global health burden. In Bangladesh, cholera is endemic and causes more than 100,000 cases each year. Established environmental reservoirs leave millions at risk of infection through the consumption of contaminated water. The Global Task Force for Cholera Control has called for increased environmental surveillance to detect contaminated water sources prior to human infection in an effort to reduce cases and deaths. The OmniVis rapid cholera detection device uses loop-mediated isothermal amplification and particle diffusometry detection methods integrated into a handheld hardware device that attaches to an iPhone 6 to identify and map contaminated water sources.ObjectiveThe aim of this study was to evaluate the usability of the OmniVis device with targeted end users to advance the iterative prototyping process and ultimately design a device that easily integrates into users' workflow.MethodsWater quality workers were trained to use the device and subsequently completed an independent device trial and usability questionnaire. Pretraining and posttraining knowledge assessments were administered to ensure training quality did not confound trial and questionnaire.ResultsDevice trials identified common user errors and device malfunctions including incorrect test kit insertion and device powering issues. We did not observe meaningful differences in user errors or device malfunctions accumulated per participant across demographic groups. Over 25 trials, the mean time to complete a test was 47 minutes, a significant reduction compared with laboratory protocols, which take approximately 3 days. Overall, participants found the device easy to use and expressed confidence and comfort in using the device independently.ConclusionsThese results are used to advance the iterative prototyping process of the OmniVis rapid cholera detection device so it can achieve user uptake, workflow integration, and scale to ultimately impact cholera control and elimination strategies. We hope this methodology will promote robust usability evaluations of rapid pathogen detection technologies in device development.

Project description:BACKGROUND:Mobile interventions hold promise as an intervention modality to engage children in improving diabetes self-management education, attitudes, and behaviors. OBJECTIVE:This pilot study aimed to explore the usability, acceptability, and feasibility of delivering a mobile diabetes educational tool to parent-child pairs in a clinical setting. METHODS:This mixed methods pilot study comprised two concurrent phases with differing study participants. Phase 1 used user testing interviews to collect qualitative data on the usability and acceptability of the tool. Phase 2 used a single-arm pre- and poststudy design to quantitatively evaluate the feasibility and preliminary efficacy of the intervention. Study participants (English-speaking families with youth aged 5-14 years with insulin-dependent diabetes) were recruited from an urban hospital in Massachusetts, United States. In phase 1, parent-child pairs were invited to complete the intervention together and participate in 90-min user testing interviews assessing the tool's usability and acceptability. Interview transcripts were analyzed using a directed content analysis approach. In phase 2, parent-child pairs were invited to complete the intervention together in the clinical setting. Measures included parental and child knowledge, attitudes, and behaviors related to diabetes management (self-report surveys) and child hemoglobin A1c levels (medical record extractions); data were collected at baseline and 1-month follow-up. Pre- and postoutcomes were compared using paired t tests and the Fisher exact test. RESULTS:A total of 11 parent-child pairs (N=22) participated in phase 1 of the study, and 10 parent-child pairs (N=20) participated in phase 2 of the study. Participants viewed the mobile educational tool as acceptable (high engagement and satisfaction with the layout, activities, and videos) and identified the areas of improvement for tool usability (duration, directions, and animation). CONCLUSIONS:The findings from this pilot study suggest that the mobile educational tool is an informative, engaging, and feasible way to deliver diabetes self-management education to parents and children in an urban hospital setting. Data will inform future iterations of this mobile diabetes educational intervention to improve usability and test intervention efficacy.

Project description:BackgroundMusculoskeletal conditions account for 16% of global disability, resulting in a negative effect on millions of patients and an increasing demand for health care use. Digital technologies to improve health care outcomes and efficiency are considered a priority; however, innovations are rarely tested with sufficient rigor in clinical trials, which is the gold standard for clinical proof of safety and efficacy. We have developed a new musculoskeletal digital assessment routing tool (DART) that allows users to self-assess and be directed to the right care. DART requires validation in a real-world setting before implementation.ObjectiveThis pilot study aims to assess the feasibility of a future trial by exploring the key aspects of trial methodology, assessing the procedures, and collecting exploratory data to inform the design of a definitive randomized crossover noninferiority trial to assess DART safety and effectiveness.MethodsWe will collect data from 76 adults with a musculoskeletal condition presenting to general practitioners within a National Health Service (NHS) in England. Participants will complete both a DART assessment and a physiotherapist-led triage, with the order determined by randomization. The primary analysis will involve an absolute agreement intraclass correlation (A,1) estimate with 95% CI between DART and the clinician for assessment outcomes signposting to condition management pathways. Data will be collected to allow the analysis of participant recruitment and retention, randomization, allocation concealment, blinding, data collection process, and bias. In addition, the impact of trial burden and potential barriers to intervention delivery will be considered. The DART user satisfaction will be measured using the system usability scale.ResultsA UK NHS ethics submission was done during June 2021 and is pending approval; recruitment will commence in early 2022, with data collection anticipated to last for 3 months. The results will be reported in a follow-up paper in 2022.ConclusionsThis study will inform the design of a randomized controlled crossover noninferiority study that will provide evidence concerning mobile health DART system clinical signposting in an NHS setting before real-world implementation. Success should produce evidence of a safe, effective system with good usability, potentially facilitating quicker and easier patient access to appropriate care while reducing the burden on primary and secondary care musculoskeletal services. This rigorous approach to mobile health system testing could be used as a guide for other developers of similar applications.Trial registrationClinicalTrials.gov NCT04904029; http://clinicaltrials.gov/ct2/show/NCT04904029.International registered report identifier (irrid)PRR1-10.2196/31541.

Project description:BackgroundReporting of medication errors is one of the essential mechanisms to identify risky health care systems and practices that lead to medication errors. Unreported medication errors are a real issue; one of the identified causes is a burdensome medication error reporting system. An anonymous and user-friendly mobile app for reporting medication errors could be an alternative method of reporting medication error in busy health care settings.ObjectiveThe objective of this paper is to report usability testing of the Medication Error Reporting App (MERA), a mobile app for reporting medication errors anonymously.MethodsQuantitative and qualitative methods were employed involving 45 different testers (pharmacists, doctors, and nurses) from a large tertiary hospital in Malaysia. Quantitative data was retrieved using task performance and rating of MERA and qualitative data were retrieved through focus group discussions. Three sessions, with 15 testers each session, were conducted from January to March 2018.ResultsThe majority of testers were pharmacists (23/45, 51%), female (35/45, 78%), and the mean age was 36 (SD 9) years. A total of 135 complete reports were successfully submitted by the testers (three reports per tester) and 79.2% (107/135) of the reports were correct. There was significant improvement in mean System Usability Scale scores in each session of the development process (P<.001) and mean time to report medication errors using the app was not significantly different between each session (P=.70) with an overall mean time of 6.7 (SD 2.4) minutes. Testers found the app easy to use, but doctors and nurses were unfamiliar with terms used especially medication process at which error occurred and type of error. Although, testers agreed the app can be used in the future for reporting, they were apprehensive about security, validation, and abuse of feedback featured in the app.ConclusionsMERA can be used to report medication errors easily by various health care personnel and it has the capacity to provide feedback on reporting. However, education on medication error reporting should be provided to doctors and nurses in Malaysia and the security of the app needs to be established to boost reporting by this method.

Project description:BackgroundYouth who experience traumatic events are at a substantially higher risk of engaging in substance use and sexual risk behaviors and problems (eg, HIV acquisition) than their non-trauma-exposed counterparts. Evidence-based substance use and risky sexual behavior prevention may reduce the risk of these outcomes. Trauma-focused mental health treatment provides a window of opportunity for the implementation of such preventive work with these youth. However, overburdened clinicians face challenges in adding prevention content while implementing evidence-based treatments. Mobile health (mHealth) tools can help reduce this burden in delivering prevention curricula. Trauma-Informed Prevention for Substance Use and Risky Sexual Behavior (TIPS) is an mHealth app that was developed to aid trauma-focused cognitive behavioral therapy (TF-CBT) clinicians in the implementation of an evidence-based risk behavior prevention curriculum.ObjectiveThe goal of this paper is to describe the rationale for and development of the TIPS app and present the results of a mixed methods approach for the initial evaluation of its usability.MethodsParticipants included clinicians (n=11), adolescents (n=11), and caregivers (n=10) who completed qualitative interviews and an adapted version of the Website Analysis and Measurement Inventory.ResultsIn total, 4 overarching themes emerged from the participants' answers to the qualitative interview questions, demonstrating a generally positive response to the app. The themes were (1) strength of app content, (2) suggestions about app content, (3) esthetics and usability, and (4) benefits to the patient and session implementation. Clinicians, adolescents, and caregivers all agreed that the content was very relevant to adolescents and used examples and language that adolescents could relate to. All 3 groups also discussed that the content was comprehensive and addressed issues often faced by adolescents. All 3 groups of users made suggestions about the esthetics, which mostly comprised suggestions to change the font, color, or pictures within the app. Of all the groups, adolescents were most positive about the esthetics and usability of the app. Results from the Website Analysis and Measurement Inventory further illustrated the users' favorable reaction to the TIPS app, with 100% (11/11) of clinicians, 100% (10/10) of caregivers, and most adolescents (7/11, 64%) selecting strongly agree or somewhat agree to the following statement: "This app has much that is of interest to me." Adolescents generally found the app easier to use than did caregivers and clinicians.ConclusionsThe TIPS app shows promise as an mHealth tool for TF-CBT clinicians to integrate evidence-based substance use, risky sexual behavior, and HIV prevention during treatment. Future research, including a randomized controlled trial comparing TF-CBT implementation with and without the inclusion of the app, is necessary to evaluate the feasibility and efficacy of the app in reducing the risk of substance use and risky sexual behavior among trauma-exposed adolescents.Trial registrationClinicalTrials.gov NCT03710720; https://clinicaltrials.gov/study/NCT03710720.

Project description:IntroductionThe prevalence of type 2 diabetes is rising steeply. National Health Service England (NHSE) is exploring the potential of a digital diabetes prevention programme (DDPP) and has commissioned a pilot with embedded evaluation.Methods and analysisThis study aims to determine whether, and if so, how, should NHSE implement a national DDPP, using a mixed-methods pretest and post-test design, underpinned by two theoretical frameworks: the Coventry, Aberdeen and London - Refined (CALO-RE) taxonomy of behavioural change techniques for the digital interventions and the Consolidated Framework for Implementation Research (CFIR) for implementation processes. In eight pilot areas across England, adults with non-diabetic hyperglycaemia (NDH) (glycated haemoglobin (HbA1c) 42-47 mmol/mol or fasting plasma glucose 5.5-6.9 mmol/L) and adults without NDH who are overweight (body mass index (BMI) >25 kg/m2) or obese (BMI >30 kg/m2) will be referred to one of five digitally delivered diabetes prevention interventions. The primary outcomes are reduction in HbA1c and weight (for people with NDH) and reduction in weight (for people who are overweight or obese) at 12 months. Secondary outcomes include use of the intervention, satisfaction, physical activity, patient activation and resources needed for successful implementation. Quantitative data will be collected at baseline, 6 months and 12 months by the digital intervention providers. Qualitative data will be collected through semistructured interviews with commissioners, providers, healthcare professionals and patients. Quantitative data will be analysed descriptively and using generalised linear models to determine whether changes in outcomes are associated with demographic and intervention factors. Qualitative data will be analysed using framework analysis, with data pertaining to implementation mapped onto the CFIR.Ethics and disseminationThe study has received ethical approval from the Public Health England Ethics and Research Governance Group (reference R&D 324). Dissemination will include a report to NHSE to inform future policy and publication in peer-reviewed journals.

Project description:BACKGROUND:Language barriers play a decisive role in determining the outcomes of medical consultations between healthcare providers and their foreign patients. This issue is a significant challenge to the German healthcare system, especially with the rising number of refugees in recent years. The communication gap between healthcare professionals and their non-German speaking patients sometimes leads to unnecessary medical re-admission, insufficient medical history, incorrect diagnosis, and treatment plans. In this study, we aim to assess the usability and accuracy of a novel digital translation tool in collecting medical history from patients in their native language and to check its effects on healthcare outcomes. METHODS:The study aims to monitor the implementation of a new digital communication assistance tool (DCAT) and to investigate its impact on the mutual understanding between refugee patients and their German general practitioners (GPs). In the first study phase, an action-oriented approach is used to implement DCAT. In the second study phase, DCAT use will be evaluated with a mixed methods design. The main outcome assesses the re-consultation rates of patients before and after using DCAT. Secondary outcomes include the usability of the tool, its acceptance and perceived quality by patients, the accuracy of the information collected as determined from analysing the reasons for the consultation (ICPC-2 codes), and diagnosis (ICD-10 codes). The acceptance by patients, socio-demographic factors and native language are also taken into account. The research designs for both study phases include questionnaires, semi-structured interviews, non-participant observation and analysis of collected patients' data. All the collected data is pseudonymised. DISCUSSION:The DCAT study is one of the new research projects in primary healthcare investigating the usability, accuracy, and acceptance of digital translation tools during medical encounters. We aim to eliminate significant communication errors and misunderstandings in medical consultations, thereby improving the quality of healthcare outcomes. By applying an action research design, we will attain a more comprehensive evaluation of DCAT scopes and limits. The results of this study are expected to give an in-depth understanding of possible applications and benefits of digital translation tools for patient care. TRIAL REGISTRATION:German Clinical Trials Register DRKS00013076 , 29/09/2017.

Project description:BackgroundAwareness of lung cancer screening remains low in the screening-eligible population, and when patients visit their clinician never having heard of lung cancer screening, engaging in shared decision making to arrive at an informed decision can be a challenge. Therefore, methods to effectively support both patients and clinicians to engage in these important discussions are essential. To facilitate shared decision making about lung cancer screening, effective methods to prepare patients to have these important discussions with their clinician are needed.ObjectiveOur objective is to develop a computer-tailored decision support tool that meets the certification criteria of the International Patient Decision Aid Standards instrument version 4.0 that will support shared decision making in lung cancer screening decisions.MethodsUsing a 3-phase process, we will develop and test a prototype of a computer-tailored decision support tool in a sample of lung cancer screening-eligible individuals. In phase I, we assembled a community advisory board comprising 10 screening-eligible individuals to develop the prototype. In phase II, we recruited a sample of 13 screening-eligible individuals to test the prototype for usability, acceptability, and satisfaction. In phase III, we are conducting a pilot randomized controlled trial (RCT) with 60 screening-eligible participants who have never been screened for lung cancer. Outcomes tested include lung cancer and screening knowledge, lung cancer screening health beliefs (perceived risk, perceived benefits, perceived barriers, and self-efficacy), perception of being prepared to engage in a patient-clinician discussion about lung cancer screening, occurrence of a patient-clinician discussion about lung cancer screening, and stage of adoption for lung cancer screening.ResultsPhases I and II are complete. Phase III is underway. As of July 15, 2017, 60 participants have been enrolled into the study, and have completed the baseline survey, intervention, and first follow-up survey. We expect to have results by December 31, 2017 and to have data analysis completed by March 1, 2018.ConclusionsResults from usability testing indicate that the computer-tailored decision support tool is easy to use, is helpful, and provides a satisfactory experience for the user. At the conclusion of phase III (pilot RCT), we will have preliminary effect sizes to inform a future fully powered RCT on changes in (1) knowledge about lung cancer and screening, (2) perceived risk of lung cancer, (3) perceived benefits of lung cancer screening, (4) perceived barriers to lung cancer screening, (5) self-efficacy for lung cancer screening, and (6) perceptions of being adequately prepared to engage in a discussion with their clinician about lung cancer screening.

Project description:BackgroundMusculoskeletal conditions account for 16% of global disability, resulting in a negative effect on patients and increasing demand for health care use. Triage directing patients to appropriate level intervention improving health outcomes and efficiency has been prioritized. We developed a musculoskeletal digital assessment routing tool (DART) mobile health (mHealth) system, which requires evaluation prior to implementation. Such innovations are rarely rigorously tested in clinical trials-considered the gold standard for evaluating safety and efficacy. This pilot study is a precursor to a trial assessing DART performance with a physiotherapist-led triage assessment.ObjectiveThe study aims to evaluate trial design, assess procedures, and collect exploratory data to establish the feasibility of delivering an adequately powered, definitive randomized trial, assessing DART safety and efficacy in an NHS primary care setting.MethodsA crossover, noninferiority pilot trial using an integrated knowledge translation approach within a National Health Service England primary care setting. Participants were patients seeking assessment for a musculoskeletal condition, completing a DART assessment and the history-taking element of a face-to-face physiotherapist-led triage in a randomized order. The primary outcome was agreement between DART and physiotherapist triage recommendation. Data allowed analysis of participant recruitment and retention, randomization, blinding, study burden, and potential barriers to intervention delivery. Participant satisfaction was measured using the System Usability Scale.ResultsOver 8 weeks, 129 patients were invited to participate. Of these, 92% (119/129) proceeded to eligibility assessment, with 60% (78/129) meeting the inclusion criteria and being randomized into each intervention arm (39/39). There were no dropouts and data were analyzed for all 78 participants. Agreement between physiotherapist and DART across all participants and all primary triage outcomes was 41% (32/78; 95% CI 22-45), intraclass correlation coefficient 0.37 (95% CI 0.16-0.55), indicating that the reliability of DART was poor to moderate. Feedback from the clinical service team led to an adjusted analysis yielding of 78% (61/78; 95% CI 47-78) and an intraclass correlation coefficient of 0.57 (95% CI 0.40-0.70). Participant satisfaction was measured quantitively using amalgamated System Usability Scale scores (n=78; mean score 84.0; 90% CI +2.94 to -2.94), equating to an "excellent" system. There were no study incidents, and the trial burden was acceptable.ConclusionsPhysiotherapist-DART agreement of 78%, with no adverse triage decisions and high patient satisfaction, was sufficient to conclude DART had the potential to improve the musculoskeletal pathway. Study validity was enhanced by the recruitment of real-world patients and using an integrated knowledge translation approach. Completion of a context-specific consensus process is recommended to provide definitive definitions of safety criteria, range of appropriateness, noninferiority margin, and sample size. This pilot demonstrated an adequately powered definitive trial is feasible, which would provide evidence of DART safety and efficacy, ultimately informing potential for DART implementation.Trial registrationClinicalTrials.gov NCT04904029; http://clinicaltrials.gov/ct2/show/NCT04904029.International registered report identifier (irrid)RR2-10.2196/31541.