Dataset Information

Evaluation of Three Feasibility Tools for Identifying Patient Data and Biospecimen Availability: Comparative Usability Study.

ABSTRACT:

Background

To meet the growing importance of real-word data analysis, clinical data and biosamples must be timely made available. Feasibility platforms are often the first contact point for determining the availability of such data for specific research questions. Therefore, a user-friendly interface should be provided to enable access to this information easily. The German Medical Informatics Initiative also aims to establish such a platform for its infrastructure. Although some of these platforms are actively used, their tools still have limitations. Consequently, the Medical Informatics Initiative consortium MIRACUM (Medical Informatics in Research and Care in University Medicine) committed itself to analyzing the pros and cons of existing solutions and to designing an optimized graphical feasibility user interface.

Objective

The aim of this study is to identify the system that is most user-friendly and thus forms the best basis for developing a harmonized tool. To achieve this goal, we carried out a comparative usability evaluation of existing tools used by researchers acting as end users.

Methods

The evaluation included three preselected search tools and was conducted as a qualitative exploratory study with a randomized design over a period of 6 weeks. The tools in question were the MIRACUM i2b2 (Informatics for Integrating Biology and the Bedside) feasibility platform, OHDSI's (Observational Health Data Sciences and Informatics) ATLAS, and the Sample Locator of the German Biobank Alliance. The evaluation was conducted in the form of a web-based usability test (usability walkthrough combined with a web-based questionnaire) with participants aged between 26 and 63 years who work as medical doctors.

Results

In total, 17 study participants evaluated the three tools. The overall evaluation of usability, which was based on the System Usability Scale, showed that the Sample Locator, with a mean System Usability Scale score of 77.03 (SD 20.62), was significantly superior to the other two tools (Wilcoxon test; Sample Locator vs i2b2: P=.047; Sample Locator vs ATLAS: P=.001). i2b2, with a score of 59.83 (SD 25.36), performed significantly better than ATLAS, which had a score of 27.81 (SD 21.79; Wilcoxon test; i2b2 vs ATLAS: P=.005). The analysis of the material generated by the usability walkthrough method confirmed these findings. ATLAS caused the most usability problems (n=66), followed by i2b2 (n=48) and the Sample Locator (n=22). Moreover, the Sample Locator achieved the highest ratings with respect to additional questions regarding satisfaction with the tools.

Conclusions

This study provides data to develop a suitable basis for the selection of a harmonized tool for feasibility studies via concrete evaluation and a comparison of the usability of three different types of query builders. The feedback obtained from the participants during the usability test made it possible to identify user problems and positive design aspects of the individual tools and compare them qualitatively.

SUBMITTER: Schuttler C

PROVIDER: S-EPMC8339981 | biostudies-literature |

REPOSITORIES: biostudies-literature

ACCESS DATA

Similar Datasets

Project description:BACKGROUND:We explored the performance of three machine learning tools designed to facilitate title and abstract screening in systematic reviews (SRs) when used to (a) eliminate irrelevant records (automated simulation) and (b) complement the work of a single reviewer (semi-automated simulation). We evaluated user experiences for each tool. METHODS:We subjected three SRs to two retrospective screening simulations. In each tool (Abstrackr, DistillerSR, RobotAnalyst), we screened a 200-record training set and downloaded the predicted relevance of the remaining records. We calculated the proportion missed and workload and time savings compared to dual independent screening. To test user experiences, eight research staff tried each tool and completed a survey. RESULTS:Using Abstrackr, DistillerSR, and RobotAnalyst, respectively, the median (range) proportion missed was 5 (0 to 28) percent, 97 (96 to 100) percent, and 70 (23 to 100) percent for the automated simulation and 1 (0 to 2) percent, 2 (0 to 7) percent, and 2 (0 to 4) percent for the semi-automated simulation. The median (range) workload savings was 90 (82 to 93) percent, 99 (98 to 99) percent, and 85 (85 to 88) percent for the automated simulation and 40 (32 to 43) percent, 49 (48 to 49) percent, and 35 (34 to 38) percent for the semi-automated simulation. The median (range) time savings was 154 (91 to 183), 185 (95 to 201), and 157 (86 to 172) hours for the automated simulation and 61 (42 to 82), 92 (46 to 100), and 64 (37 to 71) hours for the semi-automated simulation. Abstrackr identified 33-90% of records missed by a single reviewer. RobotAnalyst performed less well and DistillerSR provided no relative advantage. User experiences depended on user friendliness, qualities of the user interface, features and functions, trustworthiness, ease and speed of obtaining predictions, and practicality of the export file(s). CONCLUSIONS:The workload savings afforded in the automated simulation came with increased risk of missing relevant records. Supplementing a single reviewer's decisions with relevance predictions (semi-automated simulation) sometimes reduced the proportion missed, but performance varied by tool and SR. Designing tools based on reviewers' self-identified preferences may improve their compatibility with present workflows. SYSTEMATIC REVIEW REGISTRATION:Not applicable.

Project description:BackgroundConnecting parents to research evidence is known to improve health decision making. However, guidance on how to develop effective knowledge translation (KT) tools that synthesize child-health evidence into a form understandable by parents is lacking.ObjectiveThe aim of this study was to conduct a comparative usability analysis of three Web-based KT tools to identify differences in tool effectiveness, identify which format parents prefer, and better understand what factors affect usability for parents.MethodsWe evaluated a Cochrane plain language summary (PLS), Blogshot, and a Wikipedia page on a specific child-health topic (acute otitis media). A mixed method approach was used involving a knowledge test, written usability questionnaire, and a semistructured interview. Differences in knowledge and usability questionnaire scores for each of the KT tools were analyzed using Kruskal-Wallis tests, considering a critical significance value of P=.05. Thematic analysis was used to synthesize and identify common parent preferences among the semistructured interviews. Key elements parents wanted in a KT tool were derived through author consensus using questionnaire data and parent interviews.ResultsIn total, 16 parents (9 female) with a mean age of 39.6 (SD 11.9) years completed the study. Parents preferred the Blogshot over the PLS and Wikipedia page (P=.002) and found the Blogshot to be the most aesthetic (P=.001) and easiest to use (P=.001). Knowledge questions and usability survey data also indicated that the Blogshot was the most preferred and effective KT tool at relaying information about the topic. Four key themes were derived from thematic analysis, describing elements parents valued in KT tools. Parents wanted tools that were (1) simple, (2) quick to access and use, and (3) trustworthy, and which (4) informed how to manage the condition. Out of the three KT tools assessed, Blogshots were the most preferred tool by parents and encompassed these four key elements.ConclusionsIt is important that child health evidence be available in formats accessible and understandable by parents to improve decision making, use of health care resources, and health outcomes. Further usability testing of different KT tools should be conducted involving broader populations and other conditions (eg, acute vs chronic) to generate guidelines to improve KT tools for parents.

Project description:BackgroundPatient-physician communication during clinical encounters is essential to ensure quality of care. Many studies have attempted to improve patient-physician communication. Incorporating patient priorities into agenda setting and medical decision-making are fundamental to patient-centered communication. Efficient and scalable approaches are needed to empower patients to speak up and prepare physicians to respond. Leveraging electronic health records (EHRs) in engaging patients and health care teams has the potential to enhance the integration of patient priorities in clinical encounters. A systematic approach to eliciting and documenting patient priorities before encounters could facilitate effective communication in such encounters.ObjectiveIn this paper, we report the design and implementation of a set of EHR tools built into clinical workflows for facilitating patient-physician joint agenda setting and the documentation of patient concerns in the EHRs for ambulatory encounters.MethodsWe engaged health information technology leaders and users in three health care systems for developing and implementing a set of EHR tools. The goal of these tools is to standardize the elicitation of patient priorities by using a previsit "patient important issue" questionnaire distributed through the patient portal to the EHR. We built additional EHR documentation tools to facilitate patient-staff communication when the staff records the vital signs and the reason for the visit in the EHR while in the examination room, with a simple transmission method for physicians to incorporate patient concerns in EHR notes.ResultsThe study is ongoing. The anticipated completion date for survey data collection is November 2021. A total of 34,037 primary care patients from three health systems (n=26,441; n=5136; and n=2460 separately recruited from each system) used the previsit patient important issue questionnaire in 2020. The adoption of the digital previsit questionnaire during the COVID-19 pandemic was much higher in one health care system because it expanded the use of the questionnaire from physicians participating in trials to all primary care providers midway through the year. It also required the use of this previsit questionnaire for eCheck-ins, which are required for telehealth encounters. Physicians and staff suggested anecdotally that this questionnaire helped patient-clinician communication, particularly during the COVID-19 pandemic.ConclusionsEHR tools have the potential to facilitate the integration of patient priorities into agenda setting and documentation in real-world primary care practices. Early results suggest the feasibility and acceptability of such digital tools in three health systems. EHR tools can support patient engagement and clinicians' work during in-person and telehealth visits. They could potentially exert a sustained influence on patient and clinician communication behaviors in contrast to prior ad hoc educational efforts targeting patients or clinicians.Trial registrationClinicalTrials.gov NCT03385512; https://clinicaltrials.gov/ct2/show/NCT03385512.International registered report identifier (irrid)DERR1-10.2196/30431.

Project description:BackgroundThe design complexity of critical care ventilators (CCVs) can lead to use errors and patient harm. In this study, we present the results of a comparison of four CCVs from market leaders, using a rigorous methodology for the evaluation of use safety and user experience of medical devices.MethodsWe carried out a comparative usability study of four CCVs: Hamilton G5, Puritan Bennett 980, Maquet SERVO-U, and Dräger Evita V500. Forty-eight critical care respiratory therapists participated in this fully counterbalanced, repeated measures study. Participants completed seven clinical scenarios composed of 16 tasks on each ventilator. Use safety was measured by percentage of tasks with use errors or close calls (UE/CCs). User experience was measured by system usability and workload metrics, using the Post-Study System Usability Questionnaire (PSSUQ) and the National Aeronautics and Space Administration Task Load Index (NASA-TLX).ResultsNine of 18 post hoc contrasts between pairs of ventilators were significant after Bonferroni correction, with effect sizes between 0.4 and 1.09 (Cohen's d). There were significantly fewer UE/CCs with SERVO-U when compared to G5 (p?=?0.044) and V500 (p?=?0.020). Participants reported higher system usability for G5 when compared to PB980 (p?=?0.035) and higher system usability for SERVO-U when compared to G5 (p?<?0.001), PB980 (p?<?0.001), and V500 (p?<?0.001). Participants reported lower workload for G5 when compared to PB980 (p?<?0.001) and lower workload for SERVO-U when compared to PB980 (p?<?0.001) and V500 (p?<?0.001). G5 scored better on two of nine possible comparisons; SERVO-U scored better on seven of nine possible comparisons. Aspects influencing participants' performance and perception include the low sensitivity of G5's touchscreen and the positive effect from the quality of SERVO-U's user interface design.ConclusionsThis study provides empirical evidence of how four ventilators from market leaders compare and highlights the importance of medical technology design. Within the boundaries of this study, we can infer that SERVO-U demonstrated the highest levels of use safety and user experience, followed by G5. Based on qualitative data, differences in outcomes could be explained by interaction design, quality of hardware components used in manufacturing, and influence of consumer product technology on users' expectations.

Project description:BackgroundPatients' spontaneous speech can act as a biomarker for identifying pathological entities, such as mental illness. Despite this potential, audio recording patients' spontaneous speech is not part of clinical workflows, and health care organizations often do not have dedicated policies regarding the audio recording of clinical encounters. No previous studies have investigated the best practical approach for integrating audio recording of patient-clinician encounters into clinical workflows, particularly in the home health care (HHC) setting.ObjectiveThis study aimed to evaluate the functionality and usability of several audio-recording devices for the audio recording of patient-nurse verbal communications in the HHC settings and elicit HHC stakeholder (patients and nurses) perspectives about the facilitators of and barriers to integrating audio recordings into clinical workflows.MethodsThis study was conducted at a large urban HHC agency located in New York, United States. We evaluated the usability and functionality of 7 audio-recording devices in a laboratory (controlled) setting. A total of 3 devices-Saramonic Blink500, Sony ICD-TX6, and Black Vox 365-were further evaluated in a clinical setting (patients' homes) by HHC nurses who completed the System Usability Scale questionnaire and participated in a short, structured interview to elicit feedback about each device. We also evaluated the accuracy of the automatic transcription of audio-recorded encounters for the 3 devices using the Amazon Web Service Transcribe. Word error rate was used to measure the accuracy of automated speech transcription. To understand the facilitators of and barriers to integrating audio recording of encounters into clinical workflows, we conducted semistructured interviews with 3 HHC nurses and 10 HHC patients. Thematic analysis was used to analyze the transcribed interviews.ResultsSaramonic Blink500 received the best overall evaluation score. The System Usability Scale score and word error rate for Saramonic Blink500 were 65% and 26%, respectively, and nurses found it easier to approach patients using this device than with the other 2 devices. Overall, patients found the process of audio recording to be satisfactory and convenient, with minimal impact on their communication with nurses. Although, in general, nurses also found the process easy to learn and satisfactory, they suggested that the audio recording of HHC encounters can affect their communication patterns. In addition, nurses were not aware of the potential to use audio-recorded encounters to improve health care services. Nurses also indicated that they would need to involve their managers to determine how audio recordings could be integrated into their clinical workflows and for any ongoing use of audio recordings during patient care management.ConclusionsThis study established the feasibility of audio recording HHC patient-nurse encounters. Training HHC nurses about the importance of the audio-recording process and the support of clinical managers are essential factors for successful implementation.

Project description:BackgroundViruses, including bacteriophages, are important components of environmental and human associated microbial communities. Viruses can act as extracellular reservoirs of bacterial genes, can mediate microbiome dynamics, and can influence the virulence of clinical pathogens. Various targeted metagenomic analysis techniques detect viral sequences, but these methods often exclude large and genome integrated viruses. In this study, we evaluate and compare the ability of nine state-of-the-art bioinformatic tools, including Vibrant, VirSorter, VirSorter2, VirFinder, DeepVirFinder, MetaPhinder, Kraken 2, Phybrid, and a BLAST search using identified proteins from the Earth Virome Pipeline to identify viral contiguous sequences (contigs) across simulated metagenomes with different read distributions, taxonomic compositions, and complexities.ResultsOf the tools tested in this study, VirSorter achieved the best F1 score while Vibrant had the highest average F1 score at predicting integrated prophages. Though less balanced in its precision and recall, Kraken2 had the highest average precision by a substantial margin. We introduced the machine learning tool, Phybrid, which demonstrated an improvement in average F1 score over tools such as MetaPhinder. The tool utilizes machine learning with both gene content and nucleotide features. The addition of nucleotide features improves the precision and recall compared to the gene content features alone.Viral identification by all tools was not impacted by underlying read distribution but did improve with contig length. Tool performance was inversely related to taxonomic complexity and varied by the phage host. For instance, Rhizobium and Enterococcus phages were identified consistently by the tools; whereas, Neisseria prophage sequences were commonly missed in this study.ConclusionThis study benchmarked the performance of nine state-of-the-art bioinformatic tools to identify viral contigs across different simulation conditions. This study explored the ability of the tools to identify integrated prophage elements traditionally excluded from targeted sequencing approaches. Our comprehensive analysis of viral identification tools to assess their performance in a variety of situations provides valuable insights to viral researchers looking to mine viral elements from publicly available metagenomic data.

Project description:BackgroundThe Food and Drug Administration Center for Biologics Evaluation and Research (CBER) established the Biologics Effectiveness and Safety (BEST) Initiative with several objectives, including the expansion and enhancement of CBER's access to fit-for-purpose data sources, analytics, tools, and infrastructures to improve the understanding of patient experiences with conditions related to CBER-regulated products. Owing to existing challenges in data collection, especially for rare disease research, CBER recognized the need for a comprehensive platform where study coordinators can engage with study participants and design and deploy studies while patients or caregivers could enroll, consent, and securely participate as well.ObjectiveThis study aimed to increase awareness and describe the design, development, and novelty of the Survey of Health and Patient Experience (SHAPE) platform, its functionality and application, quality improvement efforts, open-source availability, and plans for enhancement.MethodsSHAPE is hosted in a Google Cloud environment and comprises 3 parts: the administrator application, participant app, and application programming interface. The administrator can build a study comprising a set of questionnaires and self-report entries through the app. Once the study is deployed, the participant can access the app, consent to the study, and complete its components. To build SHAPE to be scalable and flexible, we leveraged the open-source software development kit, Ionic Framework. This enabled the building and deploying of apps across platforms, including iOS, Android, and progressive web applications, from a single codebase by using standardized web technologies. SHAPE has been integrated with a leading Health Level 7 (HL7®) Fast Healthcare Interoperability Resources (FHIR®) application programming interface platform, 1upHealth, which allows participants to consent to 1-time data pull of their electronic health records. We used an agile-based process that engaged multiple stakeholders in SHAPE's design and development.ResultsSHAPE allows study coordinators to plan, develop, and deploy questionnaires to obtain important end points directly from patients or caregivers. Electronic health record integration enables access to patient health records, which can validate and enhance the accuracy of data-capture methods. The administrator can then download the study data into HL7® FHIR®-formatted JSON files. In this paper, we illustrate how study coordinators can use SHAPE to design patient-centered studies. We demonstrate its broad applicability through a hypothetical type 1 diabetes cohort study and an ongoing pilot study on metachromatic leukodystrophy to implement best practices for designing a regulatory-grade natural history study for rare diseases.ConclusionsSHAPE is an intuitive and comprehensive data-collection tool for a variety of clinical studies. Further customization of this versatile and scalable platform allows for multiple use cases. SHAPE can capture patient perspectives and clinical data, thereby providing regulators, clinicians, researchers, and patient advocacy organizations with data to inform drug development and improve patient outcomes.