Dataset Information

Evaluating Simplified Web Interfaces of Risk Models for Clinical Use: Pilot Survey Study.

ABSTRACT:

Background

In this pilot study, we investigated sociotechnical factors that affect intention to use a simplified web model to support clinical decision making.

Objective

We investigated factors that are known to affect technology adoption using the unified theory of acceptance and use of technology (UTAUT2) model. The goal was to pilot and test a tool to better support complex clinical assessments.

Methods

Based on the results of a previously published work, we developed a web-based mobile user interface, WebModel, to allow users to work with regression equations and their predictions to evaluate the impact of various characteristics or treatments on key outcomes (eg, survival time) for chronic obstructive pulmonary disease. The WebModel provides a way to combat information overload and more easily compare treatment options. It limits the number of web forms presented to a user to between 1 and 20, rather than the dozens of detailed calculations typically required. The WebModel uses responsive design and can be used on multiple devices. To test the WebModel, we designed a questionnaire to probe the efficacy of the WebModel and assess the usability and usefulness of the system. The study was live for one month, and participants had access to it over that time. The questionnaire was administered online, and data from 674 clinical users who had access to the WebModel were captured. SPSS and R were used for statistical analysis.

Results

The regression model developed from UTAUT2 constructs was a fit. Specifically, five of the seven factors were significant positive coefficients in the regression: performance expectancy (β=.2730; t=7.994; P<.001), effort expectancy (β=.1473; t=3.870; P=.001), facilitating conditions (β=.1644; t=3.849; P<.001), hedonic motivation (β=.2321; t=3.991; P<.001), and habit (β=.2943; t=12.732). Social influence was not a significant factor, while price value had a significant negative influence on intention to use the WebModel.

Conclusions

Our results indicate that multiple influences impact positive response to the system, many of which relate to the efficiency of the interface to provide clear information. Although we found that the price value was a negative factor, it is possible this was due to the removal of health workers from purchasing decisions. Given that this was a pilot test, and that the system was not used in a clinical setting, we could not examine factors related to actual workflow, patient safety, or social influence. This study shows that the concept of a simplified WebModel could be effective and efficient in reducing information overload in complex clinical decision making. We recommend further study to test this in a clinical setting and gather qualitative data from users regarding the value of the tool in practice.

SUBMITTER: Beaubien L

PROVIDER: S-EPMC8325085 | biostudies-literature |

REPOSITORIES: biostudies-literature

ACCESS DATA

Similar Datasets

Project description:BackgroundNormalization Process Theory (NPT) can be used to explain implementation processes in health care relating to new technologies and complex interventions. This paper describes the processes by which we developed a simplified version of NPT for use by clinicians, managers, and policy makers, and which could be embedded in a web-enabled toolkit and on-line users manual.MethodsBetween 2006 and 2010 we undertook four tasks. (i) We presented NPT to potential and actual users in multiple workshops, seminars, and presentations. (ii) Using what we discovered from these meetings, we decided to create a simplified set of statements and explanations expressing core constructs of the theory (iii) We circulated these statements to a criterion sample of 60 researchers, clinicians and others, using SurveyMonkey to collect qualitative textual data about their criticisms of the statements. (iv) We then reconstructed the statements and explanations to meet users' criticisms, embedded them in a web-enabled toolkit, and beta tested this 'in the wild'.ResultsOn-line data collection was effective: over a four week period 50/60 participants responded using SurveyMonkey (40/60) or direct phone and email contact (10/60). An additional nine responses were received from people who had been sent the SurveyMonkey form by other respondents. Beta testing of the web enabled toolkit produced 13 responses, from 327 visits to http://www.normalizationprocess.org. Qualitative analysis of both sets of responses showed a high level of support for the statements but also showed that some statements poorly expressed their underlying constructs or overlapped with others. These were rewritten to take account of users' criticisms and then embedded in a web-enabled toolkit. As a result we were able translate the core constructs into a simplified set of statements that could be utilized by non-experts.ConclusionNormalization Process Theory has been developed through transparent procedures at each stage of its life. The theory has been shown to be sufficiently robust to merit formal testing. This project has provided a user friendly version of NPT that can be embedded in a web-enabled toolkit and used as a heuristic device to think through implementation and integration problems.

Project description:Immersive software tools are virtual environments designed to give their users an augmented view of real-world data and ways of manipulating that data. As virtual environments, every action users make while interacting with these tools can be carefully logged, as can the state of the software and the information it presents to the user, giving these actions context. This data provides a high-resolution lens through which dynamic cognitive and behavioral processes can be viewed. In this report, we describe new methods for the analysis and interpretation of such data, utilizing a novel implementation of the Beta Process Hidden Markov Model (BP-HMM) for analysis of software activity logs. We further report the results of a preliminary study designed to establish the validity of our modeling approach. A group of 20 participants were asked to play a simple computer game, instrumented to log every interaction with the interface. Participants had no previous experience with the game's functionality or rules, so the activity logs collected during their naïve interactions capture patterns of exploratory behavior and skill acquisition as they attempted to learn the rules of the game. Pre- and post-task questionnaires probed for self-reported styles of problem solving, as well as task engagement, difficulty, and workload. We jointly modeled the activity log sequences collected from all participants using the BP-HMM approach, identifying a global library of activity patterns representative of the collective behavior of all the participants. Analyses show systematic relationships between both pre- and post-task questionnaires, self-reported approaches to analytic problem solving, and metrics extracted from the BP-HMM decomposition. Overall, we find that this novel approach to decomposing unstructured behavioral data within software environments provides a sensible means for understanding how users learn to integrate software functionality for strategic task pursuit.

Project description:BackgroundInternet-based learning for health professional education is increasing. It offers advantages over traditional learning approaches, as it enables learning to be completed at a time convenient to the user and improves access where facilities are geographically disparate. We developed and implemented the Vancomycin Interactive (VI) e-learning tool to improve knowledge on the clinical use of the antibiotic vancomycin, which is commonly used for treatment of infections caused by methicillin-resistant Staphylococcus aureus (MRSA).ObjectiveThe aims of this study were to evaluate the effect of the VI e-learning tool on (1) survey knowledge scores and (2) clinical use of vancomycin among health professionals.MethodsWe conducted a comparative pre-post intervention study across the 14 hospitals of two health districts in New South Wales, Australia. A knowledge survey was completed by nurses, doctors, and pharmacists before and after release of a Web-based e-learning tool. Survey scores were compared with those obtained following traditional education in the form of an email intervention. Survey questions related to dosing, administration, and monitoring of vancomycin. Outcome measures were survey knowledge scores among the three health professional groups, vancomycin plasma trough levels, and vancomycin approvals recorded on a computerized clinical decision support system.ResultsSurvey response rates were low at 26.87% (577/2147) preintervention and 8.24% (177/2147) postintervention. The VI was associated with an increase in knowledge scores (maximum score=5) among nurses (median 2, IQR 1-2 to median 2, IQR 1-3; P<.001), but not among other professional groups. The comparator email intervention was associated with an increase in knowledge scores among doctors (median 3, IQR 2-4 to median 4, IQR 2-4; P=.04). Participants who referred to Web-based resources while completing the e-learning tool achieved higher overall scores than those who did not (P<.001). The e-learning tool was not shown to be significantly more effective than the comparator email in the clinical use of vancomycin, as measured by plasma levels within the therapeutic range.ConclusionsThe e-learning tool was associated with improved knowledge scores among nurses, whereas the comparator email was associated with improved scores among doctors. This implies that different strategies may be required for optimizing the effectiveness of education among different health professional groups. Low survey response rates limited conclusions regarding the tool's effectiveness. Improvements to design and evaluation methodology may increase the likelihood of a demonstrable effect from e-learning tools in the future.

Project description:BackgroundPositive attitude toward evidence-based practice (EBP) principles in healthcare education may be one of the first steps for motivating a healthcare professional student to later apply EBP principles in clinical decision-making. The objectives for this project were to pilot an international web-based survey of chiropractic students and to describe student attitudes, behaviors, and knowledge about EBP principles.MethodsWe used SurveyMonkey™ to develop our survey based on an existing questionnaire used to measure basic knowledge, skills and beliefs about EBP among allied healthcare professionals and CAM practitioners. We invited 26 chiropractic educational institutions teaching in English and accredited by official organizations to participate. Academic officials and registrars at participating institutions forwarded an invitation email and two reminders to students between July and September 2010. The invitation contained a link to the 38-item web-based questionnaire. Descriptive statistics were performed for analysis.ResultsFourteen institutions from Australia, Canada, US, Denmark and New Zealand participated. Among an estimated 7,142 student recipients of invitation letters, 674 participated in the survey for an estimated response rate of 9.4%. Most respondents reported having access to medical/healthcare literature through the internet, but only 11% read literature every week and 21% did not read literature at all. Respondents generally agreed that the use of research evidence in chiropractic was important. Although 76% of respondents found it easy to understand research evidence and 81% had some level of confidence assessing the general worth of research articles, 71% felt they needed more training in EBP to be able to apply evidence in chiropractic care. Respondents without previous training in research methods had lower confidence in assessing published papers. While more than 60% marked the correct answer for two knowledge items, the mean number of correct answers to the five knowledge questions was 1.3 (SD 0.9).ConclusionsAlthough it is feasible to conduct an international web survey of chiropractic students, significant stakeholder participation is important to improve response rates. Students had relatively positive attitudes toward EBP. However, participants felt they needed more training in EBP and based on the knowledge questions they may need further training about basic research concepts.