Dataset Information

The Ultrasound Competency Assessment Tool (UCAT): Development and Evaluation of a Novel Competency-based Assessment Tool for Point-of-care Ultrasound.

ABSTRACT:

Objectives

Point-of-care ultrasound (POCUS) has become an integral diagnostic and interventional tool. Barriers to POCUS training persist, and it continues to remain heterogeneous across training programs. Structured POCUS assessment tools exist, but remain limited in their feasibility, acceptability, reliability, and validity; none of these tools are entrustment-based. The objective of this study was to derive a simple, entrustment-based POCUS competency assessment tool and pilot it in an assessment setting.

Methods

This study was composed of two phases. First, a three-step modified Delphi design surveyed 60 members of the Canadian Association of Emergency Physicians Emergency Ultrasound Committee (EUC) to derive the anchors for the tool. Subsequently, the derived ultrasound competency assessment tool (UCAT) was used to assess trainee (N = 37) performance on a simulated FAST examination. The intraclass correlation (ICC) for inter-rater reliability and Cronbach's alpha for internal consistency were calculated. A statistical analysis was performed to compare the UCAT to other competency surrogates.

Results

The three-round Delphi had 22, 26, and 26 responses from the EUC members. Consensus was reached, and anchors for the domains of preparation, image acquisition, image optimization, and clinical integration achieved approval rates between 92 and 96%. The UCAT pilot revealed excellent inter-rater reliability (with ICC values of 0.69-0.89; p < 0.01) and high internal consistency (α = 0.91). While UCAT scores were not impacted by level of training, they were significantly impacted by the number of previous POCUS studies completed.

Conclusions

We developed and successfully piloted the UCAT, an entrustment-based bedside POCUS competency assessment tool suitable for rapid deployment. The findings from this study indicate early validity evidence for the use of the UCAT as an assessment of trainee POCUS competence on FAST. The UCAT should be trialed in different populations performing several POCUS study types.

SUBMITTER: Bell C

PROVIDER: S-EPMC8138101 | biostudies-literature |

REPOSITORIES: biostudies-literature

ACCESS DATA

Similar Datasets

Project description:ObjectivesImplementation barriers and lack of standardized point-of-care ultrasound (POCUS) curricula make the development of effective POCUS curricula and methods of assessment challenging. The authors aim to develop a longitudinal POCUS curriculum through staged intervention. In the first stage, the authors hypothesized that the use of high-fidelity ultrasound simulation during the Internal Medicine clerkship would improve POCUS confidence and knowledge among medical students, minimizing the need for trained faculty.MethodsA quasi-experimental study of third-year students on the Internal Medicine clerkship at a large academic medical center in the United States was performed assessing the efficacy of ultrasound simulation use. The control group consisted of students who received baseline POCUS education during teaching rounds but did not have access to the ultrasound simulator. The experimental group consisted of students who, in addition to baseline POCUS education, had access to a high-fidelity ultrasound simulator throughout the clerkship for a minimum of 1 hour per week. Students in both the control and experimental groups completed a pre- and post-intervention confidence survey and knowledge-based examination.ResultsEighty-two percent (50/61) of students completed pre- and post-tests, with the control group demonstrating no significant difference in POCUS confidence or knowledge. After exposure to the ultrasound simulator, the experimental group demonstrated statistically significant improvement in POCUS confidence and overall POCUS knowledge (p < .01).ConclusionThe use of high-fidelity ultrasound simulation can improve POCUS confidence and knowledge among medical students while addressing common barriers to the implementation of a POCUS curriculum. Despite showing statistically significant improvement in overall knowledge, the results did not appear to hold educational significance. Additional POCUS educational methods are necessary to overcome cognitive bias and potential overconfidence. The next stage of curriculum development will include resident-led POCUS workshops to supplement simulation.

Project description:ObjectivesEye-tracking technology has emerged as a potentially useful learner assessment tool in several medical specialties. In the fields of general surgery and anesthesiology, it has been shown to reliably differentiate between different levels of expertise in procedural skills. In the field of radiology, it has been shown to be a valid assessment tool for diagnostic test interpretation. Current methods of competency assessment in point-of-care ultrasound (POCUS) remain a challenge, because they require significant direct observation time by an instructor. The purpose of this study was to determine if eye-tracking technology can accurately distinguish between novice and experts in the interpretation of POCUS clips, specifically of the focused assessment using sonography in trauma (FAST) scan.MethodsA convenience sample of medical students, residents, and emergency physicians from a single academic emergency department were invited to participate. Participants included both novices and experts in POCUS. Each participant completed a baseline questionnaire and viewed 16 video clips of a FAST ultrasound examination while their gaze patterns were recorded by a commercially available eye-tracking device. The primary outcome was total gaze time on the area of interest (AOI). Secondary outcomes included total time to fixation, mean number of fixations, and mean duration of first fixation on the AOI.ResultsFifteen novices and 15 experts completed this study. For total gaze time on the AOI, experts fixated their gaze significantly longer than novices (75.8 ± 16.2 seconds vs. 56.6 ± 12.8 seconds, p = 0.001). Similarly, experts were significantly faster to fixate on the AOI and had a higher fixation count on the AOI (8.5 ± 4.0 seconds vs. 15.1 ± 6.8 seconds, p = 0.003; and 170 ± 30 vs. 143 ± 28 seconds, p = 0.016). There were no differences on the mean duration of first fixation on the AOI (0.42 ± 0.12 seconds vs. 0.39 ± 0.09 seconds, p = 0.467).ConclusionEye-tracking technology shows the potential to differentiate between experts and novices by their gaze patterns on video clips of FAST examinations. The total gaze time on the AOI may be a useful metric to help in the assessment of competency in POCUS image interpretation. In addition, the evaluation of gaze patterns may help educators identify causes of interpretation errors. Future studies are needed to further validate these metrics in a larger cohort.