Dataset Information

Exploring Eye-tracking Technology as an Assessment Tool for Point-of-care Ultrasound Training.

ABSTRACT:

Objectives

Eye-tracking technology has emerged as a potentially useful learner assessment tool in several medical specialties. In the fields of general surgery and anesthesiology, it has been shown to reliably differentiate between different levels of expertise in procedural skills. In the field of radiology, it has been shown to be a valid assessment tool for diagnostic test interpretation. Current methods of competency assessment in point-of-care ultrasound (POCUS) remain a challenge, because they require significant direct observation time by an instructor. The purpose of this study was to determine if eye-tracking technology can accurately distinguish between novice and experts in the interpretation of POCUS clips, specifically of the focused assessment using sonography in trauma (FAST) scan.

Methods

A convenience sample of medical students, residents, and emergency physicians from a single academic emergency department were invited to participate. Participants included both novices and experts in POCUS. Each participant completed a baseline questionnaire and viewed 16 video clips of a FAST ultrasound examination while their gaze patterns were recorded by a commercially available eye-tracking device. The primary outcome was total gaze time on the area of interest (AOI). Secondary outcomes included total time to fixation, mean number of fixations, and mean duration of first fixation on the AOI.

Results

Fifteen novices and 15 experts completed this study. For total gaze time on the AOI, experts fixated their gaze significantly longer than novices (75.8 ± 16.2 seconds vs. 56.6 ± 12.8 seconds, p = 0.001). Similarly, experts were significantly faster to fixate on the AOI and had a higher fixation count on the AOI (8.5 ± 4.0 seconds vs. 15.1 ± 6.8 seconds, p = 0.003; and 170 ± 30 vs. 143 ± 28 seconds, p = 0.016). There were no differences on the mean duration of first fixation on the AOI (0.42 ± 0.12 seconds vs. 0.39 ± 0.09 seconds, p = 0.467).

Conclusion

Eye-tracking technology shows the potential to differentiate between experts and novices by their gaze patterns on video clips of FAST examinations. The total gaze time on the AOI may be a useful metric to help in the assessment of competency in POCUS image interpretation. In addition, the evaluation of gaze patterns may help educators identify causes of interpretation errors. Future studies are needed to further validate these metrics in a larger cohort.

SUBMITTER: Lee WF

PROVIDER: S-EPMC8052995 | biostudies-literature |

REPOSITORIES: biostudies-literature

ACCESS DATA

Similar Datasets

Project description:ObjectivesPoint-of-care ultrasound (POCUS) has become an integral diagnostic and interventional tool. Barriers to POCUS training persist, and it continues to remain heterogeneous across training programs. Structured POCUS assessment tools exist, but remain limited in their feasibility, acceptability, reliability, and validity; none of these tools are entrustment-based. The objective of this study was to derive a simple, entrustment-based POCUS competency assessment tool and pilot it in an assessment setting.MethodsThis study was composed of two phases. First, a three-step modified Delphi design surveyed 60 members of the Canadian Association of Emergency Physicians Emergency Ultrasound Committee (EUC) to derive the anchors for the tool. Subsequently, the derived ultrasound competency assessment tool (UCAT) was used to assess trainee (N = 37) performance on a simulated FAST examination. The intraclass correlation (ICC) for inter-rater reliability and Cronbach's alpha for internal consistency were calculated. A statistical analysis was performed to compare the UCAT to other competency surrogates.ResultsThe three-round Delphi had 22, 26, and 26 responses from the EUC members. Consensus was reached, and anchors for the domains of preparation, image acquisition, image optimization, and clinical integration achieved approval rates between 92 and 96%. The UCAT pilot revealed excellent inter-rater reliability (with ICC values of 0.69-0.89; p < 0.01) and high internal consistency (α = 0.91). While UCAT scores were not impacted by level of training, they were significantly impacted by the number of previous POCUS studies completed.ConclusionsWe developed and successfully piloted the UCAT, an entrustment-based bedside POCUS competency assessment tool suitable for rapid deployment. The findings from this study indicate early validity evidence for the use of the UCAT as an assessment of trainee POCUS competence on FAST. The UCAT should be trialed in different populations performing several POCUS study types.

Project description:BackgroundSignificant gaps currently exist in the Canadian internal medicine point-of-care ultrasound (POCUS) curriculum. From a learner's perspective, it remains unknown what key POCUS skills should be prioritized. This needs assessment study seeks to establish educational priorities for POCUS for internal medicine residents at five Canadian residency training programs.MethodsAll internal medicine trainees [postgraduate year (PGY) 1-5] from five internal medicine residency training programs in Canada (n?=?598) were invited to complete an online survey on 15 diagnostic POCUS applications, 9 bedside procedures, and 18 POCUS knowledge items. For POCUS applications and procedures, participants were asked how applicable they are to patient care in internal medicine and the participants' reported skills in those domains. Self-reported knowledge and skills were rated on a 5-point Likert scale, where 1?=?very poor and 5?=?very good. Applicability was rated, where 1?=?not at all applicable and 5?=?very applicable.ResultsA total of 253 of 598 residents (42%) participated in our study. Data from one centre (n?=?15) was removed because of low response rate (15%) and significant baseline differences between those trainees and the remaining participants. Of the remaining analyzable data from four training programs (n?=?238), participants reported highest applicability to internal medicine for the following applications and procedures: identifying ascites/free fluid [mean applicability score of 4.9?±?standard deviation (SD) 0.4]; gross left ventricular function (mean 4.8?±?SD 0.5) and pericardial effusion (mean 4.7?±?SD 0.5); thoracentesis (mean score 4.9?±?SD 0.3), central line insertion (mean 4.9?±?SD 0.3), and paracentesis (mean 4.9?±?SD 0.3), respectively. Overall reported knowledge/skills was low, with skill gaps being the highest for identifying deep vein thrombosis (mean gap 2.7?±?SD 1.1), right ventricular strain (mean 2.7?±?SD 1.1), and gross left ventricular function (mean 2.7?±?SD 1.0).ConclusionsMany POCUS applications and procedures were felt to be applicable to the practice of internal medicine. Significant skill gaps exist in the four Canadian training programs included in the study. POCUS curriculum development efforts should target training based on these perceived skill gaps.

Project description:ObjectivesThe primary objective was to survey pediatric emergency medicine (PEM) leaders and fellows regarding point-of-care ultrasound (POCUS) training in PEM fellowship programs, including teaching methods, training requirements, and applications taught. Secondary objectives were to compare fellows' and program leaders' perceptions of fellow POCUS competency and training barriers.MethodsThis was a cross-sectional survey of U.S. PEM fellows and fellowship program leaders of the 78 fellowship programs using two online group-specific surveys exploring five domains: program demographics; training strategies and requirements; perceived competency; barriers, strengths, and weaknesses of POCUS training; and POCUS satisfaction.ResultsEighty-three percent (65/78) of programs and 53% (298/558) of fellows responded. All participating PEM fellowship programs included POCUS training in their curriculum. Among the 65 programs, 97% of programs and 92% of programs utilized didactics and supervised scanning shifts as educational techniques, respectively. Sixty percent of programs integrated numerical benchmarks and 49% of programs incorporated real-time, hands-on demonstration as training requirements. Of the 19 POCUS applications deemed in the literature as core requirements for fellows, at least 75% of the 298 fellows reported training in 13 of those applications. Although less than half of fellows endorsed competency for identifying intussusception, ultrasound-guided pericardiocentesis, and transvaginal pregnancy evaluation, a higher proportion of leaders reported fellows as competent for these applications (40% vs. 68%, p ≤ 0.001; 21% vs. 39%, p = 0.003; and 21% vs. 43%, p ≤ 0.001). Forty-six percent of fellows endorsed a lack of PEM POCUS evidence as a training barrier compared to 31% of leaders (p = 0.02), and 39% of leaders endorsed a lack of local financial support as a training barrier compared to 23% of fellows (p = 0.01).ConclusionsAlthough most PEM fellowship programs provide POCUS training, there is variation in content and requirements. Training does conform to many of the expert recommended guidelines; however, there are some discrepancies and perceived barriers to POCUS training remain.

Project description:BackgroundLack of training is currently the most common barrier to implementation of point-of-care ultrasound (POCUS) use in clinical practice, and in-person POCUS continuing medical education (CME) courses have been paramount in improving this training gap. Due to travel restrictions and physical distancing requirements during the COVID-19 pandemic, most in-person POCUS training courses were cancelled. Though tele-ultrasound technology has existed for several years, use of tele-ultrasound technology to deliver hands-on training during a POCUS CME course has not been previously described.MethodsWe conducted a retrospective observational study comparing educational outcomes, course evaluations, and learner and faculty feedback from in-person versus tele-ultrasound POCUS courses. The same POCUS educational curriculum was delivered to learners by the two course formats. Data from the most recent pre-pandemic in-person course were compared to tele-ultrasound courses during the COVID-19 pandemic.ResultsPre- and post-course knowledge test scores of learners from the in-person (n = 88) and tele-ultrasound course (n = 52) were compared. Though mean pre-course knowledge test scores were higher among learners of the tele-ultrasound versus in-person course (78% vs. 71%; p = 0.001), there was no significant difference in the post-course test scores between learners of the two course formats (89% vs. 87%; p = 0.069). Both learners and faculty rated the tele-ultrasound course highly (4.6-5.0 on a 5-point scale) for effectiveness of virtual lectures, tele-ultrasound hands-on scanning sessions, and course administration. Faculty generally expressed less satisfaction with their ability to engage with learners, troubleshoot image acquisition, and provide feedback during the tele-ultrasound course but felt learners completed the tele-ultrasound course with a better basic POCUS skillset.ConclusionsCompared to a traditional in-person course, tele-ultrasound POCUS CME courses appeared to be as effective for improving POCUS knowledge post-course and fulfilling learning objectives. Our findings can serve as a roadmap for educators seeking guidance on development of a tele-ultrasound POCUS training course whose demand will likely persist beyond the COVID-19 pandemic.

Project description:BackgroundSimulators are an extensively utilized teaching tool in clinical settings. Simulation enables learners to practice and improve their skills in a safe and controlled environment before using these skills on patients. We evaluated the effect of a training session utilizing a novel intubation ultrasound simulator on the accuracy of provider detection of tracheal versus esophageal neonatal endotracheal tube (ETT) placement using point-of-care ultrasound (POCUS). We also investigated whether the time to POCUS image interpretation decreased with repeated simulator attempts.MethodsSixty neonatal health care providers participated in a three-hour simulator-based training session in the neonatal intensive care unit (NICU) of Aga Khan University Hospital (AKUH), Karachi, Pakistan. Participants included neonatologists, neonatal fellows, pediatric residents and senior nursing staff. The training utilized a novel low-cost simulator made with gelatin, water and psyllium fiber. Training consisted of a didactic session, practice with the simulator, and practice with intubated NICU patients. At the end of training, participants underwent an objective structured assessment of technical skills (OSATS) and ten rounds of simulator-based testing of their ability to use POCUS to differentiate between simulated tracheal and esophageal intubations.ResultsThe majority of the participants in the training had an average of 7.0 years (SD 4.9) of clinical experience. After controlling for gender, profession, years of practice and POCUS knowledge, linear mixed model and mixed effects logistic regression demonstrated marginal improvement in POCUS interpretation over repeated simulator testing. The mean time-to-interpretation decreased from 24.7 (SD 20.3) seconds for test 1 to 10.1 (SD 4.5) seconds for Test 10, p < 0.001. There was an average reduction of 1.3 s (β = - 1.3; 95% CI: - 1.66 to - 1.0) in time-to-interpretation with repeated simulator testing after adjusting for the covariates listed above.ConclusionWe found a three-hour simulator-based training session had a significant impact on technical skills and performance of neonatal health care providers in identification of ETT position using POCUS. Further research is needed to examine whether these skills are transferable to intubated newborns in various health settings.Trial registrationClinicalTrials.gov Identifier: NCT03533218 . Registered May 2018.