Dataset Information

Task model-specific operator skill assessment in routine fetal ultrasound scanning.

ABSTRACT:

Purpose

For highly operator-dependent ultrasound scanning, skill assessment approaches evaluate operator competence given available data, such as acquired images and tracked probe movement. Operator skill level can be quantified by the completeness, speed, and precision of performing a clinical task, such as biometry. Such clinical tasks are increasingly becoming assisted or even replaced by automated machine learning models. In addition to measurement, operators need to be competent at the upstream task of acquiring images of sufficient quality. To provide computer assistance for this task requires a new definition of skill.

Methods

This paper focuses on the task of selecting ultrasound frames for biometry, for which operator skill is assessed by quantifying how well the tasks are performed with neural network-based frame classifiers. We first develop a frame classification model for each biometry task, using a novel label-efficient training strategy. Once these task models are trained, we propose a second task model-specific network to predict two skill assessment scores, based on the probability of identifying positive frames and accuracy of model classification.

Results

We present comprehensive results to demonstrate the efficacy of both the frame-classification and skill-assessment networks, using clinically acquired data from two biometry tasks for a total of 139 subjects, and compare the proposed skill assessment with metrics of operator experience.

Conclusion

Task model-specific skill assessment is feasible and can be predicted by the proposed neural networks, which provide objective assessment that is a stronger indicator of task model performance, compared to existing skill assessment methods.

SUBMITTER: Wang Y

PROVIDER: S-EPMC9307537 | biostudies-literature |

REPOSITORIES: biostudies-literature

ACCESS DATA

Similar Datasets

Project description:Need to assess the skill of ecosystem modelsAccelerated changes to global ecosystems call for holistic and integrated analyses of past, present and future states under various pressures to adequately understand current and projected future system states. Ecosystem models can inform management of human activities in a complex and changing environment, but are these models reliable? Ensuring that models are reliable for addressing management questions requires evaluating their skill in representing real-world processes and dynamics. Skill has been evaluated for just a limited set of some biophysical models. A range of skill assessment methods have been reviewed but skill assessment of full marine ecosystem models has not yet been attempted.Northeast us atlantis marine ecosystem modelWe assessed the skill of the Northeast U.S. (NEUS) Atlantis marine ecosystem model by comparing 10-year model forecasts with observed data. Model forecast performance was compared to that obtained from a 40-year hindcast. Multiple metrics (average absolute error, root mean squared error, modeling efficiency, and Spearman rank correlation), and a suite of time-series (species biomass, fisheries landings, and ecosystem indicators) were used to adequately measure model skill. Overall, the NEUS model performed above average and thus better than expected for the key species that had been the focus of the model tuning. Model forecast skill was comparable to the hindcast skill, showing that model performance does not degenerate in a 10-year forecast mode, an important characteristic for an end-to-end ecosystem model to be useful for strategic management purposes.Skill assessment is both possible and advisableWe identify best-practice approaches for end-to-end ecosystem model skill assessment that would improve both operational use of other ecosystem models and future model development. We show that it is possible to not only assess the skill of a complicated marine ecosystem model, but that it is necessary do so to instill confidence in model results and encourage their use for strategic management. Our methods are applicable to any type of predictive model, and should be considered for use in fields outside ecology (e.g. economics, climate change, and risk assessment).

Project description:BackgroundImplementation of a point of care ultrasound curricula is valuable, but optimal integration for internal medicine residency is unclear. The purpose of this study was to evaluate if a structured ultrasound curriculum vs. structured ultrasound curriculum plus supervised thoracic ultrasounds would improve internal medicine residents' skill and retention 6 and 12 months from baseline.MethodsWe conducted a randomized controlled study evaluating internal medical residents' skill retention of thoracic ultrasound using a structured curriculum (control, n = 14) vs. structured curriculum plus 20 supervised bedside thoracic ultrasounds (intervention, n = 14). We used a stratified randomization based on program year. All subjects attended a half-day course that included 5 lectures and hands-on sessions at baseline. Assessments included written and practical exams at baseline, immediately post-course and at 6 and 12 months. Scores are reported as a percentage for the number of correct responses/number of questions (range 0-100%). The Mann Whitney U and the Friedman tests were used for analyses.ResultsTwenty-eight residents were enrolled. Two subjects withdrew prior to the 6-month exams. Written exam scores for all subjects improved, baseline median (IQR) 60 (46.47 to 66.67) post-course 80 (65 to 86.67), 6-month 80 (66.67 to 86.67) and 12-month 86.67 (80 to 88.34), p = <0.001. All subjects practical exam scores median (IQR) significantly improved, baseline 18.18 (7.95 to 32.95), post-course 59.09 (45.45 to 70.45), 6 month 71.74 (60.87 to 82.61) and 12-month 76.09 (65.22 to 88.05), p = <0.001. Comparing the control group to the intervention group, there were statistically significant higher scores, median (IQR), in the intervention group on the practical exam at 6 months 63.05 (48.92 to 69.57) vs. 82.61(72.83 to89.13), p = <0.001.ConclusionIn this cohort, internal medicine residents participating in a structured thoracic ultrasound course plus 20-supervised ultrasounds achieved higher practical exam scores long-term compared to controls.

Project description:One dimensional Doppler Ultrasound (DUS) is a low cost method for fetal auscultation. However, accuracy of any metrics derived from the DUS signals depends on their quality, which relies heavily on operator skills. In low resource settings, where skill levels are sparse, it is important for the device to provide real time signal quality feedback to allow the re-recording of data. Retrospectively, signal quality assessment can help remove low quality recordings when processing large amounts of data. To this end, we proposed a novel template-based method, to assess DUS signal quality. Data used in this study were collected from 17 pregnant women using a low-cost transducer connected to a smart phone. Recordings were split into 1990 segments of 3.75 s duration, and hand labeled for quality by three independent annotators. The proposed template-based method uses Empirical Mode Decomposition (EMD) to allow detection of the fetal heart beats and segmentation into short, time-aligned temporal windows. Templates were derived for each 15 s window of the recordings. The DUS signal quality index (SQI) was calculated by correlating the segments in each window with the corresponding running template using four different pre-processing steps: (i) no additional preprocessing, (ii) linear resampling of each beat, (iii) dynamic time warping (DTW) of each beat and (iv) weighted DTW of each beat. The template-based SQIs were combined with additional features based on sample entropy and power spectral density. To assess the performance of the method, the dataset was split into training and test subsets. The training set was used to obtain the best combination of features for predicting the DUS quality using cross validation, and the test set was used to estimate the classification accuracy using bootstrap resampling. A median out of sample classification accuracy on the test set of 85.8% was found using three features; template-based SQI, sample entropy and the relative power in the 160 to 660 Hz range. The results suggest that the new automated method can reliably assess the DUS quality, thereby helping users to consistently record DUS signals with acceptable quality for fetal monitoring.