Dataset Information

Pilot evaluation of an objective structured assessment of technical skills tool for chest tube insertion.

ABSTRACT: Background: Chest tube insertion is a standard intervention for management of various injuries of the thorax. Efficient clinical training of this and similar bed-side procedures is equally demanded and improvable. Here, we propose a nouveau means of assessment and feedback using an Objective Structured Assessment of Technical Skills (OSATS) tool. The modified OSATS for chest drain insertion is evaluated in a pilot trial focusing on chest drain insertion. Methods: Participants in the pilot trial were medical students (3rd-6th year of studies, n=9), junior residents (1st-3rd post-graduate year, n=12), senior residents (4th-6th post-graduate year, n=14), and attending surgeons (n=6) from Heidelberg University. Chest drain insertions on a cadaveric porcine model were rated by experts with the modified OSATS score. Participants' performances were videotaped and subsequently rated by two remote experts (video rating). Primary aim was to assess criterion validity of the OSATS to distinguish experience levels. Results: Kruskal-Wallis test showed significant differences between means of scores between four groups stratified by previous experience in chest tube insertion (level 0: 22.1±3.2 vs. level 1: 26.8±2.8 vs. level 2: 35.4±2.2 vs. level 3: 41.0±2.0; p=0.002; p1,3=0.049, p0,3=0.005). However, if groups were stratified by formal professional level, no statistically significant distinction could be made using OSATS. Hence, the OSATS tool showed criterion validity for differentiation between experience levels. Conclusion: In the pilot study, the modified OSATS for chest tube insertion was apt to standardize expert rating and could be used to measure skill and to depict different experience levels. The OSATS will help facilitate training and assessment of chest drain insertion and could therefore improve surgical training for trauma situations. According to our data, the OSATS might be integrated into modern curricula.

SUBMITTER: Friedrich M

PROVIDER: S-EPMC6278238 | biostudies-other | 2018

REPOSITORIES: biostudies-other

ACCESS DATA

Publications

Pilot evaluation of an objective structured assessment of technical skills tool for chest tube insertion.

Friedrich Mirco M Ober Julian J Haubruck Patrick P Bergdolt Christian C Bruckner Thomas T Kowalewski Karl-Friedrich KF Kadmon Martina M Müller-Stich Beat-Peter BP Tanner Michael Christopher MC Nickel Felix F

GMS journal for medical education 20181115 4

<b>Background:</b> Chest tube insertion is a standard intervention for management of various injuries of the thorax. Efficient clinical training of this and similar bed-side procedures is equally demanded and improvable. Here, we propose a nouveau means of assessment and feedback using an Objective Structured Assessment of Technical Skills (OSATS) tool. The modified OSATS for chest drain insertion is evaluated in a pilot trial focusing on chest drain insertion. <b>Methods:</b> Participants in th ...[more]

PMID: 30539073

Similar Datasets

Project description:PurposeRecently, some attempts have been made to integrate lung ultrasound (LUS) teaching into medical curricula. However, current education studies of LUS are extremely heterogeneous due to the lack of evidence-based guidelines on LUS education. In particular, the assessment of competencies is poorly standardized and mostly relies on non-validated scales. A new validated tool, the objective structured assessment of lung ultrasound skills (LUS-OSAUS), has the potential to overcome these limitations. Therefore, we adopted the LUS-OSAUS tool to assess the competencies of a group of LUS-trained undergraduates. Existing no prior practical applications of the LUS-OSAUS, our aim was to investigate the practical utility of this tool and its applicability in the evaluation of US-trained medical students.MethodsEight undergraduates (two males, six females) were enrolled on a voluntary basis to receive a theoretical and practical training in LUS. Once completed their training, each student performed an LUS examination on a different patient hospitalized for respiratory symptoms. The same eight patients were also scanned by a senior resident in emergency medicine for a comparison with students' results. Students and the senior resident were tested by an examiner using the LUS-OSAUS tool. We compared the scores obtained by operators in all areas of competence of the LUS-OSAUS, the total scores, and the time needed to complete the sonographic task.ResultsMedian students' score in the single items of the scale was significantly lower than the ones obtained by the senior resident (4.0 [3.3-5.0] vs. 5.0 [5.0-5.0]; p < 0.0001). Students scored significantly lower than the senior resident in each item, except for B-line identification, choice of the correct transducer, and suggested focused questions. Median total score was also lower for students compared to the senior resident (70.5 [61.0-74.8] vs. 84.0 [83.5-84.3] (p = 0.0116). Median time required to complete the examination was significantly higher for students (14.1 [12.8-16.1] vs. 4.7 [3.9-5.2] min, p = 0.0117).ConclusionsThe LUS-OSAUS tool allowed for a standardized and comprehensive assessment of student's competencies in lung ultrasound, and helped to discriminate their level of expertise from that of a more experienced operator. The scale also specifically tests the theoretical knowledge of trainees, thus making redundant the use of questionnaires designed for this purpose.

Project description:ObjectiveAssessment of the quality of the final product (QFP) is critical in simulation training, such as the clock face suture (CFS) exercise that is used to assess trainees' needle handling and suturing accuracy. Objective Structured Assessment of Technical Skill (OSATS) scores are the gold standard for the evaluation of trainees. The aim was to investigate variability in the use of OSATS checklists and to evaluate a semi-automatic method of suture analysis vs. OSATS scores.MethodsDetails of 287 CFSs performed by trainees during Fundamentals in Vascular Surgery examinations were collected. All were rated according to a seven item OSATS checklist, including QFP score and an overall score by one or two expert surgeons immediately after completion. Interassessor variability was assessed for the CFS that were assessed by two assessors.In order to assess intra- and interassessor variability, 50 CFS pictures were chosen randomly and submitted to three expert surgeons to rate the QFP twice and to carry out a semi-automatic image analysis of each CFS and the estimated cumulative error (CE; mm) recorded. It was hypothesised that the CE correlates to OSATS checklist items or overall score. Variables were compared for correlation with OSATS results using a linear regression. A Pearson's test was used to confirm the proposed hypothesis.ResultsMean ± standard deviation overall score for the OSATS checklist was 20.61 ± 6.33. Inter- and intra-assessor correlation were statistically significant regarding OSATS checklist items. Both correlations presented a low coefficient of determination, indicating variability. The mean CE was 16.07 ± 4.84 mm, and the correlation between the QFP and CE was statistically significant, proving that CE is an objective metric by which to assess the QFP.ConclusionOSATS score demonstrated intra- and interassessor variability, although there was a significant correlation between scores. CE is an objective metric that is not subject to assessor subjectivity or interassessor variability and is correlated with the gold standard of evaluation.

Project description:Performance assessment in skills training is ideally based on objective, reliable, and clinically relevant indicators of success. The Objective Structured Assessment of Technical Skill (OSATS) is a reliable and valid tool that has been increasingly used in orthopaedic skills training. It uses a global rating approach to structure expert evaluation of technical skills with the experts working from a list of operative competencies that are each rated on a 5-point Likert scale anchored by behavioral descriptors. Given the observational nature of its scoring, the OSATS might not effectively assess the quality of surgical results.(1) Does OSATS scoring in an intraarticular fracture reduction training exercise correlate with the quality of the reduction? (2) Does OSATS scoring in a cadaveric extraarticular fracture fixation exercise correlate with the mechanical integrity of the fixation?Orthopaedic residents at the University of Iowa (six postgraduate year [PGY]-1s) and at the University of Minnesota (seven PGY-1s and eight PGY-2s) undertook a skills training exercise that involved reducing a simulated intraarticular fracture under fluoroscopic guidance. Iowa residents participated three times during 1 month, and Minnesota residents participated twice with 1 month between their two sessions. A fellowship-trained orthopaedic traumatologist rated each performance using a modified OSATS scoring scheme. The quality of the articular reduction obtained was then directly measured. Regression analysis was performed between OSATS scores and two metrics of articular reduction quality: articular surface deviation and estimated contact stress. Another skills training exercise involved fixing a simulated distal radius fracture in a cadaveric specimen. Thirty residents, distributed across four PGY classes (PGY-2 and PGY-3, n = 8 each; PGY-4 and PGY-5, n = 7 each), simultaneously completed the exercise at individual stations. One of three faculty hand surgeons independently scored each performance using a validated OSATS scoring system. The mechanical integrity of each fixation construct was then assessed in a materials testing machine. Regression analysis was performed between OSATS scores and two metrics of fixation integrity: stiffness and failure load.In the intraarticular fracture model, OSATS scores did not correlate with articular reduction quality (maximum surface deviations: R = 0.17, p = 0.25; maximum contact stress: R = 0.22, p = 0.13). Similarly in the cadaveric extraarticular fracture model, OSATS scores did not correlate with the integrity of the mechanical fixation (stiffness: R = 0.10, p = 0.60; failure load: R = 0.30, p = 0.10).OSATS scoring methods do not effectively assess the quality of the surgical result. Efforts must be made to incorporate assessment metrics that reflect the quality of the surgical result.New objective, reliable, and clinically relevant measures of the quality of the surgical result obtained by a trainee are urgently needed. For intraarticular fracture reduction and extraarticular fracture fixation, direct physical measurement of reduction quality and of mechanical integrity of fixation, respectively, meet this need.

Project description:IntroductionThe aim of this study was to develop a reliable objective structured assessment of technical skills (OSATS) score for linear-stapled, hand-sewn closure of enterotomy intestinal anastomoses (A-OSATS).Materials and methodsThe Delphi methodology was used to create a traditional and weighted A-OSATS score highlighting the more important steps for patient outcomes according to an international expert consensus. Minimally invasive novices, intermediates, and experts were asked to perform a minimally invasive linear-stapled intestinal anastomosis with hand-sewn closure of the enterotomy in a live animal model either laparoscopically or robot-assisted. Video recordings were scored by two blinded raters assessing intrarater and interrater reliability and discriminative abilities between novices (n = 8), intermediates (n = 24), and experts (n = 8).ResultsThe Delphi process included 18 international experts and was successfully completed after 4 rounds. A total of 4 relevant main steps as well as 15 substeps were identified and a definition of each substep was provided. A maximum of 75 points could be reached in the unweighted A-OSATS score and 170 points in the weighted A-OSATS score respectively. A total of 41 anastomoses were evaluated. Excellent intrarater (r = 0.807-0.988, p < 0.001) and interrater (intraclass correlation coefficient = 0.923-0.924, p < 0.001) reliability was demonstrated. Both versions of the A-OSATS correlated well with the general OSATS and discriminated between novices, intermediates, and experts defined by their OSATS global rating scale.ConclusionWith the weighted and unweighted A-OSATS score, we propose a new reliable standard to assess the creation of minimally invasive linear-stapled, hand-sewn anastomoses based on an international expert consensus. Validity evidence in live animal models is provided in this study. Future research should focus on assessing whether the weighted A-OSATS exceeds the predictive capabilities of patient outcomes of the unweighted A-OSATS and provide further validity evidence on using the score on different anastomotic techniques in humans.

Project description:BackgroundCompetence in transthoracic echocardiography (TTE) is unrelated to traditional measures of TTE competence, such as duration of training and number of examinations performed. This study aims to explore aspects of validity of an instrument for structured assessment of echocardiographic technical skills.MethodsThe study included 45 physicians with three different clinical levels of echocardiography competence who all scanned the same healthy male following national guidelines. An expert in echocardiography (OG) evaluated all the recorded, de-identified TTE images blindly using the developed instrument for assessment of TTE technical skills. The instrument consisted of both a global rating scale and a procedure specific checklist. Two scores were calculated for each examination: A global rating score and a total checklist score. OG rated ten examinations twice for intra-rater reliability, and another expert rated the same ten examinations for inter-rater reliability. A small pilot study was then performed with focus on content validity. This pilot study included nine physicians who scanned three patients with different pathologies as well as different technical difficulties.ResultsValidity of the TTE technical skills assessment instrument was supported by a significant correlation found between level of expertise and both the global score (Spearman 0.76, p<0.0001) and the checklist score (Spearman 0.74, p<0.001). Both scores were able to distinguish between the three levels of competence that were represented in the physician group. Reliability was supported by acceptable inter- and intra-rater values. The pilot study showed a tendency to improved scores with increasing expertise levels, suggesting that the instrument could also be used when pathologies were present.ConclusionsWe designed and developed a structured assessment instrument of echocardiographic technical skills that showed evidence of validity in terms of high correlations between test scores on a normal person and the level of physician competence, as well as acceptable inter- and intra-rater reliability scores. Further studies should, however, be performed to determine the adequate number of assessments needed to ensure high content validity and reliability in a clinical setting.

Project description:BackgroundObjective assessment of skills after training is essential for safe implementation of lung point-of-care ultrasound (POCUS). In low-and middle-income countries (LMIC) there is a need for assessment tools without onsite experts to scale up POCUS access. Our objective is to develop a web-based assessment tool and evaluate trainees across different countries and at different time points after initial lung POCUS training.MethodsWe adapted the objective and validated lung ultrasound score (LUS-OSAUS) to a web-based tool with quiz and practical skills test. Trainees were evaluated after a short (4-day) standardized lung POCUS training and were classified in distinct groups according to (i) their geographical location (Benin vs. South-Africa) and (ii) time elapsed since training (Benin 0 months vs. Benin 6 months). The Benin 6 months group had minimal continuous education. Skills test images were read by two blinded experts. We report the overall success rates and then compare these rates based on location and timing since training, using the Fischer's exact test.ResultsA total of 35 out of 43 participants completed the online LUS-OSAUS quiz and skills test. The overall success rate was 0.84 (95%CI 0.80-0.88), with lower success rates for "correct depth" 0.54 (0.37-0.71), "correct assessment of pleura" 0.63 (0.45-0.79) and "conclusion" 0.71 (0.54-0.85). There were no differences based on location, with respective rates of 0.86 (0.80-0.92) and 0.83 (0.75-0.91) (p-value = 0.125) for Benin and South Africa at 0 months, respectively. Similarly, there were no differences according to timing with success rates of 0.86 (0.80-0.92) and 0.82 (0.72-0.93) (p-value = 0.563) for Benin at 0 months and 6 months, respectively.ConclusionWeb-based objective and structured assessment of lung POCUS skills in LMIC following a short-standardized training is feasible and has a good overall success rate with consistent results across regions and up to 6 months after training given minimal continuous education. Overall, technical and POCUS-based clinical conclusion skills are the most difficult to acquire.

Dataset Information

Pilot evaluation of an objective structured assessment of technical skills tool for chest tube insertion.

Publications

Pilot evaluation of an objective structured assessment of technical skills tool for chest tube insertion.

Similar Datasets

OmicsDI is part of the ELIXIR infrastructure

Tweets