Dataset Information

Objective structured assessment of technical competence in transthoracic echocardiography: a validity study in a standardised setting.

ABSTRACT:

Background

Competence in transthoracic echocardiography (TTE) is unrelated to traditional measures of TTE competence, such as duration of training and number of examinations performed. This study aims to explore aspects of validity of an instrument for structured assessment of echocardiographic technical skills.

Methods

The study included 45 physicians with three different clinical levels of echocardiography competence who all scanned the same healthy male following national guidelines. An expert in echocardiography (OG) evaluated all the recorded, de-identified TTE images blindly using the developed instrument for assessment of TTE technical skills. The instrument consisted of both a global rating scale and a procedure specific checklist. Two scores were calculated for each examination: A global rating score and a total checklist score. OG rated ten examinations twice for intra-rater reliability, and another expert rated the same ten examinations for inter-rater reliability. A small pilot study was then performed with focus on content validity. This pilot study included nine physicians who scanned three patients with different pathologies as well as different technical difficulties.

Results

Validity of the TTE technical skills assessment instrument was supported by a significant correlation found between level of expertise and both the global score (Spearman 0.76, p<0.0001) and the checklist score (Spearman 0.74, p<0.001). Both scores were able to distinguish between the three levels of competence that were represented in the physician group. Reliability was supported by acceptable inter- and intra-rater values. The pilot study showed a tendency to improved scores with increasing expertise levels, suggesting that the instrument could also be used when pathologies were present.

Conclusions

We designed and developed a structured assessment instrument of echocardiographic technical skills that showed evidence of validity in terms of high correlations between test scores on a normal person and the level of physician competence, as well as acceptable inter- and intra-rater reliability scores. Further studies should, however, be performed to determine the adequate number of assessments needed to ensure high content validity and reliability in a clinical setting.

SUBMITTER: Nielsen DG

PROVIDER: S-EPMC3621521 | biostudies-literature | 2013 Mar

REPOSITORIES: biostudies-literature

ACCESS DATA

Publications

Objective structured assessment of technical competence in transthoracic echocardiography: a validity study in a standardised setting.

Nielsen Dorte Guldbrand DG Gotzsche Ole O Eika Berit B

BMC medical education 20130328

<h4>Background</h4>Competence in transthoracic echocardiography (TTE) is unrelated to traditional measures of TTE competence, such as duration of training and number of examinations performed. This study aims to explore aspects of validity of an instrument for structured assessment of echocardiographic technical skills.<h4>Methods</h4>The study included 45 physicians with three different clinical levels of echocardiography competence who all scanned the same healthy male following national guide ...[more]

PMID: 23537204

Similar Datasets

Project description:ObjectiveAssessment of the quality of the final product (QFP) is critical in simulation training, such as the clock face suture (CFS) exercise that is used to assess trainees' needle handling and suturing accuracy. Objective Structured Assessment of Technical Skill (OSATS) scores are the gold standard for the evaluation of trainees. The aim was to investigate variability in the use of OSATS checklists and to evaluate a semi-automatic method of suture analysis vs. OSATS scores.MethodsDetails of 287 CFSs performed by trainees during Fundamentals in Vascular Surgery examinations were collected. All were rated according to a seven item OSATS checklist, including QFP score and an overall score by one or two expert surgeons immediately after completion. Interassessor variability was assessed for the CFS that were assessed by two assessors.In order to assess intra- and interassessor variability, 50 CFS pictures were chosen randomly and submitted to three expert surgeons to rate the QFP twice and to carry out a semi-automatic image analysis of each CFS and the estimated cumulative error (CE; mm) recorded. It was hypothesised that the CE correlates to OSATS checklist items or overall score. Variables were compared for correlation with OSATS results using a linear regression. A Pearson's test was used to confirm the proposed hypothesis.ResultsMean ± standard deviation overall score for the OSATS checklist was 20.61 ± 6.33. Inter- and intra-assessor correlation were statistically significant regarding OSATS checklist items. Both correlations presented a low coefficient of determination, indicating variability. The mean CE was 16.07 ± 4.84 mm, and the correlation between the QFP and CE was statistically significant, proving that CE is an objective metric by which to assess the QFP.ConclusionOSATS score demonstrated intra- and interassessor variability, although there was a significant correlation between scores. CE is an objective metric that is not subject to assessor subjectivity or interassessor variability and is correlated with the gold standard of evaluation.

Project description:IntroductionThe aim of this study was to develop a reliable objective structured assessment of technical skills (OSATS) score for linear-stapled, hand-sewn closure of enterotomy intestinal anastomoses (A-OSATS).Materials and methodsThe Delphi methodology was used to create a traditional and weighted A-OSATS score highlighting the more important steps for patient outcomes according to an international expert consensus. Minimally invasive novices, intermediates, and experts were asked to perform a minimally invasive linear-stapled intestinal anastomosis with hand-sewn closure of the enterotomy in a live animal model either laparoscopically or robot-assisted. Video recordings were scored by two blinded raters assessing intrarater and interrater reliability and discriminative abilities between novices (n = 8), intermediates (n = 24), and experts (n = 8).ResultsThe Delphi process included 18 international experts and was successfully completed after 4 rounds. A total of 4 relevant main steps as well as 15 substeps were identified and a definition of each substep was provided. A maximum of 75 points could be reached in the unweighted A-OSATS score and 170 points in the weighted A-OSATS score respectively. A total of 41 anastomoses were evaluated. Excellent intrarater (r = 0.807-0.988, p < 0.001) and interrater (intraclass correlation coefficient = 0.923-0.924, p < 0.001) reliability was demonstrated. Both versions of the A-OSATS correlated well with the general OSATS and discriminated between novices, intermediates, and experts defined by their OSATS global rating scale.ConclusionWith the weighted and unweighted A-OSATS score, we propose a new reliable standard to assess the creation of minimally invasive linear-stapled, hand-sewn anastomoses based on an international expert consensus. Validity evidence in live animal models is provided in this study. Future research should focus on assessing whether the weighted A-OSATS exceeds the predictive capabilities of patient outcomes of the unweighted A-OSATS and provide further validity evidence on using the score on different anastomotic techniques in humans.

Project description:PurposeRecently, some attempts have been made to integrate lung ultrasound (LUS) teaching into medical curricula. However, current education studies of LUS are extremely heterogeneous due to the lack of evidence-based guidelines on LUS education. In particular, the assessment of competencies is poorly standardized and mostly relies on non-validated scales. A new validated tool, the objective structured assessment of lung ultrasound skills (LUS-OSAUS), has the potential to overcome these limitations. Therefore, we adopted the LUS-OSAUS tool to assess the competencies of a group of LUS-trained undergraduates. Existing no prior practical applications of the LUS-OSAUS, our aim was to investigate the practical utility of this tool and its applicability in the evaluation of US-trained medical students.MethodsEight undergraduates (two males, six females) were enrolled on a voluntary basis to receive a theoretical and practical training in LUS. Once completed their training, each student performed an LUS examination on a different patient hospitalized for respiratory symptoms. The same eight patients were also scanned by a senior resident in emergency medicine for a comparison with students' results. Students and the senior resident were tested by an examiner using the LUS-OSAUS tool. We compared the scores obtained by operators in all areas of competence of the LUS-OSAUS, the total scores, and the time needed to complete the sonographic task.ResultsMedian students' score in the single items of the scale was significantly lower than the ones obtained by the senior resident (4.0 [3.3-5.0] vs. 5.0 [5.0-5.0]; p < 0.0001). Students scored significantly lower than the senior resident in each item, except for B-line identification, choice of the correct transducer, and suggested focused questions. Median total score was also lower for students compared to the senior resident (70.5 [61.0-74.8] vs. 84.0 [83.5-84.3] (p = 0.0116). Median time required to complete the examination was significantly higher for students (14.1 [12.8-16.1] vs. 4.7 [3.9-5.2] min, p = 0.0117).ConclusionsThe LUS-OSAUS tool allowed for a standardized and comprehensive assessment of student's competencies in lung ultrasound, and helped to discriminate their level of expertise from that of a more experienced operator. The scale also specifically tests the theoretical knowledge of trainees, thus making redundant the use of questionnaires designed for this purpose.

Dataset Information

Objective structured assessment of technical competence in transthoracic echocardiography: a validity study in a standardised setting.

Background

Methods

Results

Conclusions

Publications

Objective structured assessment of technical competence in transthoracic echocardiography: a validity study in a standardised setting.

Similar Datasets

OmicsDI is part of the ELIXIR infrastructure

Tweets