Dataset Information

Direct observation of procedural skills (DOPS) assessment in diagnostic gastroscopy: nationwide evidence of validity and competency development during training.

ABSTRACT: BACKGROUND:Validated competency assessment tools and the data supporting milestone development during gastroscopy training are lacking. We aimed to assess the validity of the formative direct observation of procedural skills (DOPS) assessment tool in diagnostic gastroscopy and study competency development using DOPS. METHODS:This was a prospective multicentre (N?=?275) analysis of formative gastroscopy DOPS assessments. Internal structure validity was tested using exploratory factor analysis and reliability estimated using generalisability theory. Item and global DOPS scores were stratified by lifetime procedure count to define learning curves, using a threshold determined from receiver operator characteristics (ROC) analysis. Multivariable binary logistic regression analysis was performed to identify independent predictors of DOPS competence. RESULTS:In total, 10086 DOPS were submitted for 987 trainees. Exploratory factor analysis identified three distinct item groupings, representing 'pre-procedure', 'technical', and 'post-procedure non-technical' skills. From generalisability analyses, sources of variance in overall DOPS scores included trainee ability (31%), assessor stringency (8%), assessor subjectivity (18%), and trainee case-to-case variation (43%). The combination of three assessments from three assessors was sufficient to achieve the reliability threshold of 0.70. On ROC analysis, a mean score of 3.9 provided optimal sensitivity and specificity for determining competency. This threshold was attained in the order of 'pre-procedure' (100-124 procedures), 'technical' (150-174 procedures), 'post-procedure non-technical' skills (200-224 procedures), and global competency (225-249 procedures). Higher lifetime procedure count, DOPS count, surgical trainees and assessors, higher trainee seniority, and lower case difficulty were significant multivariable predictors of DOPS competence. CONCLUSION:This study establishes milestones for competency acquisition during gastroscopy training and provides validity and reliability evidence to support gastroscopy DOPS as a competency assessment tool.

SUBMITTER: Siau K

PROVIDER: S-EPMC6946748 | biostudies-literature | 2020 Jan

REPOSITORIES: biostudies-literature

ACCESS DATA

Publications

Direct observation of procedural skills (DOPS) assessment in diagnostic gastroscopy: nationwide evidence of validity and competency development during training.

Siau Keith K Crossley James J Dunckley Paul P Johnson Gavin G Feeney Mark M Hawkes Neil D ND Beales Ian L P ILP

Surgical endoscopy 20190325 1

<h4>Background</h4>Validated competency assessment tools and the data supporting milestone development during gastroscopy training are lacking. We aimed to assess the validity of the formative direct observation of procedural skills (DOPS) assessment tool in diagnostic gastroscopy and study competency development using DOPS.<h4>Methods</h4>This was a prospective multicentre (N = 275) analysis of formative gastroscopy DOPS assessments. Internal structure validity was tested using exploratory fact ...[more]

PMID: 30911922

Similar Datasets

Project description:Sufficient teaching and assessing clinical skills in the undergraduate setting becomes more and more important. In a surgical skills-lab course at the Medical University of Innsbruck fourth year students were teached with DOPS (direct observation of procedural skills). We analyzed whether DOPS worked or not in this setting, which performance levels could be reached compared to tutor teaching (one tutor, 5 students) and which curricular side effects could be observed.In a prospective randomized trial in summer 2013 (April - June) four competence-level-based skills were teached in small groups during one week: surgical abdominal examination, urethral catheterization (phantom), rectal-digital examination (phantom), handling of central venous catheters. Group A was teached with DOPS, group B with a classical tutor system. Both groups underwent an OSCE (objective structured clinical examination) for assessment. 193 students were included in the study. Altogether 756 OSCE´s were carried out, 209 (27,6%) in the DOPS- and 547 (72,3%) in the tutor-group.Both groups reached high performance levels. In the first month there was a statistically significant difference (p<0,05) in performance of 95% positive OSCE items in the DOPS-group versus 88% in the tutor group. In the following months the performance rates showed no difference anymore and came to 90% in both groups. In practical skills the analysis revealed a high correspondence between positive DOPS (92,4%) and OSCE (90,8%) results.As shown by our data DOPS furnish high performance of clinical skills and work well in the undergraduate setting. Due to the high correspondence of DOPS and OSCE results DOPS should be considered as preferred assessment tool in a students skills-lab. The approximation of performance-rates within the months after initial superiority of DOPS could be explained by an interaction between DOPS and tutor system: DOPS elements seem to have improved tutoring and performance rates as well. DOPS in students 'skills-lab afford structured feedback and assessment without increased personnel and financial resources compared to classic small group training.In summary, this study shows that DOPS represent an efficient method in teaching clinical skills. Their effects on didactic culture reach beyond the positive influence of performance rates.

Project description:INTRODUCTION:Mini Clinical Evaluation Exercise (Mini-CEX) and Direct Observation of Procedural Skills (DOPS) are used as formative assessments worldwide. Since an up-to-date comprehensive synthesis of the educational impact of Mini-CEX and DOPS is lacking, we performed a systematic review. Moreover, as the educational impact might be influenced by characteristics of the setting in which Mini-CEX and DOPS take place or their implementation status, we additionally investigated these potential influences. METHODS:We searched Scopus, Web of Science, and Ovid, including All Ovid Journals, Embase, ERIC, Ovid MEDLINE(R), and PsycINFO, for original research articles investigating the educational impact of Mini-CEX and DOPS on undergraduate and postgraduate trainees from all health professions, published in English or German from 1995 to 2016. Educational impact was operationalized and classified using Barr's adaptation of Kirkpatrick's four-level model. Where applicable, outcomes were pooled in meta-analyses, separately for Mini-CEX and DOPS. To examine potential influences, we used Fisher's exact test for count data. RESULTS:We identified 26 articles demonstrating heterogeneous effects of Mini-CEX and DOPS on learners' reactions (Kirkpatrick Level 1) and positive effects of Mini-CEX and DOPS on trainees' performance (Kirkpatrick Level 2b; Mini-CEX: standardized mean difference (SMD) = 0.26, p = 0.014; DOPS: SMD = 3.33, p<0.001). No studies were found on higher Kirkpatrick levels. Regarding potential influences, we found two implementation characteristics, "quality" and "participant responsiveness", to be associated with the educational impact. CONCLUSIONS:Despite the limited evidence, the meta-analyses demonstrated positive effects of Mini-CEX and DOPS on trainee performance. Additionally, we revealed implementation characteristics to be associated with the educational impact. Hence, we assume that considering implementation characteristics could increase the educational impact of Mini-CEX and DOPS.

Project description:ObjectiveWhile short-term gains in performance of critical emergency procedures are demonstrated after simulation, long-term retention is relatively uncertain. Our objective was to determine whether simulation of critical emergency procedures promotes long-term retention of skills in nonsurgical physicians.MethodsWe searched multiple electronic databases using a peer-reviewed strategy. Eligible studies 1) were observational cohorts, quasi-experimental or randomized controlled trials; 2) assessed intubation, cricothyrotomy, pericardiocentesis, tube thoracostomy, or central line placement performance by nonsurgical physicians; 3) utilized any form of simulation; and 4) assessed skill performance immediately after and at ≥ 3 months after simulation. The primary outcome was skill performance at or above a preset performance benchmark at ≥ 3 months after simulation. Secondary outcomes included procedural skill performance at 3, 6, and ≥ 12 months after simulation.ResultsWe identified 1,712 citations, with 10 being eligible for inclusion. Methodologic quality was moderate with undefined primary outcomes; inadequate sample sizes; and use of nonstandardized, unvalidated tools. Three studies assessed performance to a specific performance benchmark. Two demonstrated maintenance of the minimum performance benchmark while two demonstrated significant skill decay. A significant decline in the mean performance scores from immediately after simulation to 3, 6, and ≥ 12 months after simulation was observed in four of four, three of four, and two of five studies, respectively. Scores remained significantly above baseline at 3, 6, and ≥ 12 months after simulation in three of four, three of four, and four of four studies, respectively.ConclusionThere were a limited number of studies examining the retention of critical skills after simulation training. While there was some evidence of skill retention after simulation, overall most studies demonstrated skill decline over time.

Project description:ObjectiveDespite procedural skills being recognized as an important component of medical school education, medical students are not confident in their ability to carry out a range of medical procedures. We conducted an institutional needs assessment and used the results to inform the creation of a procedure-based preclinical elective for first- and second-year students.MethodsWe surveyed second-, third-, and fourth-year medical students at Alpert Medical School as well as select program directors to guide selection of a list of procedures to be taught in the elective. We then created an extracurricular 10-week procedural skills course for preclerkship medical students utilizing a hands-on, flipped classroom practice model. Volunteer preceptors were recruited from the Department of Emergency Medicine to participate with a student-to-faculty ratio not exceeding 5:1. Knowledge and skill acquisition were assessed using a multiple-choice knowledge exam and 4-station practical exam, respectively. Pre- and post-course online surveys were used to assess self-perceived confidence for all procedures.ResultsWe implemented our procedural skills training course for first- and second-year medical students in the fall of 2015. Forty-four students applied for the first iteration of the course and 15 students were selected to participate. Fourteen students ultimately completed the elective as well as the subsequent course surveys, multiple-choice exam, and practical exam. Students who participated in the elective had increased levels of self-reported confidence at the conclusion of the elective and performed better on a practical exam and multiple-choice exam compared to students who participated in only the standard curriculum.ConclusionA longitudinal preclerkship procedural course early during medical school is a feasible method of teaching procedural skills to a cohort of learners. A number of adjustments could be made to the course in order to scale up and include a larger cohort of students at our own or another institution.

Project description:ObjectivesHospitalists are expected to be competent in performing bedside procedures, which are associated with significant morbidity and mortality. A national decline in procedures performed by hospitalists has prompted questions about their procedural competency. Additionally, though simulation-based mastery learning (SBML) has been shown to be effective among trainees whether this approach has enduring benefits for independent practitioners who already have experience is unknown. We aimed to assess the baseline procedural skill of hospitalists already credentialed to perform procedures. We hypothesised that simulation-based training of hospitalists would result in durable skill gains after several months.DesignProspective cohort study with pretraining and post-training measurements.SettingSingle, large, urban academic medical centre in the USA.ParticipantsTwenty-two out of 38 eligible participants defined as hospitalists working on teaching services where they would supervise trainees performing procedures.InterventionsOne-on-one, 60 min SBML of lumbar puncture (LP) and abdominal paracentesis (AP).Primary and secondary outcome measuresOur primary outcome was the percentage of hospitalists obtaining minimum passing scores (MPS) on LP and AP checklists; our secondary outcomes were average checklist scores and self-reported confidence.ResultsAt baseline, only 16% hospitalists met or exceeded the MPS for LP and 32% for AP. Immediately after SBML, 100% of hospitalists reached this threshold. Reassessment an average of 7 months later revealed that only 40% of hospitalists achieved the MPS. Confidence increased initially after training but declined over time.ConclusionsHospitalists may be performing invasive bedside procedures without demonstration of adequate skill. A single evidence-based training intervention was insufficient to sustain skills for the majority of hospitalists over a short period of time. More stringent practices for certifying hospitalists who perform risky procedures are warranted, as well as mechanisms to support skill maintenance, such as periodic simulation-based training and assessment.

Dataset Information

Direct observation of procedural skills (DOPS) assessment in diagnostic gastroscopy: nationwide evidence of validity and competency development during training.

Publications

Direct observation of procedural skills (DOPS) assessment in diagnostic gastroscopy: nationwide evidence of validity and competency development during training.

Similar Datasets

OmicsDI is part of the ELIXIR infrastructure

Tweets