Dataset Information

Evaluation of a National Competency-Based Assessment System in Emergency Medicine: A CanDREAM Study.

ABSTRACT:

Background

In 2018, Canadian postgraduate emergency medicine (EM) programs began implementing a competency-based medical education (CBME) assessment program. Studies evaluating these programs have focused on broad outcomes using data from national bodies and lack data to support program-specific improvement.

Objective

We evaluated the implementation of a CBME assessment program within and across programs to identify successes and opportunities for improvement at the local and national levels.

Methods

Program-level data from the 2018 resident cohort were amalgamated and analyzed. The number of entrustable professional activity (EPA) assessments (overall and for each EPA) and the timing of resident promotion through program stages were compared between programs and to the guidelines provided by the national EM specialty committee. Total EPA observations from each program were correlated with the number of EM and pediatric EM rotations.

Results

Data from 15 of 17 (88%) programs containing 9842 EPA observations from 68 of 77 (88%) EM residents in the 2018 cohort were analyzed. Average numbers of EPAs observed per resident in each program varied from 92.5 to 229.6, correlating with the number of blocks spent on EM and pediatric EM (r = 0.83, P < .001). Relative to the specialty committee's guidelines, residents were promoted later than expected (eg, one-third of residents had a 2-month delay to promotion from the first to second stage) and with fewer EPA observations than suggested.

Conclusions

There was demonstrable variation in EPA-based assessment numbers and promotion timelines between programs and with national guidelines.

SUBMITTER: Thoma B

PROVIDER: S-EPMC7450748 | biostudies-literature |

REPOSITORIES: biostudies-literature

ACCESS DATA

Similar Datasets

Project description:ImportanceGender disparities exist throughout medicine. Recent studies have highlighted an attainment gap between male and female residents in performance evaluations on Accreditation Council for Graduate Medical Education (ACGME) milestones. Because of difficulties in blinding evaluators to gender, it remains unclear whether these observed disparities are because of implicit bias or other causes.ObjectiveTo estimate the magnitude of implicit gender bias in assessments of procedural competency in emergency medicine residents and whether the gender of the evaluator is associated with identified implicit gender bias.Design, setting, and participantsA cross-sectional study was performed from 2018 to 2020 in which emergency medicine residency faculty assessed procedural competency by evaluating videos of residents performing 3 procedures in a simulated environment. They were blinded to the intent of the study. Proceduralists were filmed performing each procedure from 2 different viewpoints simultaneously by 2 different cameras. One was a gender-blinded (ie, hands-only) view, and the other a wide-angled gender-evident (ie, whole-body) view. The faculty evaluators viewed videos in a random order and assessed procedural competency on a global rating scale with extensive validity evidence for the evaluation of video-recorded procedural performance.Main outcomes and measuresThe primary outcome was to determine if there was a difference in the evaluation of procedural competency based on gender. The secondary outcome was to determine if there was a difference in the evaluations based on the gender of the evaluator.ResultsFifty-one faculty evaluators enrolled from 19 states, with 22 male participants (43.1%), 29 female participants (56.9%), and a mean (SD) age of 37 (6.4) years. Each evaluator assessed all 60 procedures: 30 gender-blinded (hands-only view) videos and 30 identical gender-evident (wide angle) videos. There were no statistically significant differences in the study evaluators' scores of the proceduralists based on their gender, and the gender of the evaluator was not associated with the difference in mean scores.Conclusions and relevanceIn this study, we did not identify a difference in the evaluation of procedural competency based upon the gender of the resident proceduralist or the gender of the faculty evaluator.

Project description:ObjectivesIn 2016, a national consensus conference created the National Clinical Assessment Tool for Medical Students in Emergency Medicine (NCAT-EM), a standardized end-of-shift assessment tool. We report the first large-scale analysis of professionalism concerns collected from May 2017 through December 2018 by a multisite consortium using the NCAT-EM. Our primary objective was to characterize the nature and frequency of professionalism concerns. Our secondary objective was to identify characteristics associated with giving or receiving a professionalism flag.MethodsThe consortium database includes assessments for all students on EM clerkships at participating sites. This report presents descriptive statistics about the frequency of different flags, the distribution of flags among different student categories, assessor and student characteristics, and distribution of global assessment scores on assessments citing concerns. We used Fisher's exact test to look for associations between the frequency of professionalism flags and the sex of the students and assessors and across student categories. We used logistic regression to look for relationships between professionalism concerns and global assessment scores as well as intent to apply in EM.ResultsWe screened 6,768 assessments of 784 students by 719 assessors from 13 sites. After excluding assessments without flags and assessments with apparent data entry errors, we analyzed 57 (0.8%) assessments containing 79 flags. The most frequent flags were punctuality (25/79, 31.6%) and initiative (20/79, 25.3%). Few students received flags (42/784, 5.4%). Few assessors flagged concerns (41/719, 5.7%). We detected no correlation between the frequency of flags and whether a student was applying in EM or between the sex of students and assessors. Global scores of lower one-third appeared more often in assessments with a flag (30/57, 52.6% vs. 233/6,711, 3.5%).ConclusionsOnly 5.4% of students received flags. Punctuality and initiative accounted for a majority of citations. Professionalism flags correlated strongly with lower global assessment scores.

Project description:IntroductionRadiology training is an important component of emergency medicine (EM) education, but its delivery has been variable. Program directors have reported a lack of radiology skills in incoming interns. A needs assessment is a crucial first step toward improving radiology education among EM residencies. Our objective was to explore the current state of radiology education in EM residency programs.MethodsThis was a cross-sectional survey study of all Accreditation Council for Graduate Medical Education-accredited EM programs in the United States. Program leadership completed an online survey consisting of multiple choice, Likert scale, and free-response items. We calculated and reported descriptive statistics.ResultsOf eligible EM programs, 142/252 (56%) completed the survey including 105 postgraduate year (PGY) 1-3 and 36 PGY 1-4 programs. One respondent opted out of answering demographic questions. 23/141 (16%) were from the Western region, 29/141 (21%) were from the North Central region, 14/141 (10%) were from the South-Central region, 28/141 (20%) were from the Southeast region, and 47/141 (33%) were from the Northeast region. A total of 88/142 (62%) of responding programs did not have formal radiology instruction. Of the education that is provided, 127/142 (89%) provide it via didactics/lectures and 115/142 (81%) rely on instruction during clinical shifts. Only 51/142 (36%) provide asynchronous opportunities, and 23/142 (16%) have a dedicated radiology rotation. The majority of respondents reported spending 0-2 hours per month on radiology instruction (108/142; 76%); 95/141 (67%) reported that EM faculty "often" or "always" provide radiology instruction; 134/142 (95%), felt that it was "extremely" or "very important" for ED providers to be able to independently interpret radiograph results; and 129/142 (90.84%) either "sometimes" or "always" rely on their independent radiograph interpretations to make clinical decisions. The radiology studies identified as most important to be able to independently interpret were radiographs obtained for lines/tubes, chest radiographs, and radiographs obtained for musculoskeletal-related complaints.ConclusionA minority of EM residency programs have formal instruction in radiology despite the majority of responding program leadership believing that these are important skills. The most important curricular areas were identified. These results may inform the development of formal radiology curricula in EM graduate medical education.

Project description:IntroductionPediatric emergency medicine physicians struggle to maintain their critical procedural and resuscitation skills. Continuing professional development programs incorporating simulation and competency-based standards may help ensure skill maintenance. Using a logic model framework, we sought to evaluate the effectiveness of a mandatory annual competency-based medical education (CBME) simulation program.MethodsThe CBME program, evaluated from 2016 to 2018, targeted procedural, point-of-care ultrasound (POCUS) and resuscitation skills. Delivery of educational content included a flipped-classroom website, deliberate practice, mastery-based learning, and stop-pause debriefing. Participants' competence was assessed using a 5-point global rating scale (GRS; 3 = competent, 5 = mastery). Statistical process control charts were used to measure the effect of the CBME program on team performance during in situ simulations (ISS), measured using the Team Emergency Assessment Measure (TEAM) scale. Faculty completed an online program evaluation survey.ResultsForty physicians and 48 registered nurses completed at least one course over 3 years (physician mean ± SD 2.2 ± 0.92). Physicians achieved competence on 430 of 442 stations (97.3%). Mean ± SD GRS scores for procedural, POCUS, and resuscitation stations were 4.34 ± 0.43, 3.96 ± 0.35, and 4.17 ± 0.27, respectively. ISS TEAM scores for "followed standards and guidelines" improved significantly. No signals of special cause variation emerged for the other 11 TEAM items, indicating skills maintenance. Physicians rated CBME training as highly valuable (mean question scores 4.15-4.85/5). Time commitment and scheduling were identified as barriers to participation.ConclusionsOur mandatory simulation-based CBME program had high completion rates and very low station failures. The program was highly rated and faculty improved or maintained their ISS performance across TEAM scale domains.

Project description:ObjectivesPoint-of-care ultrasound (POCUS) has become an integral diagnostic and interventional tool. Barriers to POCUS training persist, and it continues to remain heterogeneous across training programs. Structured POCUS assessment tools exist, but remain limited in their feasibility, acceptability, reliability, and validity; none of these tools are entrustment-based. The objective of this study was to derive a simple, entrustment-based POCUS competency assessment tool and pilot it in an assessment setting.MethodsThis study was composed of two phases. First, a three-step modified Delphi design surveyed 60 members of the Canadian Association of Emergency Physicians Emergency Ultrasound Committee (EUC) to derive the anchors for the tool. Subsequently, the derived ultrasound competency assessment tool (UCAT) was used to assess trainee (N = 37) performance on a simulated FAST examination. The intraclass correlation (ICC) for inter-rater reliability and Cronbach's alpha for internal consistency were calculated. A statistical analysis was performed to compare the UCAT to other competency surrogates.ResultsThe three-round Delphi had 22, 26, and 26 responses from the EUC members. Consensus was reached, and anchors for the domains of preparation, image acquisition, image optimization, and clinical integration achieved approval rates between 92 and 96%. The UCAT pilot revealed excellent inter-rater reliability (with ICC values of 0.69-0.89; p < 0.01) and high internal consistency (α = 0.91). While UCAT scores were not impacted by level of training, they were significantly impacted by the number of previous POCUS studies completed.ConclusionsWe developed and successfully piloted the UCAT, an entrustment-based bedside POCUS competency assessment tool suitable for rapid deployment. The findings from this study indicate early validity evidence for the use of the UCAT as an assessment of trainee POCUS competence on FAST. The UCAT should be trialed in different populations performing several POCUS study types.

Project description:Background:Assessment of trainees' competency is challenging; the predictive power of traditional evaluations is debatable especially in regard to noncognitive traits. New assessments need to be sought to better understand affective areas like personality. Grit, defined as "perseverance and passion for long-term goals," can assess aspects of personality. Grit predicts educational attainment and burnout rates in other populations and is accurate with an informant report version. Self-assessments, while useful, have inherent limitations. Faculty's ability to accurately assess trainees' grit could prove helpful in identifying learner needs and avenues for further development. Objective:This study sought to determine the correlation between EM resident self-assessed and faculty-assessed Grit Scale (Grit-S) scores of that same resident. Methods:Subjects were PGY-1 to -4 EM residents and resident-selected faculty as part of a larger multicenter trial involving 10 EM residencies during 2017. The Grit-S Scale was administered to participating EM residents; an informant version was completed by their self-selected faculty. Correlation coefficients were computed to assess the relationship between residents' self-assessed and the residents' faculty-assessed Grit-S score. Results:A total of 281 of 303 residents completed the Grit-S, for a 93% response rate; 200 of 281 residents had at least one faculty-assessed Grit-S score. No correlation was found between residents' self-assessed and faculty-assessed Grit-S scores. There was a correlation between the two faculty-assessed Grit-S scores for the same resident. Conclusion:There was no correlation between resident and faculty-assessed Grit-S scores; additionally, faculty-assessed Grit-S scores of residents were higher. This corroborates the challenges faculty face at accurately assessing aspects of residents they supervise. While faculty and resident Grit-S scores did not show significant concordance, grit may still be a useful predictive personality trait that could help shape future training.