Dataset Information

Does direct observation happen early in a new competency-based residency program?

ABSTRACT:

Background

A key component of competency-based medical education is workplace-based assessment, which includes observation (direct or indirect) of residents. Direct observation has been emphasized as an ideal form of assessment yet challenges have been identified that may limit its adoption. At present, it remains unclear how often direct and indirect observation are being used within the clinical setting. The objective of this study was to describe patterns of observation in an emergency medicine competency-based program 2 years postimplementation.

Methods

Emergency medicine residents (n = 19) recorded the type of observation they received (direct or indirect) following workplace-based entrustable professional activity (EPA) assessments from December 15, 2019, to April 30, 2020. Assessment forms were reviewed and analyzed to describe patters of observation.

Results

Assessments were collected on all 19 eligible residents (100% participation). A total of 1,070 EPA assessments were completed during the study period, of which 798 (74.6%) had the type of observation recorded. Of these recorded observations, 546 (68.4%) were directly observed and 252 (31.6%) were indirectly observed. The length of written comments contained within assessments following direct and indirect observation did not differ significantly. There was no significant association between resident gender and observation type or resident stage of training and observation type. Certain EPA assessments showed a clear preference toward either direct or indirect observation.

Conclusions

To the best of our knowledge, this study is the first to report patterns of observation in a competency-based residency program. The results suggest that direct observation can be quickly adopted as the primary means of workplace-based assessment. Indirect observation comprised a sizeable minority of observations and may be an underrecognized contributor to workplace-based assessment. The preference toward either direct or indirect observation for certain EPA assessments suggests that the entrustable professional activity itself may influence the type of observation.

SUBMITTER: Landreville JM

PROVIDER: S-EPMC8019151 | biostudies-literature | 2021 Apr

REPOSITORIES: biostudies-literature

ACCESS DATA

Publications

Does direct observation happen early in a new competency-based residency program?

Landreville Jeffrey M JM Frank Jason R JR Cheung Warren J WJ

AEM education and training 20210401 2

<h4>Background</h4>A key component of competency-based medical education is workplace-based assessment, which includes observation (direct or indirect) of residents. Direct observation has been emphasized as an ideal form of assessment yet challenges have been identified that may limit its adoption. At present, it remains unclear how often direct and indirect observation are being used within the clinical setting. The objective of this study was to describe patterns of observation in an emergenc ...[more]

PMID: 33842816

Similar Datasets

Project description:BackgroundThis article, developed for the Betty Ford Institute Consensus Conference on Graduate Medical Education (December, 2008), presents a model curriculum for Family Medicine residency training in substance abuse.MethodsThe authors reviewed reports of past Family Medicine curriculum development efforts, previously-identified barriers to education in high risk substance use, approaches to overcoming these barriers, and current training guidelines of the Accreditation Council for Graduate Medical Education (ACGME) and their Family Medicine Residency Review Committee. A proposed eight-module curriculum was developed, based on substance abuse competencies defined by Project MAINSTREAM and linked to core competencies defined by the ACGME. The curriculum provides basic training in high risk substance use to all residents, while also addressing current training challenges presented by U.S. work hour regulations, increasing international diversity of Family Medicine resident trainees, and emerging new primary care practice models.ResultsThis paper offers a core curriculum, focused on screening, brief intervention and referral to treatment, which can be adapted by residency programs to meet their individual needs. The curriculum encourages direct observation of residents to ensure that core skills are learned and trains residents with several "new skills" that will expand the basket of substance abuse services they will be equipped to provide as they enter practice.ConclusionsBroad-based implementation of a comprehensive Family Medicine residency curriculum should increase the ability of family physicians to provide basic substance abuse services in a primary care context. Such efforts should be coupled with faculty development initiatives which ensure that sufficient trained faculty are available to teach these concepts and with efforts by major Family Medicine organizations to implement and enforce residency requirements for substance abuse training.

Project description:BACKGROUND:Validated competency assessment tools and the data supporting milestone development during gastroscopy training are lacking. We aimed to assess the validity of the formative direct observation of procedural skills (DOPS) assessment tool in diagnostic gastroscopy and study competency development using DOPS. METHODS:This was a prospective multicentre (N = 275) analysis of formative gastroscopy DOPS assessments. Internal structure validity was tested using exploratory factor analysis and reliability estimated using generalisability theory. Item and global DOPS scores were stratified by lifetime procedure count to define learning curves, using a threshold determined from receiver operator characteristics (ROC) analysis. Multivariable binary logistic regression analysis was performed to identify independent predictors of DOPS competence. RESULTS:In total, 10086 DOPS were submitted for 987 trainees. Exploratory factor analysis identified three distinct item groupings, representing 'pre-procedure', 'technical', and 'post-procedure non-technical' skills. From generalisability analyses, sources of variance in overall DOPS scores included trainee ability (31%), assessor stringency (8%), assessor subjectivity (18%), and trainee case-to-case variation (43%). The combination of three assessments from three assessors was sufficient to achieve the reliability threshold of 0.70. On ROC analysis, a mean score of 3.9 provided optimal sensitivity and specificity for determining competency. This threshold was attained in the order of 'pre-procedure' (100-124 procedures), 'technical' (150-174 procedures), 'post-procedure non-technical' skills (200-224 procedures), and global competency (225-249 procedures). Higher lifetime procedure count, DOPS count, surgical trainees and assessors, higher trainee seniority, and lower case difficulty were significant multivariable predictors of DOPS competence. CONCLUSION:This study establishes milestones for competency acquisition during gastroscopy training and provides validity and reliability evidence to support gastroscopy DOPS as a competency assessment tool.

Dataset Information

Does direct observation happen early in a new competency-based residency program?

Background

Methods

Results

Conclusions

Publications

Does direct observation happen early in a new competency-based residency program?

Similar Datasets

OmicsDI is part of the ELIXIR infrastructure

Tweets