Dataset Information

Influence of Clinical Competency Committee Review Process on Summative Resident Assessment Decisions.

ABSTRACT:

Background?

Clinical Competency Committees (CCCs) are charged with making summative assessment decisions about residents.

Objective?

We explored how review processes CCC members utilize influence their decisions regarding residents' milestone levels and supervisory roles.

Methods?

We conducted a multisite longitudinal prospective observational cohort study at 14 pediatrics residency programs during academic year 2015-2016. Individual CCC members biannually reported characteristics of their review process and Accreditation Council for Graduate Medical Education milestone levels and recommended supervisory role categorizations assigned to residents. Relationships among characteristics of CCC member reviews, mean milestone levels, and supervisory role categorizations were analyzed using mixed-effects linear regression, reported as mean differences with 95% confidence intervals (CIs), and Bayesian mixed-effects ordinal regression, reported as odds ratios (ORs) and 95% credible intervals (CrIs).

Results?

A total of 155 CCC members participated. Members who provided milestones or other professional development feedback after CCC meetings assigned significantly lower mean milestone levels (mean 1.4 points; CI -2.2 to -0.6; P < .001) and were significantly less likely to recommend supervisory responsibility in any setting (OR = 0.23, CrI 0.05-0.83) compared with CCC members who did not. Members recommended less supervisory responsibility when they reviewed more residents (OR = 0.96, 95% CrI 0.94-0.99) and participated in more review cycles (OR = 0.22, 95% CrI 0.07-0.63).

Conclusions?

This study explored the association between characteristics of individual CCC member reviews and their summative assessment decisions about residents. Further study is needed to gain deeper understanding of factors influencing CCC members' summative assessment decisions.

SUBMITTER: Schumacher DJ

PROVIDER: S-EPMC6108376 | biostudies-literature | 2018 Aug

REPOSITORIES: biostudies-literature

ACCESS DATA

Publications

Influence of Clinical Competency Committee Review Process on Summative Resident Assessment Decisions.

Schumacher Daniel J DJ King Beth B Barnes Michelle M MM Elliott Sean P SP Gibbs Kathleen K McGreevy Jon F JF Del Rey Javier Gonzalez JG Sharma Tanvi T Michelson Catherine C Schwartz Alan A

Journal of graduate medical education 20180801 4

<h4>Background</h4>Clinical Competency Committees (CCCs) are charged with making summative assessment decisions about residents.<h4>Objective</h4>We explored how review processes CCC members utilize influence their decisions regarding residents' milestone levels and supervisory roles.<h4>Methods</h4>We conducted a multisite longitudinal prospective observational cohort study at 14 pediatrics residency programs during academic year 2015-2016. Individual CCC members biannually reported characteris ...[more]

PMID: 30154975

Similar Datasets

Project description:BackgroundResidents receive a numeric performance rating (eg, 1-7 scoring scale) along with a narrative (ie, qualitative) feedback based on their performance in each workplace-based assessment (WBA). Aggregated qualitative data from WBA can be overwhelming to process and fairly adjudicate as part of a global decision about learner competence. Current approaches with qualitative data require a human rater to maintain attention and appropriately weigh various data inputs within the constraints of working memory before rendering a global judgment of performance.ObjectiveThis study explores natural language processing (NLP) and machine learning (ML) applications for identifying trainees at risk using a large WBA narrative comment data set associated with numerical ratings.MethodsNLP was performed retrospectively on a complete data set of narrative comments (ie, text-based feedback to residents based on their performance on a task) derived from WBAs completed by faculty members from multiple hospitals associated with a single, large, residency program at McMaster University, Canada. Narrative comments were vectorized to quantitative ratings using the bag-of-n-grams technique with 3 input types: unigram, bigrams, and trigrams. Supervised ML models using linear regression were trained with the quantitative ratings, performed binary classification, and output a prediction of whether a resident fell into the category of at risk or not at risk. Sensitivity, specificity, and accuracy metrics are reported.ResultsThe database comprised 7199 unique direct observation assessments, containing both narrative comments and a rating between 3 and 7 in imbalanced distribution (scores 3-5: 726 ratings; and scores 6-7: 4871 ratings). A total of 141 unique raters from 5 different hospitals and 45 unique residents participated over the course of 5 academic years. When comparing the 3 different input types for diagnosing if a trainee would be rated low (ie, 1-5) or high (ie, 6 or 7), our accuracy for trigrams was 87%, bigrams 86%, and unigrams 82%. We also found that all 3 input types had better prediction accuracy when using a bimodal cut (eg, lower or higher) compared with predicting performance along the full 7-point rating scale (50%-52%).ConclusionsThe ML models can accurately identify underperforming residents via narrative comments provided for WBAs. The words generated in WBAs can be a worthy data set to augment human decisions for educators tasked with processing large volumes of narrative assessments.

Project description:IntroductionA Clinical Ethics Committee (CEC) is a multi-professional service whose aim is to support healthcare professionals (HPs) and healthcare organisations to deal with the ethical issues of clinical practice.Although CEC are quite common worldwide, their successful implementation in a hospital setting presents many challenges.EVAluating a Clinical Ethics Committee implementation process (EvaCEC) will evaluate the implementation of a CEC in a comprehensive cancer centre in Northern Italy 16 months after its establishment.Methods and analysisEvaCEC is a mixed-method study with a retrospective quantitative analysis and a prospective qualitative evaluation by a range of data collection tools to enable the triangulation of data sources and analysis. Quantitative data related to the amount of CEC activities will be collected using the CEC's internal databases. Data on the level of knowledge, use and perception of the CEC will be collected through a survey with closed-ended questions disseminated among all the HPs employed at the healthcare centre. Data will be analysed with descriptive statistics.The Normalisation Process Theory (NPT) will be used for the qualitative evaluation to determine whether and how the CEC can be successfully integrated into clinical practice. We will perform one-to-one semistructured interviews and a second online survey with different groups of stakeholders who had different roles in the implementation process of the CEC. Based on NPT concepts, the interviews and the survey will assess the acceptability of the CEC within the local context and needs and expectations to further develop the service.Ethics and disseminationThe protocol has been approved by the local ethics committee. The project is co-chaired by a PhD candidate and by a healthcare researcher with a doctorate in bioethics and expertise in research. Findings will be disseminated widely through peer-reviewed publications, conferences and workshops.Trial registration numberNCT05466292.

Dataset Information

Influence of Clinical Competency Committee Review Process on Summative Resident Assessment Decisions.

Background?

Objective?

Methods?

Results?

Conclusions?

Publications

Influence of Clinical Competency Committee Review Process on Summative Resident Assessment Decisions.

Similar Datasets

OmicsDI is part of the ELIXIR infrastructure

Tweets