Dataset Information

Longitudinal Reliability of Milestones-Based Learning Trajectories in Family Medicine Residents.

ABSTRACT:

Importance

Longitudinal Milestones data reported to the Accreditation Council for Graduate Medical Education (ACGME) can be used to measure the developmental and educational progression of learners. Learning trajectories illustrate the pattern and rate at which learners acquire competencies toward unsupervised practice.

Objective

To investigate the reliability of learning trajectories and patterns of learning progression that can support meaningful intervention and remediation for residents.

Design, setting, and participants

This national retrospective cohort study included Milestones data from residents in family medicine, representing 6 semi-annual reporting periods from July 2016 to June 2019.

Interventions

Longitudinal formative assessment using the Milestones assessment system reported to the ACGME.

Main outcomes and measures

To estimate longitudinal consistency, growth rate reliability (GRR) and growth curve reliability (GCR) for 22 subcompetencies in the ACGME family medicine Milestones were used, incorporating clustering effects at the program level. Latent class growth curve models were used to examine longitudinal learning trajectories.

Results

This study included Milestones ratings from 3872 residents in 514 programs. The Milestones reporting system reliably differentiated individual longitudinal patterns for formative purposes (mean [SD] GRR, 0.63 [0.03]); there was also evidence of precision for model-based rates of change (mean [SD] GCR, 0.91 [0.02]). Milestones ratings increased significantly across training years and reporting periods (mean [SD] of 0.55 [0.04] Milestones units per reporting period; P < .001); patterns of developmental progress varied by subcompetency. There were 3 or 4 distinct patterns of learning trajectories for each of the 22 subcompetencies. For example, for the professionalism subcompetency, residents were classified to 4 groups of learning trajectories; during the 3-year family medicine training period, trajectories diverged further after postgraduate year (PGY) 1, indicating a potential remediation point between the end of PGY 1 and the beginning of PGY 2 for struggling learners, who represented 16% of learners (620 residents). Similar inferences for learning trajectories were found for practice-based learning and improvement, systems-based practice, and interpersonal and communication skills. Subcompetencies in medical knowledge and patient care demonstrated more consistent patterns of upward growth.

Conclusions and relevance

These findings suggest that the Milestones reporting system provides reliable longitudinal data for individualized tracking of progress in all subcompetencies. Learning trajectories with supporting reliability evidence could be used to understand residents' developmental progress and tailored for individualized learning plans and remediation.

SUBMITTER: Park YS

PROVIDER: S-EPMC8652607 | biostudies-literature |

REPOSITORIES: biostudies-literature

ACCESS DATA

Similar Datasets

Project description:IntroductionMedical education is moving toward a competency-based framework with a focus on assessment using the Accreditation Council for Graduate Medical Education Milestones. Assessment of individual competencies through milestones can be challenging. While competencies describe characteristics of the person, the entrustable professional activities (EPAs) concept refers to work-related activities. EPAs would not replace the milestones but would be linked to them, integrating these frameworks. Many core specialties have already defined EPAs for resident trainees, but EPAs have not yet been created for emergency medicine (EM). This paper describes the development of milestone-linked EPAs for EM.MethodsTen EM educators from across North America formed a consensus working group to draft EM EPAs, using a modified Glaser state-of-the-art approach. A reactor panel with EPA experts from the United States, Canada and the Netherlands was created, and an iterative process with multiple revisions was performed based on reactor panel input. Following this, the EPAs were sent to the Council of Residency Directors for EM (CORD-EM) listserv for additional feedback.ResultsThe product was 11 core EPAs that every trainee from every EM program should be able to perform independently by the time of graduation. Each EPA has associated knowledge, skills, attitudes and behaviors (KSAB), which are either milestones themselves or KSABs linked to individual milestones. We recognize that individual programs may have additional focus areas or work-based activities they want their trainees to achieve by graduation; therefore, programs are also encouraged to create additional program-specific EPAs.ConclusionThis set of 11 core, EM-resident EPAs can be used as an assessment tool by EM residency programs, allowing supervising physicians to document the multiple entrustment decisions they are already making during clinical shifts with trainees. The KSAB list within each EPA could assist supervisors in giving specific, actionable feedback to trainees and allow trainees to use this list as an assessment-for-learning tool. Linking each KSAB to individual EM milestones allows EPAs to directly inform milestone assessment for clinical competency committees. These EPAs serve as another option for workplace-based assessment, and are linked to the milestones to create an integrated framework.

Project description:Malnutrition is common, morbid, and often correctable, but subject to missed and delayed diagnosis. Better screening and prediction could improve clinical, functional, and economic outcomes. This study aimed to assess the predictability of malnutrition from longitudinal patient records, and the external generalizability of a predictive model. Predictive models were developed and validated on statewide emergency department (ED) and hospital admission databases for California, Florida and New York, including visits from October 1, 2015 to December 31, 2018. Visit features included patient demographics, diagnosis codes, and procedure categories. Models included long short-term memory (LSTM) recurrent neural networks trained on longitudinal trajectories, and gradient-boosted tree and logistic regression models trained on cross-sectional patient data. The dataset used for model training and internal validation (California and Florida) included 62,811 patient trajectories (266,951 visits). Test sets included 63,997 (California), 63,112 (Florida), and 62,472 (New York) trajectories, such that each cohort's composition was proportional to the prevalence of malnutrition in that state. Trajectories contained seven patient characteristics and up to 2,008 diagnosis categories. Area under the receiver-operating characteristic (AUROC) and precision-recall curves (AUPRC) were used to characterize prediction of first malnutrition diagnoses in the test sets. Data analysis was performed from September 2020 to May 2021. Between 4.0% (New York) and 6.2% (California) of patients received malnutrition diagnoses. The longitudinal LSTM model produced the most accurate predictions of malnutrition, with comparable predictive performance in California (AUROC 0.854, AUPRC 0.258), Florida (AUROC 0.869, AUPRC 0.234), and New York (AUROC 0.869, AUPRC 0.190). Deep learning models can reliably predict malnutrition from existing longitudinal patient records, with better predictive performance and lower data-collection requirements than existing instruments. This approach may facilitate early nutritional intervention via automated screening at the point of care.