Dataset Information

Evaluation of Validity Evidence for Personality, Emotional Intelligence, and Situational Judgment Tests to Identify Successful Residents.

ABSTRACT:

Importance

The ability to identify candidates who will thrive and successfully complete their residency is especially critical for general surgery programs.

Objective

To assess the extent to which 3 screening tools used extensively in industrial selection settings-emotional intelligence (EQ), personality profiles, and situational judgment tests (SJTs)-could identify successful surgery residents.

Design, setting, and participants

In this analysis, personality profiles, EQ assessments, and SJTs were administered from July through August 2015 to 51 postgraduate year 1 through 5 general surgery residents in a large general surgery residency program. Associations between these variables and residency performance were investigated through correlation and hierarchical regression analyses.

Interventions

Completion of EQ, personality profiles, and SJT assessments.

Main outcomes and measures

Performance in residency as measured by a comprehensive performance metric. A score of zero represented a resident whose performance was consistent with that of their respective cohort's performance; below zero, worse performance; and greater than zero, better performance.

Results

Of the 61 eligible residents, 51 (84%) chose to participate and 22 (43%) were women. US Medical Licensing Examination Step 1 (USMLE1), but not USMLE2, emerged as a significant factor (t2,49 = 1.98; β = 0.30; P = .03) associated with overall performance. Neither EQ facets nor overall EQ offered significant incremental validity over USMLE1 scores. Inclusion of the personality factors did not significantly alter the test statistic and did not explain any additional portion of the variance. By contrast, inclusion of SJT scores accounted for 15% more of the variance than USMLE1 scores alone, resulting in a total of 25% of the variance explained by both USMLE1 and SJT scores (F2,57 = 7.47; P = .002). Both USMLE1 (t = 2.21; P = .03) and SJT scores (t = 2.97; P = .005) were significantly associated with overall resident performance.

Conclusions and relevance

This study found little support for the use of EQ assessment and only weak support for some distinct personality factors (ie, agreeableness, extraversion, and independence) in surgery resident selection. Performance on the SJT was associated with overall resident performance more than traditional cognitive measures (ie, USMLE scores). These data support further exploration of these 2 screening assessments on a larger scale across specialties and institutions.

SUBMITTER: Gardner AK

PROVIDER: S-EPMC6145666 | biostudies-literature |

REPOSITORIES: biostudies-literature

ACCESS DATA

Similar Datasets

Project description:Teachers' social-emotional competence is considered important in order to master the social and emotional challenges inherent in their profession and to build positive teacher-student relationships. In turn, this is key to both teachers' occupational well-being and positive student development. Nonetheless, an instrument assessing the profession-specific knowledge and skills that teachers need to master the social and emotional demands in the classroom is still lacking. Therefore, we developed the Test of Regulation in and Understanding of Social Situations in Teaching (TRUST), which is a theory-based situational judgment test measuring teachers' knowledge about strategies for emotion regulation and relationship management in emotionally and socially challenging situations with students. Results from three studies (N = 166 in-service teachers, N = 73 in-service teachers, N = 107 pre-service teachers) showed satisfactory internal consistency for both the emotion regulation and relationship management subtests. Furthermore, confirmatory factor analyses supported the differentiation between the two facets of social-emotional competence. Regarding convergent validity, results from Study 3 revealed a positive association between the profession-specific TRUST and pre-service teachers' general emotional intelligence. Furthermore, small to moderate correlations with the Big Five personality traits provided evidence for the discriminant validity of TRUST. In Studies 1 and 2, we found evidence for a correlation with external criteria, that is, teachers with higher test scores reported providing more emotional support for students and having better teacher-student relationships. For teachers' occupational well-being, we found a link with symptoms of depersonalization and job satisfaction, but none for emotional exhaustion. We will discuss the use of TRUST in research, for the evaluation of interventions, in teacher education, and professional development and will illustrate ideas for enhancing the tool.

Project description:BackgroundSituational judgment tests (SJTs) are used in health sciences education to measure examinee knowledge using case-based scenarios. Despite their popularity, there is a significant gap in the validity research on the response process that demonstrates how SJTs measure their intended constructs. A model of SJT response processes has been proposed in the literature by Robert Ployhart; however, few studies have explored and expanded the factors. The purpose of this study was to describe the factors involved in cognitive processes that examinees use as they respond to SJT items in a health professions education context.MethodsThirty participants-15 student pharmacists and 15 practicing pharmacists-completed a 12-item SJT designed to measure empathy. Each participant engaged in a think-aloud interview while completing the SJT, followed by a cognitive interview probing their decision-making processes. Interviews were transcribed and independently coded by three researchers to identify salient factors that contributed to response processes.ResultsThe findings suggest SJT response processes include all four stages (comprehension, retrieval, judgment, and response selection) as initially proposed by Ployhart. The study showed factors from other published research were present, including job-specific knowledge and experiences, emotional intelligence, and test-taking. The study also identified new factors not yet described, including identifying a task objective in the scenario, assumptions about the scenario, perceptions about the scenario, and the setting of the item.ConclusionsThis study provides additional SJT validity evidence by exploring participants' response processes through cognitive and think-aloud interviews. It also confirmed the four-stage model previously described by Ployhart and identified new factors that may influence SJT response processes. This study contributes to the literature with an expanded SJT response process model in a health professions education context and offers an approach to evaluate SJT response processes in the future.