Dataset Information

Developing a situational judgment test blueprint for assessing the non-cognitive skills of applicants to the University of Utah School of Medicine, the United States.

ABSTRACT:

Purpose

The situational judgment test (SJT) shows promise for assessing the non-cognitive skills of medical school applicants, but has only been used in Europe. Since the admissions processes and education levels of applicants to medical school are different in the United States and in Europe, it is necessary to obtain validity evidence of the SJT based on a sample of United States applicants.

Methods

Ninety SJT items were developed and Kane's validity framework was used to create a test blueprint. A total of 489 applicants selected for assessment/interview day at the University of Utah School of Medicine during the 2014-2015 admissions cycle completed one of five SJTs, which assessed professionalism, coping with pressure, communication, patient focus, and teamwork. Item difficulty, each item's discrimination index, internal consistency, and the categorization of items by two experts were used to create the test blueprint.

Results

The majority of item scores were within an acceptable range of difficulty, as measured by the difficulty index (0.50-0.85) and had fair to good discrimination. However, internal consistency was low for each domain, and 63% of items appeared to assess multiple domains. The concordance of categorization between the two educational experts ranged from 24% to 76% across the five domains.

Conclusion

The results of this study will help medical school admissions departments determine how to begin constructing a SJT. Further testing with a more representative sample is needed to determine if the SJT is a useful assessment tool for measuring the non-cognitive skills of medical school applicants.

SUBMITTER: Colbert-Getz JM

PROVIDER: S-EPMC4671183 | biostudies-literature | 2015

REPOSITORIES: biostudies-literature

ACCESS DATA

Publications

Developing a situational judgment test blueprint for assessing the non-cognitive skills of applicants to the University of Utah School of Medicine, the United States.

Colbert-Getz Jorie M JM Pippitt Karly K Chan Benjamin B

Journal of educational evaluation for health professions 20151031

<h4>Purpose</h4>The situational judgment test (SJT) shows promise for assessing the non-cognitive skills of medical school applicants, but has only been used in Europe. Since the admissions processes and education levels of applicants to medical school are different in the United States and in Europe, it is necessary to obtain validity evidence of the SJT based on a sample of United States applicants.<h4>Methods</h4>Ninety SJT items were developed and Kane's validity framework was used to create ...[more]

PMID: 26582629

Similar Datasets

Project description:BackgroundFor academic physicians, teaching represents an essential skill. The proliferation of educator training programs aimed at residents and medical students signals the increasing commitment of training programs to develop teaching skills in their trainees as early as possible. However, clinical fellowships represent an important opportunity to advance training as educators. In addition to enriching the pipeline of future teachers, developing fellows as teachers augments the training experience for more junior trainees and may impact patient care. Fellows' needs for programs to improve teaching skills have been largely unexplored.MethodsWe conducted a multi-institutional needs assessment of internal medicine (IM) subspecialty fellows to gauge interest in teaching and improvement of teaching skills. We surveyed IM subspecialty fellows at three academic medical centers about their access to fellow-as-teacher programs and other mechanisms to improve their teaching skills during fellowship. We also elicited their attitudes towards teaching and interest in training related to teaching skills.ResultsOne hundred eighty-three fellows representing 20 programs and nine different subspecialties responded to the survey (48% response rate). The majority of participants (67%) reported having no specific training focused on teaching skills and only 12% reported receiving regular feedback about their teaching during their fellowship. Seventy-nine percent of fellows anticipated teaching to be part of their careers, and 22% planned to participate in medical education scholarship. Fellows reported a strong interest in teaching and programs aimed at improving their teaching skills.ConclusionsThe majority of fellows reported a lack of mechanisms to advance their teaching skills as fellows, despite anticipating teaching to be an important aspect of their future careers and having strong interest in such programs. Our findings at three academic medical centers confirm a lost opportunity among subspecialty fellowships to accelerate teaching skills development for future educators.

Project description:The motivational and other factors used by medical students in making their career choices for specific medical specialities have been looked at in a number of studies in the literature. There are however few studies that assess the generic factors which make medicine itself of interest to medical students and to potential medical students. This study describes a novel questionnaire that assesses the interests and attractions of different aspects of medical practice in a varied range of medical scenarios, and relates them to demographic, academic, personality and learning style measures in a large group of individuals considering applying to medical school.A questionnaire study was conducted among those attending Medlink, a two-day conference for individuals considering applying to medical school for a career in medicine. The main outcome measure was the Medical Situations Questionnaire, in which individuals ranked the attraction of three different aspects of medical practise in each of nine detailed, realistic medical scenarios in a wide range of medical specialities. As well as requiring clear choices, the questionnaire was also designed so that all of the possible answers were attractive and positive, thereby helping to eliminate social demand characteristics. Factor analysis of the responses found four generic motivational dimensions, which we labelled Indispensability, Helping People, Respect and Science. Background factors assessed included sex, ethnicity, class, medical parents, GCSE academic achievement, the 'Big Five' personality factors, empathy, learning styles, and a social desirability scale.2867 individuals, broadly representative of applicants to medical schools, completed the questionnaire. The four generic motivational factors correlated with a range of background factors. These correlations were explored by multiple regression, and by path analysis, using LISREL to assess direct and indirect effects upon the factors. Helping People was particularly related to agreeableness; Indispensability to a strategic approach to learning; Respect to a surface approach to learning; and Science to openness to experience. Sex had many indirect influences upon generic motivations. Ethnic origin also had indirect influences via neuroticism and surface learning, and social class only had indirect influences via lower academic achievement. Coming from a medical family had no influence upon generic motivations.Generic motivations for medicine as a career can be assessed using the Medical Situations Questionnaire, without undue response bias due to demand characteristics. The validity of the motivational factors is suggested by the meaningful and interpretable correlations with background factors such as demographics, personality, and learning styles. Further development of the questionnaire is needed if it is to be used at an individual level, either for counselling or for student selection.

Project description:BACKGROUND:Sex and gender influence health and disease outcomes, therefore, doctors should be able to deliver gender-sensitive care. To train gender-sensitive doctors, relevant sex and gender differences have to be included in medical education. In order to develop appealing, relevant, and effective education for undergraduate medical students, education should be tailored to students' level and anticipated on their ideas and assumptions. Therefore, we wanted to answer the following research questions: 1. What do aspiring medical students want to learn about gender medicine?; 2. How would they like to learn about gender medicine?; and 3. What are their ideas and assumptions about sex and gender differences in health and disease? METHODS:We performed an explorative thematic document analysis of educational assignments made by successful applicants (n?=?50) during the selection procedure of their entry into medical school. To test aspirants' capacity for self-directed learning, students were asked to formulate their own study plan after they watched a video that resembled a future practical experience (a consultation with a patient). As the content of this video was gender-sensitive, the assignments of the successful applicants gave us the unique opportunity to examine aspiring medical students' views about gender medicine. RESULTS:Aspiring medical students were eager to start their training to become gender-sensitive doctors. They believed in better care for all patients and thought doctors should obtain gender competences during their medical training. Students preferred to start with acquiring basic biomedical knowledge about differences between men and women and continue their training by developing gender-sensitive communication skills in (simulated) practical settings. Students differed in their interpretation of the gender-sensitive video, some generalized potential differences to all men and all women. Teachers were considered as important role models in learning about gender medicine. CONCLUSIONS:We advise medical schools to teach gender medicine from the beginning of medical school, by focusing on sex differences first and adding gender related themes later on in the curriculum. As students may interpret gender-sensitive information differently, structurally embedding reflection on gender medicine with gender competent teachers is necessary.

Project description:BackgroundSelection into medical school is highly competitive with more applicants than places. Little is known about the preparation that applicants undertake for this high stakes process. The study aims to determine what preparatory activities applicants undertake and what difficulties they encounter for each stage of the application process to medical school and in particular what impact these have on the outcome.MethodsA cross-sectional survey of 1097 applicants who applied for a place in the University of Adelaide Medical School in 2007 and participated in the UMAT (Undergraduate Medicine and Health Sciences Admission Test) and oral assessment components of the selection process. The main outcome measures were an offer of an interview and offer of a place in the medical school and were analysed using logistic regression.ResultsThe odds of a successful outcome increased with each additional preparatory activity undertaken for the UMAT (odds ratio 1.22, 95% confidence interval 1.11 to 1.33; P < 0.001) and the oral assessment (1.36, 1.19 to 1.55; P < 0.001) stage of selection. The UMAT preparatory activities associated with the offer of an interview were attendance of a training course by a private organisation (1.75, 1.35 to 2.27: P < 0.001), use of online services of a private organisation (1.58, 1.23 to 2.04; P < 0.001), and familiarising oneself with the process (1.52, 1.15 to 2.00; p = 0.021). The oral assessment activities associated with an offer of a place included refining and learning a personal resume (9.73, 2.97 to 31.88; P < 0.001) and learning about the course structure (2.05, 1.29 to 3.26; P = 0.022).For the UMAT, applicants who found difficulties with learning for this type of test (0.47, 0.35 to 0.63: P < 0.001), with the timing of UMAT in terms of school exams (0.48, 0.5 to 0.66; P < 0.001) and with the inability to convey personal skills with the UMAT (0.67, 0.52 to 0.86; P = 0.026) were significantly less likely to be offered an interview.ConclusionsMedical schools make an enormous effort to undertake a selection process that is fair and equitable and which selects students most appropriate for medical school and the course they provide. Our results indicate that performance in the selection processes can be improved by training. However, if these preparatory activities may be limited to those who can access them, the playing field is not even and increasing equity of access to medical schools will not be achieved.

Project description:Teachers' social-emotional competence is considered important in order to master the social and emotional challenges inherent in their profession and to build positive teacher-student relationships. In turn, this is key to both teachers' occupational well-being and positive student development. Nonetheless, an instrument assessing the profession-specific knowledge and skills that teachers need to master the social and emotional demands in the classroom is still lacking. Therefore, we developed the Test of Regulation in and Understanding of Social Situations in Teaching (TRUST), which is a theory-based situational judgment test measuring teachers' knowledge about strategies for emotion regulation and relationship management in emotionally and socially challenging situations with students. Results from three studies (N = 166 in-service teachers, N = 73 in-service teachers, N = 107 pre-service teachers) showed satisfactory internal consistency for both the emotion regulation and relationship management subtests. Furthermore, confirmatory factor analyses supported the differentiation between the two facets of social-emotional competence. Regarding convergent validity, results from Study 3 revealed a positive association between the profession-specific TRUST and pre-service teachers' general emotional intelligence. Furthermore, small to moderate correlations with the Big Five personality traits provided evidence for the discriminant validity of TRUST. In Studies 1 and 2, we found evidence for a correlation with external criteria, that is, teachers with higher test scores reported providing more emotional support for students and having better teacher-student relationships. For teachers' occupational well-being, we found a link with symptoms of depersonalization and job satisfaction, but none for emotional exhaustion. We will discuss the use of TRUST in research, for the evaluation of interventions, in teacher education, and professional development and will illustrate ideas for enhancing the tool.

Dataset Information

Developing a situational judgment test blueprint for assessing the non-cognitive skills of applicants to the University of Utah School of Medicine, the United States.

Purpose

Methods

Results

Conclusion

Publications

Developing a situational judgment test blueprint for assessing the non-cognitive skills of applicants to the University of Utah School of Medicine, the United States.

Similar Datasets

OmicsDI is part of the ELIXIR infrastructure

Tweets