Dataset Information

Influence of response instructions and response format on applicant perceptions of a situational judgement test for medical school selection.

ABSTRACT:

Background

This study examined the influence of two Situational Judgement Test (SJT) design features (response instructions and response format) on applicant perceptions. Additionally, we investigated demographic subgroup differences in applicant perceptions of an SJT.

Methods

Medical school applicants (N?=?372) responded to an online survey on applicant perceptions, including a description and two example items of an SJT. Respondents randomly received one of four SJT versions (should do-rating, should do-pick-one, would do-rating, would do-pick-one). They rated overall favourability and items on four procedural justice factors (face validity, applicant differentiation, study relatedness and chance to perform) and ease-of-cheating. Additionally, applicant perceptions were compared for subgroups based on gender, ethnic background and first-generation university status.

Results

Applicants rated would-do instructions as easier to cheat than should-do instructions. Rating formats received more favourable judgements than pick-one formats on applicant differentiation, study-relatedness, chance to perform and ease of cheating. No significant main effect for demographic subgroup on applicant perceptions was found, but significant interaction effects showed that certain subgroups might have more pronounced preferences for certain SJT design features. Specifically, ethnic minority applicants - but not ethnic majority applicants - showed greater preference for should-do than would-do instructions. Additionally, first-generation university students - but not non-first-generation university students - were more favourable of rating formats than of pick-one formats.

Conclusions

Findings indicate that changing SJT design features may positively affect applicant perceptions by promoting procedural justice factors and reducing perceived ease of cheating and that response instructions and response format can increase the attractiveness of SJTs for minority applicants.

SUBMITTER: De Leng WE

PROVIDER: S-EPMC6258459 | biostudies-literature | 2018 Nov

REPOSITORIES: biostudies-literature

ACCESS DATA

Publications

Influence of response instructions and response format on applicant perceptions of a situational judgement test for medical school selection.

De Leng Wendy E WE Stegers-Jager Karen M KM Born Marise Ph MP Themmen Axel P N APN

BMC medical education 20181126 1

<h4>Background</h4>This study examined the influence of two Situational Judgement Test (SJT) design features (response instructions and response format) on applicant perceptions. Additionally, we investigated demographic subgroup differences in applicant perceptions of an SJT.<h4>Methods</h4>Medical school applicants (N = 372) responded to an online survey on applicant perceptions, including a description and two example items of an SJT. Respondents randomly received one of four SJT versions (sh ...[more]

PMID: 30477494

Similar Datasets

Project description:ContextDespite their widespread use in medical school selection, there remains a lack of clarity on exactly what situational judgement tests (SJTs) measure.ObjectivesWe aimed to develop an SJT that measures integrity by combining critical incident interviews (inductive approach) with an innovative deductive approach. The deductive approach guided the development of the SJT according to two established theoretical models, of which one was positively related to integrity (honesty-humility [HH]) and one was negatively related to integrity (cognitive distortions [CD]). The Integrity SJT covered desirable (HH-based) and undesirable (CD-based) response options. We examined the convergent and discriminant validity of the Integrity SJT and compared the validity of the HH-based and CD-based subscores.MethodsThe Integrity SJT was administered to 402 prospective applicants at a Dutch medical school. The Integrity SJT consisted of 57 scenarios, each followed by four response options, of which two represented HH facets and two represented CD categories. Three SJT scores were computed, including a total, an HH-based and a CD-based score. The validity of these scores was examined according to their relationships with external integrity-related measures (convergent validity) and self-efficacy (discriminant validity).ResultsThe three SJT scores correlated significantly with all integrity-related measures and not with self-efficacy, indicating convergent and discriminant validity. In addition, the CD-based SJT score correlated significantly more strongly than the HH-based SJT score with two of the four integrity-related measures.ConclusionsAn SJT that assesses the ability to correctly recognise CD-based response options as inappropriate (i.e. what one should not do) seems to have stronger convergent validity than an SJT that assesses the ability to correctly recognise HH-based response options as appropriate (i.e. what one should do). This finding might be explained by the larger consensus on what is considered inappropriate than on what is considered appropriate in a challenging situation. It may be promising to focus an SJT on the ability to recognise what one should not do.

Project description:ImportanceThe ophthalmology residency application process is critical for applicants and residency programs, and knowledge about the preferences of applicants would assist both groups in improving the process.ObjectiveTo evaluate the experiences and preferences of ophthalmology residency applicants.Design, setting, and participantsThis cross-sectional, nonvalidated survey was conducted online. All applicants to the Bascom Palmer Eye Institute ophthalmology residency program during the 2018-2019 application cycle were invited to complete the survey. Data collection occurred from April 1, 2019, to April 30, 2019.Main outcomes and measuresApplicant demographics, application submissions, interview experiences, financial considerations, match results, and suggestions for improvement of the application process.ResultsResponses were received from 185 applicants (36.4%), including 77 women (41.6%). A successful match into an ophthalmology residency was achieved by 172 respondents (93.0%). There was a mean (SD) US Medical Licensing Examination Step 1 score of 245.8 (13.3) points. Respondents applied to a mean (SD) of 76.4 (23.5) ophthalmology residency programs, received 14.0 (9.0) invitations to interview, and attended 10.3 (4.4) interviews. Choices regarding applications and interviews were based mostly on program reputation, location, and advisor recommendation. A usual lead time of at least 3 weeks between the invitation and interview was reported by 126 respondents (69.2%), which was reduced to 14 respondents (15.1%) when a wait-list was involved. The ophthalmology residency application process cost a mean (SD) of $5704 ($2831) per applicant. Respondents reported that they were most able to reduce costs through housing choices (hotel stays or similar arrangements) and least able to reduce costs by limiting the number of programs to which they applied or at which they interviewed.Conclusions and relevanceThe ophthalmology residency application process is complex and poses substantial challenges to applicants and residency programs. These findings suggest that many current applicants have difficulty selecting programs to apply to, and most respondents desired changes to the current system of interview invitations and scheduling.

Project description:BackgroundThis paper explores local community perceptions of a relatively new rural medical school. For the purposes of this paper, community engagement is conceptualized as involvement in planning, delivering, and evaluating the medical program. Although there are several reviews of patient involvement in medical curricula development, this study was designed to pilot an approach to exploring the perspectives of well members of the community in the transition of institutional policy on community engagement to one medical school.MethodsAn advertisement in the local newspaper invited volunteers to participate in a telephone interview about the new medical school. An independent researcher external to the medical school conducted the interviews using a topic guide. Audio recordings were not made, but detailed notes including verbatim statements were recorded. At least two research team members analyzed interview records for emergent themes. Human research ethics approval was obtained.ResultsTwelve interviews were conducted. Participants offered rich imaginings on the role of the school and expectations and opportunities for students. Most participants expressed strong and positive views, especially in addressing long-term health workforce issues. It was considered important that students live, mix, and study in the community. Some participants had very clear ideas about the need of the school to address specified needs, such as indigenous health, obesity, aging, drug and alcohol problems, teenage pregnancy, ethnic diversity, and working with people of low socioeconomic status.ConclusionThis study has initiated a dialogue with potential partners in the community, which can be built upon to shape the medical school's mission and contribution to the society it serves. The telephone interview approach and thematic analysis yielded valuable insights and is recommended for further studies. Our study was limited by its small study size and the single recruitment source. The community is a rich resource for medical education, but there is a dearth of literature on the perspectives of the community and its role in medical education.

Project description:BackgroundIncreasing access to general practice work experience placements for school students is a strategy for improving general practice recruitment, despite limited evidence and concerns surrounding equity of access to general practice experiences.AimsTo examine the association between undertaking general practice experience and the perceptions of general practice as an appealing future career among prospective medical applicants. To identify socioeconomic factors associated with obtaining general practice experience.Design & settingCross-sectional questionnaire study in the UK.MethodParticipants were UK residents aged ≥16 years and seriously considering applying to study medicine in 2019/2020. They were invited to take part via the University Clinical Aptitude Test (UCAT). Questionnaire data were analysed using a linear regression of general practice appeal on general practice experience, adjusting for career motivations and demographics, and a logistic regression of general practice experience on measures of social capital and demographics.ResultsOf 6391 responders, 4031 were in their last year of school. General practice experience predicted general practice appeal after adjusting for career motivation and demographics (b = 0.37, standard error [SE] = 0.06, P<0.00001). General practice experience was more common among students at private (odds ratio [OR] = 1.65, 95% confidence interval [CI] = 1.31 to 2.08, P<0.0001) or grammar schools (OR = 1.33, 95% CI = 1.02 to 1.72, P = 0.03) and in the highest socioeconomic group (OR = 1.62, 95% CI = 1.28 to 2.05, P<0.0001), and less likely among students of 'other' ethnicity (OR = 0.37, 95% CI = 0.20 to 0.67, P = 0.0011).ConclusionHaving general practice experience prior to medical school was associated with finding general practice appealing, which supports its utility in recruitment. Applicants from more deprived backgrounds were less likely to have had a general practice experience, possibly through lack of accessible opportunities.

Project description:AIMS: Tests with natural-scientific content are predictive of the success in the first semesters of medical studies. Some universities in the German speaking countries use the 'Test for medical studies' (TMS) for student selection. One of its test modules, namely "medical and scientific comprehension", measures the ability for deductive reasoning. In contrast, the Hamburg Assessment Test for Medicine, Natural Sciences (HAM-Nat) evaluates knowledge in natural sciences. In this study the predictive power of the HAM-Nat test will be compared to that of the NatDenk test, which is similar to the TMS module "medical and scientific comprehension" in content and structure. METHODS: 162 medical school beginners volunteered to complete either the HAM-Nat (N=77) or the NatDenk test (N=85) in 2007. Until spring 2011, 84.2% of these successfully completed the first part of the medical state examination in Hamburg. Via different logistic regression models we tested the predictive power of high school grade point average (GPA or "Abiturnote") and the test results (HAM-Nat and NatDenk) with regard to the study success criterion "first part of the medical state examination passed successfully up to the end of the 7(th) semester" (Success7Sem). The Odds Ratios (OR) for study success are reported. RESULTS: For both test groups a significant correlation existed between test results and study success (HAM-Nat: OR=2.07; NatDenk: OR=2.58). If both admission criteria are estimated in one model, the main effects (GPA: OR=2.45; test: OR=2.32) and their interaction effect (OR=1.80) are significant in the HAM-Nat test group, whereas in the NatDenk test group only the test result (OR=2.21) significantly contributes to the variance explained. CONCLUSIONS: On their own both HAM-Nat and NatDenk have predictive power for study success, but only the HAM-Nat explains additional variance if combined with GPA. The selection according to HAM-Nat and GPA has under the current circumstances of medical school selection (many good applicants and only a limited number of available spaces) the highest predictive power of all models.

Project description:Background:The Standardized Video Interview (SVI) was developed by the American Association of Medical Colleges to allow applicants to include objective data about professional behaviors and interpersonal and communication skills. Although the SVI pilot was administered to individuals applying to emergency medicine (EM) residency programs during the 2018 Electronic Residency Application Service (ERAS) cycle, little data have been published evaluating the applicant's perceptions. This survey aims to assess EM residency applicant attitudes toward the SVI. Methods:During the 2018 ERAS application season an anonymous survey was administered to interviewees at one urban Accreditation Council for Graduate Medical Education-approved EM residency. Respondents were asked questions regarding the production of their video interviews, thoughts regarding the additive value of the SVI, and individual demographic data such as ethnicity and sex. Participation was optional. Results:A total of 219 of 238 candidates completed the survey representing a 92% response rate. While the majority of applicants did not feel that their ethnicity impacted their application, 58.1% of those who did self-identified as African American or Asian. A total of 8.7% of respondents felt the SVI added information about their professional behaviors and 11% felt that it added information about interpersonal and communication skills. Only 2.8% of survey respondents felt the SVI should remain a portion of the ERAS application. Conclusions:Most respondents felt that the SVI was not an accurate representation of their interpersonal and communication skills or their professionalism and that it did not add value to their applications. While most cohorts were not concerned about bias regarding sex, ethnicity, sex, or age, a small subset felt that there was a potential for the SVI to bias the party reviewing their applications. Very few applicants felt the SVI should remain a part of the ERAS application. Applicant attitudes toward the SVI are largely negative and require further investigation prior to becoming a standard part of applicants' ERAS files.

Dataset Information

Influence of response instructions and response format on applicant perceptions of a situational judgement test for medical school selection.

Background

Methods

Results

Conclusions

Publications

Influence of response instructions and response format on applicant perceptions of a situational judgement test for medical school selection.

Similar Datasets

OmicsDI is part of the ELIXIR infrastructure