Dataset Information

Variability of residents' ratings of faculty's teaching performance measured by five- and seven-point response scales.

ABSTRACT: BACKGROUND:Medical faculty's teaching performance is often measured using residents' feedback, collected by questionnaires. Researchers extensively studied the psychometric qualities of resulting ratings. However, these studies rarely consider the number of response categories and its consequences for residents' ratings of faculty's teaching performance. We compared the variability of residents' ratings measured by five- and seven-point response scales. METHODS:This retrospective study used teaching performance data from Dutch anaesthesiology residency training programs. Questionnaires with five- and seven-point response scales from the extensively studied System for Evaluation of Teaching Qualities (SETQ) collected the ratings. We inspected ratings' variability by comparing standard deviations, interquartile ranges, and frequency (percentage) distributions. Relevant statistical tests were used to test differences in frequency distributions and teaching performance scores. RESULTS:We examined 3379 residents' ratings and 480 aggregated faculty scores. Residents used the additional response categories provided by the seven-point scale - especially those differentiating between positive performances. Residents' ratings and aggregated faculty scores showed a more even distribution on the seven-point scale compared to the five-point scale. Also, the seven-point scale showed a smaller ceiling effect. After rescaling, the mean scores and (most) standard deviations of ratings from both scales were comparable. CONCLUSIONS:Ratings from the seven-point scale were more evenly distributed and could potentially yield more nuanced, specific and user-friendly feedback. Still, both scales measured (almost) similar teaching performance outcomes. In teaching performance practice, residents and faculty members should discuss whether response scales fit their preferences and goals.

SUBMITTER: Debets MPM

PROVIDER: S-EPMC7510269 | biostudies-literature | 2020 Sep

REPOSITORIES: biostudies-literature

ACCESS DATA

Publications

Variability of residents' ratings of faculty's teaching performance measured by five- and seven-point response scales.

Debets Maarten P M MPM Scheepers Renée A RA Boerebach Benjamin C M BCM Arah Onyebuchi A OA Lombarts Kiki M J M H KMJMH

BMC medical education 20200922 1

<h4>Background</h4>Medical faculty's teaching performance is often measured using residents' feedback, collected by questionnaires. Researchers extensively studied the psychometric qualities of resulting ratings. However, these studies rarely consider the number of response categories and its consequences for residents' ratings of faculty's teaching performance. We compared the variability of residents' ratings measured by five- and seven-point response scales.<h4>Methods</h4>This retrospective ...[more]

PMID: 32962692

Similar Datasets

Project description:BackgroundTo understand teaching performance of individual faculty, the climate in which residents' learning takes place, the learning climate, may be important. There is emerging evidence that specific climates do predict specific outcomes. Until now, the effect of learning climate on the performance of the individual faculty who actually do the teaching was unknown.ObjectivesTHIS STUDY: (i) tested the hypothesis that a positive learning climate was associated with better teaching performance of individual faculty as evaluated by residents, and (ii) explored which dimensions of learning climate were associated with faculty's teaching performance.Methods and materialsWe conducted two cross-sectional questionnaire surveys amongst residents from 45 residency training programs and multiple specialties in 17 hospitals in the Netherlands. Residents evaluated the teaching performance of individual faculty using the robust System for Evaluating Teaching Qualities (SETQ) and evaluated the learning climate of residency programs using the Dutch Residency Educational Climate Test (D-RECT). The validated D-RECT questionnaire consisted of 11 subscales of learning climate. Main outcome measure was faculty's overall teaching (SETQ) score. We used multivariable adjusted linear mixed models to estimate the separate associations of overall learning climate and each of its subscales with faculty's teaching performance.ResultsIn total 451 residents completed 3569 SETQ evaluations of 502 faculty. Residents also evaluated the learning climate of 45 residency programs in 17 hospitals in the Netherlands. Overall learning climate was positively associated with faculty's teaching performance (regression coefficient 0.54, 95% confidence interval: 0.37 to 0.71; P<0.001). Three out of 11 learning climate subscales were substantially associated with better teaching performance: 'coaching and assessment', 'work is adapted to residents' competence', and 'formal education'.ConclusionsIndividual faculty's teaching performance evaluations are positively affected by better learning climate of residency programs.

Project description:BackgroundIt is uncommon for faculty development professionals to assess faculty attitudes towards their teaching responsibilities and their perceived obstacles to teaching effectiveness. The purposes of this study were (a) to document faculty attitudes and practices related to applying motivation principles, and (b) to identify the perceived contextual factors that may shape these attitudes and practices.MethodsA sequential explanatory mixed methods design was used. Faculty members (n = 272; 32% response rate) were surveyed about their responsibility for and application of the five motivational principles that are part of the MUSIC Model of Motivation: eMpowerment, Usefulness, Success, Interest, and Caring. Repeated measures ANOVAs and Student's t-tests were computed to detect differences. Subsequently, two focus groups of faculty members (n = 11) interpreted the survey results. We conducted a thematic analysis and used the focus group results to explain the survey results.ResultsFaculty rated their responsibilities for applying principles related to Usefulness, Interest, and Caring significantly higher than they did for Success and eMpowerment. Most faculty also reported that they actually applied Usefulness, Interest, and Caring strategies within the past year, whereas over half of the faculty applied Success strategies and about a third of faculty applied eMpowerment strategies. Focus group participants identified factors that affected their ability to apply eMpowerment strategies, (e.g., offering choices), including students lacking generic skills (e.g., critical thinking, problem-solving), a lack of confidence in their abilities to implement empowering strategies and meet the needs of students, passive students, and large lecture-type courses. Focus group participants cited obstacles to implementing Success strategies (e.g., providing feedback), including difficulty in providing feedback in large courses, lacking time and assistant teachers, limited knowledge of technologies, and lacking skills related to guiding effective student peer feedback.ConclusionsFaculty appear adequately prepared to implement some types of motivational strategies, but not others, in part due to contextual factors that can influence their attitudes and, ultimately, their application of these strategies. We discuss how these factors affect attitudes and application of motivational strategies and formulate suggestions based on the results.

Project description:BACKGROUND:Multisource feedback (MSF) is increasingly being used to assess trainee performance, with different assessor groups fulfilling a crucial role in utility of assessment data. However, in health professions education, research on assessor behaviors in MSF is limited. When assessing trainee performance in work settings, assessors use multidimensional conceptualizations of what constitutes effective performance, also called personal performance theories, to distinguish between various behaviors and sub competencies., This may not only explain assessor variability in Multi Source Feedback, but also result in differing acceptance (and use) of assessment data for developmental purposes. The purpose of this study was to explore performance theories of various assessor groups (residents and nurses) when assessing performance of residents. METHODS:A constructivist, inductive qualitative research approach and semi-structured interviews following MSF were used to explore performance theories of 14 nurses and 15 residents in the department of internal medicine at Aga Khan University (AKU). Inductive thematic content analysis of interview transcripts was used to identify and compare key dimensions in residents' and nurses' performance theories used in evaluation of resident performance. RESULTS:Seven major themes, reflecting key dimensions of assessors' performance theories, emerged from the qualitative data, namely; communication skills, patient care, accessibility, teamwork skills, responsibility, medical knowledge and professional attitude. There were considerable overlaps, but also meaningful differences in the performance theories of residents and the nurses, especially with respect to accessibility, teamwork and medical knowledge. CONCLUSION:Residents' and nurses' performance theories for assessing resident performance overlap to some extent, yet also show meaningful differences with respect to the performance dimensions they pay attention to or consider most important. In MSF, different assessor groups may therefore hold different performance theories, depending on their role. Our results further our understanding of assessor source effects in MSF. Implications of our findings are related to implementation of MSF, design of rating scales as well as interpretation and use of MSF data for selection and performance improvement.

Dataset Information

Variability of residents' ratings of faculty's teaching performance measured by five- and seven-point response scales.

Publications

Variability of residents' ratings of faculty's teaching performance measured by five- and seven-point response scales.

Similar Datasets

OmicsDI is part of the ELIXIR infrastructure