Dataset Information

Measuring competency-relevant knowledge in the competency-oriented student progress test.

ABSTRACT: Background: Since 2013 a competency-oriented student progress test (SKPT) has been administered at a number of German medical schools. The questions are generated on the basis of a two-dimensional blueprint, on which one axis contains the five competency domains - communicative competence (CO), practical clinical competence (CP), theoretical clinical competence (CT), scientific competence (SC), and professional decision-making competence (PR) - that form part of the competency model of the National Competency-based Catalogue of Learning Objectives for Undergraduate Medicine (NKLM). The feedback for students is structured in part according to these domains. The aim of our study is to examine if the results differentiated by competency domain show adequate measurement accuracy and if the results for the different domains also contain different information. Methods: The SKPTs for the years 2013 to 2017, taken by a total of 3,027 students, were examined. The measurement accuracy was determined using the coefficient glb (greatest bound to reliability) and the standard error of measurement; discriminant analysis of the principal components was carried out to demonstrate differentiation between the competency domains. Results: The reliability of the competency domains was above 0.8 for all SKPTs; exceptions to this were seen in two of the tests for CO and PR that had a reliability of 0.7-0.8. The results for all of the individual competency domains differed in their informational content compared to the overall of the other domains; the same applies for all pairwise comparisons, with the exceptions of CP and CT. Discussion: The SKPT feedback for students that is differentiated by competency domains basically fulfills the requirements for measurement reliability and distinctness. An improvement of the measurement quality for CO and PR and a better differentiation between CP and CT is desirable.

SUBMITTER: Moltner A

PROVIDER: S-EPMC7105759 | biostudies-literature | 2020

REPOSITORIES: biostudies-literature

ACCESS DATA

Publications

Measuring competency-relevant knowledge in the competency-oriented student progress test.

Möltner Andreas A Wagener Stefan S Burkert Mirka M

GMS journal for medical education 20200217 1

<b>Background:</b> Since 2013 a competency-oriented student progress test (SKPT) has been administered at a number of German medical schools. The questions are generated on the basis of a two-dimensional blueprint, on which one axis contains the five competency domains - communicative competence (CO), practical clinical competence (CP), theoretical clinical competence (CT), scientific competence (SC), and professional decision-making competence (PR) - that form part of the competency model of th ...[more]

PMID: 32270020

Similar Datasets

Project description:INTRODUCTION:Progress tests provide students feedback on their level of proficiency over the course of their medical studies. Peer-assisted learning and competency-based education have become increasingly important in medical education. Although progress tests have been proven to be useful as a longitudinal feedback instrument, there are currently no progress tests that have been created in cooperation with students or that focus on competency in medical education. In this study, we investigated the extent to which students can be included in the development of a progress test and demonstrated that aspects of knowledge related to competency can be represented on a competency-based progress test. METHODS:A two-dimensional blueprint for 144 multiple-choice questions (MCQs) covering groups of medical subjects and groups of competency areas was generated by three expert groups for developing the competency-based progress test. A total of 31 students from seven medical schools in Germany actively participated in this exercise. After completing an intensive and comprehensive training programme, the students generated and reviewed the test questions for the competency-based progress test using a separate platform of the ItemManagementSystem (IMS). This test was administered as a formative test to 469 students in a pilot study in November 2013 at eight medical schools in Germany. The scores were analysed for the overall test and differentiated according to the subject groups and competency areas. RESULTS:A pool of more than 200 MCQs was compiled by the students for pilot use, of which 118 student-generated MCQs were used in the progress test. University instructors supplemented this pool with 26 MCQs, which primarily addressed the area of scientific skills. The post-review showed that student-generated MCQs were of high quality with regard to test statistic criteria and content. Overall, the progress test displayed a very high reliability. When the academic years were compared, the progress test mapped out over the course of study not only by the overall test but also in terms of the subject groups and competency areas. OUTLOOK:Further development in cooperation with students will be continued. Focus will be on compiling additional questions and test formats that can represent competency at a higher skill level, such as key feature questions, situational judgement test questions and OSCE. In addition, the feedback formats will be successively expanded. The intention is also to offer the formative competency-based progress test online.

Project description:Background To ascertain if undergraduate medical students attain adequate knowledge to practice in paediatrics, we designed the minimum accepted competency (MAC) examination. This was a set of MCQ’s designed to test the most basic, ‘must know’ knowledge as determined by non-faculty paediatric clinicians. Only two-thirds of undergraduate students passed this exam, despite 96% of the same cohort passing their official university paediatric examination. We aim to describe the psychometric properties of the MAC examination to explore why there was a difference in student performance between these two assessments which should, in theory, be testing the same subject area. We will also investigate if the MAC examination is a potentially reliable method of assessing undergraduate knowledge. Methods The MAC examination was sat by three groups of undergraduate medical students and paediatric trainee doctors. Test item analysis was performed using facility index, discrimination index and Cronbach’s alpha. Results Test item difficulty on the MAC between each group was positively correlated. Correlation of item difficulty with the standard set for each item showed a statistically significant positive relationship. However, for 10 of the items, the mean score achieved by the candidates did not even reach two standard deviations below the standard set by the faculty. Medical students outperformed the trainee doctors on three items. 18 of 30 items achieved a discrimination index > 0.2. Cronbach’s alpha ranged from 0.22–0.59. Conclusion Despite faculty correctly judging that this would be a difficult paper for the candidates, there were a significant number of items on which students performed particularly badly. It is possible that the clinical emphasis in these non-faculty derived questions was juxtaposed with the factual recall often required for university examinations. The MAC examination highlights the difference in the level of knowledge expected of a junior doctor starting work in paediatrics between faculty and non-faculty clinicians and can identify gaps between the current curriculum and the ‘hidden curriculum’ required for real world clinical practice. The faculty comprises physicians in employment by the University whose role it is to design the paediatric curriculum and deliver teaching to undergraduate students. Non-faculty clinicians are paediatric physicians who work soley as clinicians with no affiliation to an educational institution. The concept of a MAC examination to test basic medical knowledge is feasible and the study presented is an encouraging first step towards this method of assessment. Supplementary information The online version contains supplementary material available at 10.1186/s12909-022-03475-8.

Project description:Background: General practitioners and general internists occupy a key position in German and Austrian healthcare systems. They provide primary care and act as gatekeepers between medical disciplines and sectors of care. Their explicit medical knowledge levels, however, can be quite disparate. Objective: This study analyses whether general practitioners' performances on a standardised knowledge test changes with four relevant socio-demographic variables. Design: The survey was based on the Progress Test Medicine (PTM), a standardised 200 item knowledge test on graduate level. After formal blueprinting and item analysis, 60 items of PTM were selected ("PTM-GP") for our study. Participants: PTM-GP was presented ad hoc to general practitioners and internists from Germany and Austria at a number of professional meetings in 2011. 161 volunteers completed the survey. Main measures: For evaluation, correlation analysis (Spearman), Kruskal Wallis-tests for non-parametric data and an analysis of covariance (ANCOVA) were calculated. Results: Overall, four indicators turned out to be slightly significant for the performance on the PTM-GP, namely: time passed since graduation, the grade received in the licensing examination, the type of institution for postgraduate training, and the medical specialisation. Conclusions: Recent graduates performed better in the PTM-GP; a doctor's licensing examination grade as well as training at a university hospital correlated positively with PTM-GP performance. A general doctor's knowledge level is moderately influenced by exam grades, time since graduation, the institutional affiliation of postgraduate training and medical specialisation. Individual changes in knowledge over time have to be deliberately considered in lifelong learning. In consequence, the on-going teaching of medical knowledge should be integrated mandatory and verifiable into general doctors' everyday practices, e.g. through repetitive knowledge tests with individual feedback and recommendations for further continuing medical education (CME).

Dataset Information

Measuring competency-relevant knowledge in the competency-oriented student progress test.

Publications

Measuring competency-relevant knowledge in the competency-oriented student progress test.

Similar Datasets

OmicsDI is part of the ELIXIR infrastructure

Tweets