Construction and utilization of a script concordance test as an assessment tool for DCEM3 (5th year) medical students in rheumatology.
Ontology highlight
ABSTRACT: Background
The script concordance test (SCT) is a method for assessing clinical reasoning of medical students by placing them in a context of uncertainty such as they will encounter in their future daily practice. Script concordance testing is going to be included as part of the computer-based national ranking examination (iNRE).This study was designed to create a script concordance test in rheumatology and use it for DCEM3 (fifth year) medical students administered via the online platform of the Clermont-Ferrand medical school.Methods
Our SCT for rheumatology teaching was constructed by a panel of 19 experts in rheumatology (6 hospital-based and 13 community-based). One hundred seventy-nine DCEM3 (fifth year) medical students were invited to take the test. Scores were computed using the scoring key available on the University of Montreal website. Reliability of the test was estimated by the Cronbach alpha coefficient for internal consistency.Results
The test comprised 60 questions. Among the 26 students who took the test (26/179: 14.5%), 15 completed it in its entirety. The reference panel of rheumatologists obtained a mean score of 76.6 and the 15 students had a mean score of 61.5 (p = 0.001). The Cronbach alpha value was 0.82.Conclusions
An online SCT can be used as an assessment tool for medical students in rheumatology. This study also highlights the active participation of community-based rheumatologists, who accounted for the majority of the 19 experts in the reference panel.A script concordance test in rheumatology for 5th year medical students.
SUBMITTER: Mathieu S
PROVIDER: S-EPMC3878954 | biostudies-literature |
REPOSITORIES: biostudies-literature
Similar Datasets
Project description:BackgroundThe evaluation process of French medical students will evolve in the next few years in order to improve assessment validity. Script concordance testing (SCT) offers the possibility to assess medical knowledge alongside clinical reasoning under conditions of uncertainty. In this study, we aimed at comparing the SCT scores of a large cohort of undergraduate medical students, according to the experience level of the reference panel.MethodsIn 2019, the authors developed a 30-item SCT and sent it to experts with varying levels of experience. Data analysis included score comparisons with paired Wilcoxon rank sum tests and concordance analysis with Bland & Altman plots.ResultsA panel of 75 experts was divided into three groups: 31 residents, 21 non-experienced physicians (NEP) and 23 experienced physicians (EP). Among each group, random samples of N = 20, 15 and 10 were selected. A total of 985 students from nine different medical schools participated in the SCT examination. No matter the size of the panel (N = 20, 15 or 10), students' SCT scores were lower with the NEP group when compared to the resident panel (median score 67.1 vs 69.1, p < 0.0001 if N = 20; 67.2 vs 70.1, p < 0.0001 if N = 15 and 67.7 vs 68.4, p < 0.0001 if N = 10) and with EP compared to NEP (65.4 vs 67.1, p < 0.0001 if N = 20; 66.0 vs 67.2, p < 0.0001 if N = 15 and 62.5 vs 67.7, p < 0.0001 if N = 10). Bland & Altman plots showed good concordances between students' SCT scores, whatever the experience level of the expert panel.ConclusionsEven though student SCT scores differed statistically according to the expert panels, these differences were rather weak. These results open the possibility of including less-experienced experts in panels for the evaluation of medical students.
| S-EPMC7499961 | biostudies-literature
Project description:ObjectivesThis investigation aimed to determine the validity of script concordance test (SCT), compared with clinical-case-related short-answer management problems (SAMP), in fourth-year medical students.MethodsThis retrospective study was conducted at the Medical School of Lille University. Cardiology and gynecology examinations both included 3 SCT and 2 clinical-case-related SAMP. Final score did not include SCT results, and was out of 20 points. The passing score was ≥10/20. Wilcoxon and McNemar tests were used to compare quantitative and qualitative variables, respectively. Correlation between scores was also analyzed.ResultsA total of 519 and 521 students completed SAMP and SCT in cardiology and gynecology, respectively. Cardiology score was significantly higher in SCT than SAMP (mean ± SD 13.5±2.4 versus 11.4±2.6, Wilcoxon test, p<0.001). In gynecology, SCT score was significantly lower than SAMP score (10.8±2.6 versus 11.4±2.7, Wilcoxon test, p=0.001). SCT and SAMP scores were significantly correlated (p <0.05, Pearson's correlation). However, percentage of students with SCT score ≥ 10/20 was similar among those who passed or failed cardiology (327 of 359 (91%) vs 146 of 160 (91%), χ2=0.004, df =1, p=0.952), or gynecology (274 of 379 (65%) vs 84 of 142 (59%), χ2=1.614, df=1, p=0.204) SAMP test. Cronbach alpha coefficient was 0.31 and 0.92 for all SCT and SAMP, respectively.Conclusions Although significantly correlated, the scores obtained in SCT and SAMP were significantly different in fourth-year medical students. These findings suggest that SCT should not be used for summative purposes in fourth-year medical students.
| S-EPMC5339020 | biostudies-literature
Project description:IntroductionA primary aim of residency training is to develop competence in clinical reasoning. However, there are few instruments that can accurately, reliably, and efficiently assess residents' clinical decision-making ability. This study aimed to externally validate the script concordance test in emergency medicine (SCT-EM), an assessment tool designed for this purpose.MethodsUsing established methodology for the SCT-EM, we compared EM residents' performance on the SCT-EM to an expert panel of emergency physicians at three urban academic centers. We performed adjusted pairwise t-tests to compare differences between all residents and attending physicians, as well as among resident postgraduate year (PGY) levels. We tested correlation between SCT-EM and Accreditation Council for Graduate Medical Education Milestone scores using Pearson's correlation coefficients. Inter-item covariances for SCT items were calculated using Cronbach's alpha statistic.ResultsThe SCT-EM was administered to 68 residents and 13 attendings. There was a significant difference in mean scores among all groups (mean + standard deviation: PGY-1 59 + 7; PGY-2 62 + 6; PGY-3 60 + 8; PGY-4 61 + 8; 73 + 8 for attendings, p < 0.01). Post hoc pairwise comparisons demonstrated that significant difference in mean scores only occurred between each PGY level and the attendings (p < 0.01 for PGY-1 to PGY-4 vs attending group). Performance on the SCT-EM and EM Milestones was not significantly correlated (r = 0.12, p = 0.35). Internal reliability of the exam was determined using Cronbach's alpha, which was 0.67 for all examinees, and 0.89 in the expert-only group.ConclusionThe SCT-EM has limited utility in reliably assessing clinical reasoning among EM residents. Although the SCT-EM was able to differentiate clinical reasoning ability between residents and expert faculty, it did not between PGY levels, or correlate with Milestones scores. Furthermore, several limitations threaten the validity of the SCT-EM, suggesting further study is needed in more diverse settings.
| S-EPMC7390545 | biostudies-literature
Project description:IntroductionClinical reasoning is a fundamental part of a physician's daily workflow. Yet it remains a challenging skill to develop formally, especially in preclerkship-level early learners. Traditionally, medical students learn clinical reasoning informally through experiential opportunities during their clerkship years. This occurs in contrast to the more structured, explicit learning of the basic sciences and physical diagnosis during the preclerkship years. To address this need, we present a flipped classroom case-based approach for developing clinical reasoning skills based on problem representation and the use of a structured illness script worksheet as a model.MethodsStudents were given a short introduction via screencast to introduce clinical reasoning and related terminology such as problem representation and semantic qualifiers. They also received a case vignette and an illness script worksheet to prepare them for in-class discussion. Students used this worksheet to practice clinical reasoning in a small-group session that was held in our last organ system-based second-year course, prior to the start of the clerkships.ResultsIn comparison to the traditional facilitator-led small-group sessions, where students would sequentially answer a set of defined content-based questions to explore a clinical case, 80% of students preferred the new framework that incorporates problem representation and the illness script worksheets. Faculty facilitators found the structure of the illness script worksheet helpful in leading a clinical reasoning small-group session.DiscussionBased on the results of this pilot, we plan to systematically implement this clinical reasoning framework in our preclerkship curriculum.
| S-EPMC6464440 | biostudies-literature
Project description:Introduction: Clinical reasoning is a crucial skill in the practice of pediatric emergency medicine and a vital element of the various competencies achieved during the clinical training of resident doctors. Pediatric emergency physicians are often required to stabilize patients and make correct diagnoses with limited clinical information, time and resources. The Pediatric Emergency Medicine Script Concordance Test (PEM-SCT) has been developed specifically for assessing physician's reasoning skills in the context of the uncertainties in pediatric emergency practice. In this study, we developed the Japanese version of the PEM-SCT (Jpem-SCT) and confirmed its validity by collecting relevant evidence. Methods: The Jpem-SCT was developed by translating the PEM-SCT into Japanese using the Translation, Review, Adjudication, Pretest, Documentation team translation model, which follows cross-cultural survey guidelines for proper translation and cross-cultural and linguistic equivalences between the English and Japanese version of the survey. First, 15 experienced pediatricians participated in the pre-test session, serving as a reference panel for modifying the test descriptions, incorporating Japanese context, and establishing the basis for the scoring process. Then, a 1-h test containing 60 questions was administered to 75 trainees from three academic institutions. Following data collection, we calculated the item-total correlations of the scores to optimize selection of the best items in the final version of the Jpem-SCT. The reliability of the finalized Jpem-SCT was calculated using Cronbach's α coefficient for ensuring generalizability of the evidence. We also conducted multiple regression analysis of the test score to collect evidence on validity of the extrapolation. Results: The final version of the test, based on item-total correlation data analysis, contained 45 questions. The participant's specialties were as follows: Transitional interns 12.0%, pediatric residents 56.0%, emergency medicine residents 25.3%, and PEM fellows 6.7%. The mean score of the final version of the Jpem-SCT was 68.6 (SD 9.8). The reliability of the optimized test (Cronbach's α) was 0.70. Multiple regression analysis showed that being a transitional intern was a negative predictor of test scores, indicating that clinical experience relates to performance on the Jpem-SCT. Conclusion: This pediatric emergency medicine Script Concordance Test was reliable and valid for assessing the development of clinical reasoning by trainee doctors during residency training.
| S-EPMC8688734 | biostudies-literature
Project description:OBJECTIVE: Positive attitudes towards General Practice can be understood as a prerequisite for becoming a General Practitioner (GP) and for collaboration with GPs later on. This study aimed to assess attitudes of medical students at the beginning and the end of medical school. METHODS: A total of 160 1(st) year students at Hannover Medical School were surveyed. Their attitudes were compared to those of 287 5th year students. Descriptive, bi- and multivariate analyses were performed to investigate influences of year of study and gender. RESULTS: Year of study and gender both were associated with the attitudes towards General Practice. The interest in General Practice and patient-orientation (communication, care of older patients with chronic diseases) was higher in 1st year students compared to more advanced students. Female students valued such requirements more than male students, the differences in attitudes between the years of study being more pronounced in male students. CONCLUSION: Despite some limitations caused by the cross-sectional design, the attitudes towards General Practice competencies changed to their disadvantage during medical school. This suggests a formative influence of the strategies used in medical education. Educational strategies, however, could be used to bring about a change of attitudes in the other direction.
| S-EPMC3525916 | biostudies-other
Project description:ObjectivesThis study investigated whether medical student responses to Script Concordance Testing (SCT) items represent valid clinical reasoning. Using a think-aloud approach students provided written explanations of the reasoning that underpinned their responses, and these were reviewed for concordance with an expert reference panel.MethodsA set of 12, 11 and 15 SCT items were administered online to Year 3 (2018), Year 4 (2018) and Year 3 (2019) medical students respectively. Students' free-text descriptions of the reasoning supporting each item response were analysed, and compared with those of the expert panel. Response process validity was quantified as the rate of true positives (percentage of full and partial credit responses derived through correct clinical reasoning); and true negatives (percentage of responses with no credit derived through faulty clinical reasoning).ResultsTwo hundred and nine students completed the online tests (response rate = 68.3%). The majority of students who had chosen the response which attracted full or partial credit also provided justifications which were concordant with the experts (true positive rate of 99.6% for full credit; 99.4% for partial credit responses). Most responses that attracted no credit were based on faulty clinical reasoning (true negative of 99.0%).ConclusionsThe findings provide support for the response process validity of SCT scores in the setting of undergraduate medicine. The additional written think-aloud component, to assess clinical reasoning, provided useful information to inform student learning. However, SCT scores should be validated on each testing occasion, and in other contexts.
| S-EPMC7870454 | biostudies-literature
Project description:Background and aimConcept maps hold great potential for the students as it helps in the formulation of new concepts as well to evaluate the learning ability of the students. However, the use of concept maps as a teaching tool in medical students in India has not been explored, to the best of our knowledge. This study was conducted to assess the use of concept maps in improving learning among medical students.Materials and methodsThis study was conducted among III MBBS students belonging to two batches. This study was carried out in two stages. In first stage, pretest was taken to test the knowledge of students. Later, introductory class was taken on tuberculosis and concept map was used to explain the given concept and later posttest was taken. In second stage, feedback was taken from the students regarding the concept map. Pre- and post-test results were compared using Wilcoxon test.ResultsSignificant difference between pretest (4 ± 1.593) and posttest (10 ± 0.762) score was seen (P < 0.0001). More than 50% of students scored full marks in posttest, whereas no one got full marks in the pretest. Positive response (82.09%) was received from the students when feedback was taken regarding the use of concept map.ConclusionConcept maps are found to be an effective teaching and learning tool for medical students. They can be used to enhance meaningful learning in students and can be practiced more in the students for better understandings of the concepts.
| S-EPMC7933622 | biostudies-literature
Project description:In self-directed learning environments, students must monitor and assess their learning progress themselves. Formative assessments play a central role as instructional tools to train students in these self-assessment skills. Key to the effectiveness of a formative assessment as a training instrument is the degree of student involvement in the design and implementation of the formative assessment used to train them. Five characteristics are essential to secure a sufficient level of student involvement: (1) congruence (i.e., the tasks of a formative assessment must reflect the instructional content), (2) authenticity (i.e., the tasks of a formative assessment should be related to students' background and study context), (3) consultation (i.e., students should have a say in how their answers are evaluated based on which criteria), (4) transparency (i.e., no "mystery items"-the wording of the items must clearly addresses the targeted content), and (5) accommodation (i.e., all students should have the same chance to complete the items of a formative assessment). These five characteristics of student involvement were used to evaluate the quality of formative assessments that are employed to train undergraduates in a medical school in self-assessment skills. Different from extant approaches that typically rely on instructors' judgments to infer the degree of student involvement, this study uses a paradigm in higher educational research asking students directly about their perceptions of an instructional tool. Thus, here, students had to rate their level of involvement in the design and implementation of the formative assessments used to train them. A total of 140 medical students participated in the study. Multivariate statistical methods were used to analyze the data. The majority of medical students appreciate the use of formative assessments to teach them self-directed learning skills; but the results also show that there is still room for increasing student involvement-especially, for minority students.
| S-EPMC8368904 | biostudies-literature
Project description:IntroductionThere is growing interest in, and emphasis on, electronic teaching tools in medicine. Despite relevant testing on the United States Medical Licensing Examination (USMLE), American medical schools offer limited training in skin disorders. Teaching visual topics like dermatology in classroom formats is challenging. We hypothesized that an electronic module would enhance students' dermatology competency.MethodsA self-directed, case-based module was created. To test its efficacy, 40 medical students were randomized to have module access (interventional group) or none (conventional group). Learning outcomes were compared using a multiple-choice exam, including questions relevant and irrelevant to the module. Outcomes included proportions of correctly answered module questions (module scores) and nonmodule questions (nonmodule scores). Difference scores were calculated: (module score) - (nonmodule score). Positive values indicated that knowledge of module questions surpassed that of nonmodule questions. If there were a training effect, the interventional group's difference score should exceed that of the conventional group.ResultsThe interventional group scored significantly higher than did the conventional group on module questions-75% (interquartile range [IQR], 69-88) versus 50% (IQR, 38-63), p < .001-and nonmodule questions-85% (IQR, 69-92) versus 69% (IQR, 54-77), p = .02. The Hodges-Lehman median difference estimate of the training effect was 13.0 (95% confidence interval, 0.5-25.5).DiscussionThis e-module is effective at enhancing students' competency in dermatology while emphasizing detailed pathophysiology that prepares them for USMLE Step 1. A module-based curriculum may enhance learning in supplement to traditional teaching modalities.
| S-EPMC6338178 | biostudies-literature