Construction and utilization of a script concordance test as an assessment tool for DCEM3 (5th year) medical students in rheumatology.
Ontology highlight
ABSTRACT: Background
The script concordance test (SCT) is a method for assessing clinical reasoning of medical students by placing them in a context of uncertainty such as they will encounter in their future daily practice. Script concordance testing is going to be included as part of the computer-based national ranking examination (iNRE).This study was designed to create a script concordance test in rheumatology and use it for DCEM3 (fifth year) medical students administered via the online platform of the Clermont-Ferrand medical school.Methods
Our SCT for rheumatology teaching was constructed by a panel of 19 experts in rheumatology (6 hospital-based and 13 community-based). One hundred seventy-nine DCEM3 (fifth year) medical students were invited to take the test. Scores were computed using the scoring key available on the University of Montreal website. Reliability of the test was estimated by the Cronbach alpha coefficient for internal consistency.Results
The test comprised 60 questions. Among the 26 students who took the test (26/179: 14.5%), 15 completed it in its entirety. The reference panel of rheumatologists obtained a mean score of 76.6 and the 15 students had a mean score of 61.5 (p = 0.001). The Cronbach alpha value was 0.82.Conclusions
An online SCT can be used as an assessment tool for medical students in rheumatology. This study also highlights the active participation of community-based rheumatologists, who accounted for the majority of the 19 experts in the reference panel.A script concordance test in rheumatology for 5th year medical students.
SUBMITTER: Mathieu S
PROVIDER: S-EPMC3878954 | biostudies-literature |
REPOSITORIES: biostudies-literature
Similar Datasets
Project description:BackgroundThe evaluation process of French medical students will evolve in the next few years in order to improve assessment validity. Script concordance testing (SCT) offers the possibility to assess medical knowledge alongside clinical reasoning under conditions of uncertainty. In this study, we aimed at comparing the SCT scores of a large cohort of undergraduate medical students, according to the experience level of the reference panel.MethodsIn 2019, the authors developed a 30-item SCT and sent it to experts with varying levels of experience. Data analysis included score comparisons with paired Wilcoxon rank sum tests and concordance analysis with Bland & Altman plots.ResultsA panel of 75 experts was divided into three groups: 31 residents, 21 non-experienced physicians (NEP) and 23 experienced physicians (EP). Among each group, random samples of N = 20, 15 and 10 were selected. A total of 985 students from nine different medical schools participated in the SCT examination. No matter the size of the panel (N = 20, 15 or 10), students' SCT scores were lower with the NEP group when compared to the resident panel (median score 67.1 vs 69.1, p < 0.0001 if N = 20; 67.2 vs 70.1, p < 0.0001 if N = 15 and 67.7 vs 68.4, p < 0.0001 if N = 10) and with EP compared to NEP (65.4 vs 67.1, p < 0.0001 if N = 20; 66.0 vs 67.2, p < 0.0001 if N = 15 and 62.5 vs 67.7, p < 0.0001 if N = 10). Bland & Altman plots showed good concordances between students' SCT scores, whatever the experience level of the expert panel.ConclusionsEven though student SCT scores differed statistically according to the expert panels, these differences were rather weak. These results open the possibility of including less-experienced experts in panels for the evaluation of medical students.
| S-EPMC7499961 | biostudies-literature
Project description:ObjectivesThis investigation aimed to determine the validity of script concordance test (SCT), compared with clinical-case-related short-answer management problems (SAMP), in fourth-year medical students.MethodsThis retrospective study was conducted at the Medical School of Lille University. Cardiology and gynecology examinations both included 3 SCT and 2 clinical-case-related SAMP. Final score did not include SCT results, and was out of 20 points. The passing score was ≥10/20. Wilcoxon and McNemar tests were used to compare quantitative and qualitative variables, respectively. Correlation between scores was also analyzed.ResultsA total of 519 and 521 students completed SAMP and SCT in cardiology and gynecology, respectively. Cardiology score was significantly higher in SCT than SAMP (mean ± SD 13.5±2.4 versus 11.4±2.6, Wilcoxon test, p<0.001). In gynecology, SCT score was significantly lower than SAMP score (10.8±2.6 versus 11.4±2.7, Wilcoxon test, p=0.001). SCT and SAMP scores were significantly correlated (p <0.05, Pearson's correlation). However, percentage of students with SCT score ≥ 10/20 was similar among those who passed or failed cardiology (327 of 359 (91%) vs 146 of 160 (91%), χ2=0.004, df =1, p=0.952), or gynecology (274 of 379 (65%) vs 84 of 142 (59%), χ2=1.614, df=1, p=0.204) SAMP test. Cronbach alpha coefficient was 0.31 and 0.92 for all SCT and SAMP, respectively.Conclusions Although significantly correlated, the scores obtained in SCT and SAMP were significantly different in fourth-year medical students. These findings suggest that SCT should not be used for summative purposes in fourth-year medical students.
| S-EPMC5339020 | biostudies-literature
Project description:IntroductionA primary aim of residency training is to develop competence in clinical reasoning. However, there are few instruments that can accurately, reliably, and efficiently assess residents' clinical decision-making ability. This study aimed to externally validate the script concordance test in emergency medicine (SCT-EM), an assessment tool designed for this purpose.MethodsUsing established methodology for the SCT-EM, we compared EM residents' performance on the SCT-EM to an expert panel of emergency physicians at three urban academic centers. We performed adjusted pairwise t-tests to compare differences between all residents and attending physicians, as well as among resident postgraduate year (PGY) levels. We tested correlation between SCT-EM and Accreditation Council for Graduate Medical Education Milestone scores using Pearson's correlation coefficients. Inter-item covariances for SCT items were calculated using Cronbach's alpha statistic.ResultsThe SCT-EM was administered to 68 residents and 13 attendings. There was a significant difference in mean scores among all groups (mean + standard deviation: PGY-1 59 + 7; PGY-2 62 + 6; PGY-3 60 + 8; PGY-4 61 + 8; 73 + 8 for attendings, p < 0.01). Post hoc pairwise comparisons demonstrated that significant difference in mean scores only occurred between each PGY level and the attendings (p < 0.01 for PGY-1 to PGY-4 vs attending group). Performance on the SCT-EM and EM Milestones was not significantly correlated (r = 0.12, p = 0.35). Internal reliability of the exam was determined using Cronbach's alpha, which was 0.67 for all examinees, and 0.89 in the expert-only group.ConclusionThe SCT-EM has limited utility in reliably assessing clinical reasoning among EM residents. Although the SCT-EM was able to differentiate clinical reasoning ability between residents and expert faculty, it did not between PGY levels, or correlate with Milestones scores. Furthermore, several limitations threaten the validity of the SCT-EM, suggesting further study is needed in more diverse settings.
| S-EPMC7390545 | biostudies-literature
Project description:BackgroundUndergraduate medical studies represent a wide range of learning opportunities served in the form of various teaching-learning modalities for medical learners. A clinical scenario is frequently used as a modality, followed by multiple-choice and open-ended questions among other learning and teaching methods. As such, script concordance tests (SCTs) can be used to promote a higher level of clinical reasoning. Recent technological developments have made generative artificial intelligence (AI)-based systems such as ChatGPT (OpenAI) available to assist clinician-educators in creating instructional materials.ObjectiveThe main objective of this project is to explore how SCTs generated by ChatGPT compared to SCTs produced by clinical experts on 3 major elements: the scenario (stem), clinical questions, and expert opinion.MethodsThis mixed method study evaluated 3 ChatGPT-generated SCTs with 3 expert-created SCTs using a predefined framework. Clinician-educators as well as resident doctors in psychiatry involved in undergraduate medical education in Quebec, Canada, evaluated via a web-based survey the 6 SCTs on 3 criteria: the scenario, clinical questions, and expert opinion. They were also asked to describe the strengths and weaknesses of the SCTs.ResultsA total of 102 respondents assessed the SCTs. There were no significant distinctions between the 2 types of SCTs concerning the scenario (P=.84), clinical questions (P=.99), and expert opinion (P=.07), as interpretated by the respondents. Indeed, respondents struggled to differentiate between ChatGPT- and expert-generated SCTs. ChatGPT showcased promise in expediting SCT design, aligning well with Diagnostic and Statistical Manual of Mental Disorders, Fifth Edition criteria, albeit with a tendency toward caricatured scenarios and simplistic content.ConclusionsThis study is the first to concentrate on the design of SCTs supported by AI in a period where medicine is changing swiftly and where technologies generated from AI are expanding much faster. This study suggests that ChatGPT can be a valuable tool in creating educational materials, and further validation is essential to ensure educational efficacy and accuracy.
| S-EPMC11007379 | biostudies-literature
Project description:IntroductionClinical reasoning is a fundamental part of a physician's daily workflow. Yet it remains a challenging skill to develop formally, especially in preclerkship-level early learners. Traditionally, medical students learn clinical reasoning informally through experiential opportunities during their clerkship years. This occurs in contrast to the more structured, explicit learning of the basic sciences and physical diagnosis during the preclerkship years. To address this need, we present a flipped classroom case-based approach for developing clinical reasoning skills based on problem representation and the use of a structured illness script worksheet as a model.MethodsStudents were given a short introduction via screencast to introduce clinical reasoning and related terminology such as problem representation and semantic qualifiers. They also received a case vignette and an illness script worksheet to prepare them for in-class discussion. Students used this worksheet to practice clinical reasoning in a small-group session that was held in our last organ system-based second-year course, prior to the start of the clerkships.ResultsIn comparison to the traditional facilitator-led small-group sessions, where students would sequentially answer a set of defined content-based questions to explore a clinical case, 80% of students preferred the new framework that incorporates problem representation and the illness script worksheets. Faculty facilitators found the structure of the illness script worksheet helpful in leading a clinical reasoning small-group session.DiscussionBased on the results of this pilot, we plan to systematically implement this clinical reasoning framework in our preclerkship curriculum.
| S-EPMC6464440 | biostudies-literature
Project description:Introduction: Clinical reasoning is a crucial skill in the practice of pediatric emergency medicine and a vital element of the various competencies achieved during the clinical training of resident doctors. Pediatric emergency physicians are often required to stabilize patients and make correct diagnoses with limited clinical information, time and resources. The Pediatric Emergency Medicine Script Concordance Test (PEM-SCT) has been developed specifically for assessing physician's reasoning skills in the context of the uncertainties in pediatric emergency practice. In this study, we developed the Japanese version of the PEM-SCT (Jpem-SCT) and confirmed its validity by collecting relevant evidence. Methods: The Jpem-SCT was developed by translating the PEM-SCT into Japanese using the Translation, Review, Adjudication, Pretest, Documentation team translation model, which follows cross-cultural survey guidelines for proper translation and cross-cultural and linguistic equivalences between the English and Japanese version of the survey. First, 15 experienced pediatricians participated in the pre-test session, serving as a reference panel for modifying the test descriptions, incorporating Japanese context, and establishing the basis for the scoring process. Then, a 1-h test containing 60 questions was administered to 75 trainees from three academic institutions. Following data collection, we calculated the item-total correlations of the scores to optimize selection of the best items in the final version of the Jpem-SCT. The reliability of the finalized Jpem-SCT was calculated using Cronbach's α coefficient for ensuring generalizability of the evidence. We also conducted multiple regression analysis of the test score to collect evidence on validity of the extrapolation. Results: The final version of the test, based on item-total correlation data analysis, contained 45 questions. The participant's specialties were as follows: Transitional interns 12.0%, pediatric residents 56.0%, emergency medicine residents 25.3%, and PEM fellows 6.7%. The mean score of the final version of the Jpem-SCT was 68.6 (SD 9.8). The reliability of the optimized test (Cronbach's α) was 0.70. Multiple regression analysis showed that being a transitional intern was a negative predictor of test scores, indicating that clinical experience relates to performance on the Jpem-SCT. Conclusion: This pediatric emergency medicine Script Concordance Test was reliable and valid for assessing the development of clinical reasoning by trainee doctors during residency training.
| S-EPMC8688734 | biostudies-literature
Project description:BackgroundThe illness script method employs a theoretical outline (e.g., epidemiology, pathophysiology, signs and symptoms, diagnostic tests, interventions) to clarify how clinicians organized medical knowledge for clinical reasoning in the diagnosis domain. We hypothesized that an educational intervention based on the illness script method would improve medical students' clinical reasoning skills in the diagnosis domain.MethodsThis study is a randomized controlled trial involving 100 fourth-year medical students in Shiraz Medical School, Iran. Fifty students were randomized to the intervention group, who were taught clinical reasoning skills based on the illness script method for three diseases during one clinical scenario. Another 50 students were randomized to the control group, who were taught the clinical presentation based on signs and symptoms of the same three diseases as the intervention group. The outcomes of interest were learner satisfaction with the intervention and posttest scores on both an internally developed knowledge test and a Script Concordance Test (SCT).ResultsOf the hundred participating fourth-year medical students, 47 (47%) were male, and 53 (53%) were female. On the knowledge test, there was no difference in pretest scores between the intervention and control group, which suggested a similar baseline knowledge in both groups; however, posttest scores in the intervention group were (15.74 ± 2.47 out of 20) statistically significantly higher than the control group (14.38 ± 2.59 out of 20, P = 0.009). On the SCT, the mean score for the intervention group (6.12 ± 1.95 out of 10) was significantly higher than the control group (4.54 ± 1.56 out of 10; P = 0.0001). Learner satisfaction data indicated that the intervention was well-received by students.ConclusionTeaching with the illness script method was an effective way to improve students' clinical reasoning skills in the diagnosis domain suggested by posttest and SCT scores for specific clinical scenarios. Whether this approach translates to improved generalized clinical reasoning skills in real clinical settings merits further study.
| S-EPMC7856771 | biostudies-literature
Project description:OBJECTIVE: Positive attitudes towards General Practice can be understood as a prerequisite for becoming a General Practitioner (GP) and for collaboration with GPs later on. This study aimed to assess attitudes of medical students at the beginning and the end of medical school. METHODS: A total of 160 1(st) year students at Hannover Medical School were surveyed. Their attitudes were compared to those of 287 5th year students. Descriptive, bi- and multivariate analyses were performed to investigate influences of year of study and gender. RESULTS: Year of study and gender both were associated with the attitudes towards General Practice. The interest in General Practice and patient-orientation (communication, care of older patients with chronic diseases) was higher in 1st year students compared to more advanced students. Female students valued such requirements more than male students, the differences in attitudes between the years of study being more pronounced in male students. CONCLUSION: Despite some limitations caused by the cross-sectional design, the attitudes towards General Practice competencies changed to their disadvantage during medical school. This suggests a formative influence of the strategies used in medical education. Educational strategies, however, could be used to bring about a change of attitudes in the other direction.
| S-EPMC3525916 | biostudies-other
Project description:ObjectivesThis study investigated whether medical student responses to Script Concordance Testing (SCT) items represent valid clinical reasoning. Using a think-aloud approach students provided written explanations of the reasoning that underpinned their responses, and these were reviewed for concordance with an expert reference panel.MethodsA set of 12, 11 and 15 SCT items were administered online to Year 3 (2018), Year 4 (2018) and Year 3 (2019) medical students respectively. Students' free-text descriptions of the reasoning supporting each item response were analysed, and compared with those of the expert panel. Response process validity was quantified as the rate of true positives (percentage of full and partial credit responses derived through correct clinical reasoning); and true negatives (percentage of responses with no credit derived through faulty clinical reasoning).ResultsTwo hundred and nine students completed the online tests (response rate = 68.3%). The majority of students who had chosen the response which attracted full or partial credit also provided justifications which were concordant with the experts (true positive rate of 99.6% for full credit; 99.4% for partial credit responses). Most responses that attracted no credit were based on faulty clinical reasoning (true negative of 99.0%).ConclusionsThe findings provide support for the response process validity of SCT scores in the setting of undergraduate medicine. The additional written think-aloud component, to assess clinical reasoning, provided useful information to inform student learning. However, SCT scores should be validated on each testing occasion, and in other contexts.
| S-EPMC7870454 | biostudies-literature
Project description:Background and aimConcept maps hold great potential for the students as it helps in the formulation of new concepts as well to evaluate the learning ability of the students. However, the use of concept maps as a teaching tool in medical students in India has not been explored, to the best of our knowledge. This study was conducted to assess the use of concept maps in improving learning among medical students.Materials and methodsThis study was conducted among III MBBS students belonging to two batches. This study was carried out in two stages. In first stage, pretest was taken to test the knowledge of students. Later, introductory class was taken on tuberculosis and concept map was used to explain the given concept and later posttest was taken. In second stage, feedback was taken from the students regarding the concept map. Pre- and post-test results were compared using Wilcoxon test.ResultsSignificant difference between pretest (4 ± 1.593) and posttest (10 ± 0.762) score was seen (P < 0.0001). More than 50% of students scored full marks in posttest, whereas no one got full marks in the pretest. Positive response (82.09%) was received from the students when feedback was taken regarding the use of concept map.ConclusionConcept maps are found to be an effective teaching and learning tool for medical students. They can be used to enhance meaningful learning in students and can be practiced more in the students for better understandings of the concepts.
| S-EPMC7933622 | biostudies-literature