Dataset Information

Machine Scoring of Medical Students' Written Clinical Reasoning: Initial Validity Evidence.

ABSTRACT:

Purpose

Developing medical students' clinical reasoning requires a structured longitudinal curriculum with frequent targeted assessment and feedback. Performance-based assessments, which have the strongest validity evidence, are currently not feasible for this purpose because they are time-intensive to score. This study explored the potential of using machine learning technologies to score one such assessment-the diagnostic justification essay.

Method

From May to September 2018, machine scoring algorithms were trained to score a sample of 700 diagnostic justification essays written by 414 third-year medical students from the Southern Illinois University School of Medicine classes of 2012-2017. The algorithms applied semantically based natural language processing metrics (e.g., coherence, readability) to assess essay quality on 4 criteria (differential diagnosis, recognition and use of findings, workup, and thought process); the scores for these criteria were summed to create overall scores. Three sources of validity evidence (response process, internal structure, and association with other variables) were examined.

Results

Machine scores correlated more strongly with faculty ratings than faculty ratings did with each other (machine: .28-.53, faculty: .13-.33) and were less case-specific. Machine scores and faculty ratings were similarly correlated with medical knowledge, clinical cognition, and prior diagnostic justification. Machine scores were more strongly associated with clinical communication than were faculty ratings (.43 vs .31).

Conclusions

Machine learning technologies may be useful for assessing medical students' long-form written clinical reasoning. Semantically based machine scoring may capture the communicative aspects of clinical reasoning better than faculty ratings, offering the potential for automated assessment that generalizes to the workplace. These results underscore the potential of machine scoring to capture an aspect of clinical reasoning performance that is difficult to assess with traditional analytic scoring methods. Additional research should investigate machine scoring generalizability and examine its acceptability to trainees and educators.

SUBMITTER: Cianciolo AT

PROVIDER: S-EPMC8243833 | biostudies-literature |

REPOSITORIES: biostudies-literature

ACCESS DATA

Similar Datasets

Project description:BACKGROUND:The clinical reasoning process, which requires biomedical knowledge, knowledge about problem-solving strategies, and knowledge about reasons for diagnostic procedures, is a key element of physicians' daily practice but difficult to assess. The aim of this study was to empirically develop a Clinical Reasoning Indicators-History Taking-Scale (CRI-HT-S) and to assess the clinical reasoning ability of advanced medical students during a simulation involving history taking. METHODS:The Clinical Reasoning Indictors-History Taking-Scale (CRI-HT-S) including a 5-point Likert scale for assessment was designed from clinical reasoning indicators identified in a qualitative study in 2017. To assess indicators of clinical reasoning ability, 65 advanced medical students (semester 10, n =?25 versus final year, n =?40) from three medical schools participated in a 360-degree competence assessment in the role of beginning residents during a simulated first workday in hospital. This assessment included a consultation hour with five simulated patients which was videotaped. Videos of 325 patient consultations were assessed using the CRI-HT-S. A factor analysis was conducted and the students' results were compared according to their advancement in undergraduate medical training. RESULTS:The clinical reasoning indicators of the CRI-HT-S loaded on three factors relevant for clinical reasoning: 1) focusing questions, 2) creating context, and 3) securing information. Students reached significantly different scores (p <?.001) for the three factors (factor 1: 4.07?±?.47, factor 2: 3.72?±?.43, factor 3: 2.79?±?.83). Students in semester 10 reached significantly lower scores for factor 3 than students in their final year (p <?.05). CONCLUSIONS:The newly developed CRI-HT-S worked well for quantitative assessment of clinical reasoning indicators during history taking. Its three-factored structure helped to explore different aspects of clinical reasoning. Whether the CRI-HT-S has the potential to be used as a scale in objective structured clinical examinations (OCSEs) or in workplace-based assessments of clinical reasoning has to be investigated in further studies with larger student cohorts.

Project description:PurposeGraduating medical students need broad clinical diagnostic reasoning skills that integrate learning across clinical specialties to deal with undifferentiated patient problems. The opportunity to acquire these skills may be limited during clinical placements on increasingly specialized hospital wards. We developed an intervention of regular general practitioner (GP) facilitated teaching in hospital placements to enable students to develop broad clinical diagnostic reasoning. The intervention was piloted, refined and delivered to a whole cohort of medical students at the start of their third year. This paper examines whether students perceived opportunities to improve their broad diagnostic clinical reasoning through our intervention.MethodsGP-facilitated teaching sessions were delivered weekly in hospital placements to small groups of 6–8 students for 90 mins over 6 weeks. Students practiced clinical reasoning with real patient cases that they encountered on their placements. Evaluation of learning outcomes was conducted through a student questionnaire using Likert scales with free-text boxes for additional explanation. Focus groups were conducted to gain a more in-depth understanding of student perspectives.ResultsAs high as 87% of students agreed that their broad clinical diagnostic reasoning ability had improved. Thematic analysis of the qualitative data revealed four factors supporting this improvement: practicing the hypothetico-deductive method, using real patient cases, composing student groups from different speciality placements and the breadth of the facilitators’ knowledge. Students additionally reported enhanced person-centredness in terms of understanding the patient’s perspective and journey. Students perceived that the added value of general practitioner facilitators lay in their broad knowledge base and knowledge of patient needs in the community.ConclusionOur results suggest that medical students can develop broad clinical diagnostic reasoning skills in hospital settings through regular GP-facilitated teaching. Our approach has the advantage of working within the established curricular format of hospital placements and being deliverable at scale to whole student cohorts.

Project description:BackgroundImproving clinical reasoning skills-the thought processes used by clinicians to formulate appropriate questions and diagnoses-is essential for reducing missed diagnostic opportunities. The electronic Clinical Reasoning Educational Simulation Tool (eCREST) was developed to improve the clinical reasoning of future physicians. A feasibility trial demonstrated acceptability and potential impacts; however, the processes by which students gathered data were unknown.ObjectiveThis study aims to identify the data gathering patterns of final year medical students while using eCREST and how eCREST influences the patterns.MethodsA mixed methods design was used. A trial of eCREST across 3 UK medical schools (N=148) measured the potential effects of eCREST on data gathering. A qualitative think-aloud and semistructured interview study with 16 medical students from one medical school identified 3 data gathering strategies: Thorough, Focused, and Succinct. Some had no strategy. Reanalysis of the trial data identified the prevalence of data gathering patterns and compared patterns between the intervention and control groups. Patterns were identified based on 2 variables that were measured in a patient case 1 month after the intervention: the proportion of Essential information students identified and the proportion of irrelevant information gathered (Relevant). Those who scored in the top 3 quartiles for Essential but in the lowest quartile for Relevant displayed a Thorough pattern. Those who scored in the top 3 quartiles for Relevant but in the lowest quartile for Essential displayed a Succinct pattern. Those who scored in the top 3 quartiles on both variables displayed a Focused pattern. Those whose scores were in the lowest quartile on both variables displayed a Nonspecific pattern.ResultsThe trial results indicated that students in the intervention group were more thorough than those in the control groups when gathering data. The qualitative data identified data gathering strategies and the mechanisms by which eCREST influenced data gathering. Students reported that eCREST promoted thoroughness by prompting them to continuously reflect and allowing them to practice managing uncertainty. However, some found eCREST to be less useful, and they randomly gathered information. Reanalysis of the trial data revealed that the intervention group was significantly more likely to display a Thorough data gathering pattern than controls (21/78, 27% vs 6/70, 9%) and less likely to display a Succinct pattern (13/78, 17% vs 20/70, 29%; χ23=9.9; P=.02). Other patterns were similar across groups.ConclusionsQualitative data suggested that students applied a range of data gathering strategies while using eCREST and that eCREST encouraged thoroughness by continuously prompting the students to reflect and manage their uncertainty. Trial data suggested that eCREST led students to demonstrate more Thorough data gathering patterns. Virtual patients that encourage thoroughness could help future physicians avoid missed diagnostic opportunities and enhance the delivery of clinical reasoning teaching.