Evaluating the validity evidence of an OSCE: results from a new medical school.
Ontology highlight
ABSTRACT:
SUBMITTER: Yazbeck Karam V
PROVIDER: S-EPMC6302424 | biostudies-literature | 2018 Dec
REPOSITORIES: biostudies-literature
Publications
Evaluating the validity evidence of an OSCE: results from a new medical school.
Yazbeck Karam Vanda V Park Yoon Soo YS Tekian Ara A Youssef Nazih N
BMC medical education 20181220 1
<h4>Background</h4>To prevent the problems of traditional clinical evaluation, the "Objective Structured Clinical Examination (OSCE)" was presented by Harden as a more valid and reliable assessment instrument. However, an essential condition to guarantee a high-quality and effective OSCE is the assurance of evidence to support the validity of its scores. This study examines the psychometric properties of OSCE scores, with an emphasis on consequential and internal structure validity evidence.<h4> ...[more]
Similar Datasets
Project description:BackgroundMedical school admission procedures have the common goal to select applicants with the greatest potential of becoming successful physicians. Hamburg Medical Faculty selects medical students by grade point average (GPA) and employs a two-step selection process of a natural sciences test (HAM-Nat), in some cases followed by multiple mini-interviews (HAM-Int). Multiple mini-interviews can predict non-cognitive outcomes, while GPA has predictive validity for cognitive outcomes. The aim of our study was to explore communication skills and clinical knowledge of advanced medical students according to their respective admission procedure.MethodsIn July 2019, 146 students grouped according to their admission procedure into GPA-only (19.2 %), HAM-Nat (33.6 %), HAM-Int (30.8 %), and Waiting List (16.4 %) participated in four OSCE stations which equally assessed students' communication skills (OSCE part 1) and clinical knowledge (OSCE part 2) in simulated patient encounters, rated by physicians with checklists. Additionally, psychosocial assessors ranked communication skills with a global rating scale (GR). The students also participated in a multiple choice (MC) exam testing clinical knowledge. Kruskal-Wallis analyses of variance of test performance and Spearman correlation of instruments were calculated.ResultsStudents from the Waiting List group performed significantly worse on the MC exam compared to GPA-only and HAM-Int (adjusted p = .029 and 0.018, respectively). No significant differences were found between the admission groups with respect to communication skills. Global Rating and OSCE part 1 (communication) correlated significantly (ρ = 0.228, p = .006) as did OSCE part 2 (clinical knowledge) and MC exam (ρ = 0.242, p = .003), indicating criterion validity. Constructs did not overlap, indicating divergent validity.ConclusionsAdvanced medical students selected for undergraduate studies by multiple mini-interviews assessing psychosocial skills showed similar communication skills compared to students admitted to medical school by other entryways. It is unclear whether these similarities are due to an effective undergraduate longitudinal communication curriculum. Assessing baseline communication skills of all medical students at entry-level may aid with this question.
| S-EPMC8103591 | biostudies-literature
Project description:BackgroundAlthough used widely, there is limited evidence of the BioMedical Admissions Test's (BMAT) predictive validity and incremental validity over prior educational attainment (PEA). We investigated BMAT's predictive and incremental validity for performance in two undergraduate medical schools, Imperial College School of Medicine (ICSM), UK, and Lee Kong Chian School of Medicine (LKCMedicine), Singapore. Our secondary goal was to compare the evidence collected with published evidence relating to comparable tools.MethodsThis was a retrospective cohort study of four ICSM (1188 students, entering 2010-2013) and three LKCMedicine cohorts (222 students, 2013-2015). We investigated associations between BMAT Section 1 ('Thinking Skills'), Section 2 ('Scientific Knowledge and Applications') and Section 3a ('Writing Task') scores, with written and clinical assessment performance across all programme years. Incremental validity was investigated over PEA (A-levels) in a subset of ICSM students.ResultsWhen BMAT sections were investigated independently, Section 2 scores predicted performance on all written assessments in both institutions with mainly small effect sizes (standardised coefficient ranges: ICSM: 0.08-0.19; LKCMedicine: 0.22-0.36). Section 1 scores predicted Years 5 and 6 written assessment performance at ICSM (0.09-0.14) but nothing at LKCMedicine. Section 3a scores only predicted Year 5 clinical assessment performance at ICSM with a coefficient <0.1. There were no positive associations with standardised coefficients >0.1 between BMAT performance and clinical assessment performance. Multivariable regressions confirmed that Section 2 scores were the most predictive. We found no clear evidence of incremental validity for any BMAT section scores over A-level grades.DiscussionSchools who wish to assess scientific knowledge independently of A-levels may find BMAT Section 2 useful. Comparison with previous studies indicates that, overall, BMAT seems less useful than comparable tools. Larger scale studies are needed. Broader questions regarding why institutions adopt certain admissions tests, including those with little evidence, need consideration.
| S-EPMC9545404 | biostudies-literature
Project description:Recent advances in large language models (LLMs) have demonstrated remarkable successes in zero- and few-shot performance on various downstream tasks, paving the way for applications in high-stakes domains. In this study, we systematically examine the capabilities and limitations of LLMs, specifically GPT-3.5 and ChatGPT, in performing zero-shot medical evidence summarization across six clinical domains. We conduct both automatic and human evaluations, covering several dimensions of summary quality. Our study demonstrates that automatic metrics often do not strongly correlate with the quality of summaries. Furthermore, informed by our human evaluations, we define a terminology of error types for medical evidence summarization. Our findings reveal that LLMs could be susceptible to generating factually inconsistent summaries and making overly convincing or uncertain statements, leading to potential harm due to misinformation. Moreover, we find that models struggle to identify the salient information and are more error-prone when summarizing over longer textual contexts.
| S-EPMC10449915 | biostudies-literature
Project description:BACKGROUND:Mentorship plays an essential role in enhancing the success of junior faculty. Previous evaluation tools focused on specific types of mentors or mentees. The main objective was to develop and provide validity evidence for a Mentor Evaluation Tool (MET) to assess the effectiveness of one-on-one mentoring for faculty in the academic health sciences. METHODS:Evidence was collected for the validity domains of content, internal structure and relationship to other variables. The 13 item MET was tested for internal structure evidence with 185 junior faculty from Schools of Dentistry, Medicine, Nursing, and Pharmacy. Finally, the MET was studied for additional validity evidence by prospectively enrolling mentees of three different groups of faculty (faculty nominated for, or winners of, a lifetime achievement in mentoring award; faculty graduates of a mentor training program; and faculty mentors not in either of the other two groups) at the University of California San Francisco (UCSF) and asking them to rate their mentors using the MET. Mentors and mentees were clinicians, educators and/or researchers. RESULTS:The 13 MET items mapped well to the five mentoring domains and six competencies described in the literature. The standardized Cronbach's coefficient alpha was 0.96. Confirmatory factor analysis supported a single factor (CFI = 0.89, SRMR = 0.05). The three mentor groups did not differ in the single overall assessment item (P = 0.054) or mean MET score (P = 0.288), before or after adjusting for years of mentoring. The mentorship score means were relatively high for all three groups. CONCLUSIONS:The Mentor Evaluation Tool demonstrates evidence of validity for research, clinical, educational or career mentors in academic health science careers. However, MET did not distinguish individuals nominated as outstanding mentors from other mentors. MET validity evidence can be studied further with mentor-mentee pairs and to follow prospectively the rating of mentors before and after a mentorship training program.
| S-EPMC7297334 | biostudies-literature
Project description:PurposeDeveloping medical students' clinical reasoning requires a structured longitudinal curriculum with frequent targeted assessment and feedback. Performance-based assessments, which have the strongest validity evidence, are currently not feasible for this purpose because they are time-intensive to score. This study explored the potential of using machine learning technologies to score one such assessment-the diagnostic justification essay.MethodFrom May to September 2018, machine scoring algorithms were trained to score a sample of 700 diagnostic justification essays written by 414 third-year medical students from the Southern Illinois University School of Medicine classes of 2012-2017. The algorithms applied semantically based natural language processing metrics (e.g., coherence, readability) to assess essay quality on 4 criteria (differential diagnosis, recognition and use of findings, workup, and thought process); the scores for these criteria were summed to create overall scores. Three sources of validity evidence (response process, internal structure, and association with other variables) were examined.ResultsMachine scores correlated more strongly with faculty ratings than faculty ratings did with each other (machine: .28-.53, faculty: .13-.33) and were less case-specific. Machine scores and faculty ratings were similarly correlated with medical knowledge, clinical cognition, and prior diagnostic justification. Machine scores were more strongly associated with clinical communication than were faculty ratings (.43 vs .31).ConclusionsMachine learning technologies may be useful for assessing medical students' long-form written clinical reasoning. Semantically based machine scoring may capture the communicative aspects of clinical reasoning better than faculty ratings, offering the potential for automated assessment that generalizes to the workplace. These results underscore the potential of machine scoring to capture an aspect of clinical reasoning performance that is difficult to assess with traditional analytic scoring methods. Additional research should investigate machine scoring generalizability and examine its acceptability to trainees and educators.
| S-EPMC8243833 | biostudies-literature
Project description:IntroductionThere is an increasing call for developing validity evidence in medical education assessment. The literature lacks a practical resource regarding an actual development process. Our workshop teaches how to apply principles of validity evidence to existing assessment instruments and how to develop new instruments that will yield valid data.MethodsThe literature, consensus findings of curricula and content experts, and principles of adult learning guided the content and methodology of the workshop. The workshop underwent stringent peer review prior to presentation at one international and three national academic conferences. In the interactive workshop, selected domains of validity evidence were taught with sequential cycles of didactics, demonstration, and deliberate practice with facilitated feedback. An exercise guide steered participants through a stepwise approach. Using Likert-scale items and open-response questions, an evaluation form rated the workshop's effectiveness, captured details of how learners reached the objectives, and determined participants' plans for future work.ResultsThe workshop demonstrated generalizability with successful implementation in diverse settings. Sixty-five learners, the majority being clinician-educators, completed evaluations. Learners rated the workshop favorably for each prompt. Qualitative comments corroborated the workshop's effectiveness. The active application and facilitated feedback components allowed learners to reflect in real time as to how they were meeting a particular objective.DiscussionThis feasible and practical educational intervention fills a literature gap by showing the medical educator how to apply validity evidence to both existing and in-development assessment instruments. Thus, it holds the potential to significantly impact learner and, subsequently, patient outcomes.
| S-EPMC6507922 | biostudies-literature
Project description:BackgroundEvidence-based medicine (EBM) is an important element of medical education. However, limited information is available on effective curricula.ObjectiveTo evaluate a longitudinal medical school EBM curriculum using validated instruments.Design, participants, measurementsWe evaluated EBM attitudes and knowledge of medical students as they progressed through an EBM curriculum. The first component of the curriculum was an EBM "short course" with didactic and small-group sessions occurring at the end of the second year. The second component integrated EBM assignments with third-year clinical rotations. The 15-point Berlin Questionnaire was administered before the course in 2006 and 2007, after the short course, and at the end of the third year. The 212-point Fresno Test was administered before the course in 2007 and 2008, after the short course, and at the end of the third year. Self-reported knowledge and attitudes were also assessed in all three classes of medical students.ResultsEBM knowledge scores on the 15-point Berlin Questionnaire increased from baseline by 3.0 points (20.0%) at the end of the second year portion of the course (p < 001) and by 3.4 points (22.7%) at the end of the third year (p < 001). EBM knowledge scores on the 212-point Fresno Test increased from baseline by 39.7 points (18.7%) at the end of the second year portion of the course (p < 001) and by 54.6 points (25.8%) at the end of the third year (p < 001). On a 5-point scale, self-rated EBM knowledge increased from baseline by 1.0 and 1.4 points, respectively (both p < 001). EBM was felt to be of high importance for medical education and clinical practice at all time points, with increases noted after both components of the curriculum.ConclusionsA longitudinal medical school EBM was associated with markedly increased EBM knowledge on two validated instruments. Both components of the curriculum were associated with gains in knowledge. The curriculum was also associated with increased perceptions of the importance of EBM for medical education and clinical practice.
| S-EPMC3101983 | biostudies-literature
Project description:The Arnett Caregiver Interaction Scale (CIS) has been widely used in research studies to measure the quality of caregiver-child interactions. The scale was modeled on a well-established theory of parenting, but there are few psychometric studies of its validity. We applied factor analyses and item response theory methods to assess the psychometric properties of the Arnett CIS in a national sample of toddlers in home-based care and preschoolers in center-based care from the Early Childhood Longitudinal Study-Birth Cohort. We found that a bifactor structure (one common factor and a second set of specific factors) best fits the data. In the Arnett CIS, the bifactor model distinguishes a common substantive dimension from two methodological dimensions (for positively and negatively oriented items). Despite the good fit of this model, the items are skewed (most teachers/caregivers display positive interactions with children) and, as a result, the Arnett CIS is not well suited to distinguish between caregivers who are "highly" versus "moderately" positive in their interactions with children, according to the items on the scale. Regression-adjusted associations between the Arnett CIS and child outcomes are small, especially for preschoolers in centers. We encourage future scale development work on measures of child care quality by early childhood scholars.
| S-EPMC3775379 | biostudies-literature
Project description:RATIONALE:Non-medical prescription opioid use and opioid use disorder (OUD) present a significant public health concern. Identifying behavioral mechanisms underlying OUD will assist in developing improved prevention and intervention approaches. Behavioral economic demand has been extensively evaluated as a measure of reinforcer valuation for alcohol and cigarettes, whereas prescription opioids have received comparatively little attention. OBJECTIVES:Utilize a purchase task procedure to measure the incremental validity and test-retest reliability of opioid demand. METHODS:Individuals reporting past year non-medical prescription opioid use were recruited using the crowdsourcing platform Amazon Mechanical Turk (mTurk). Participants completed an opioid purchase task as well as measures of cannabis demand, delay discounting, and self-reported pain. A 1-month follow-up was used to evaluate test-retest reliability. RESULTS:More intense and inelastic opioid demand was associated with OUD and more intense cannabis demand was associated with cannabis use disorder. Multivariable models indicated that higher opioid intensity and steeper opioid delay discounting rates each significantly and uniquely predicted OUD. Increased opioid demand intensity, but not elasticity, was associated with higher self-reported pain, and no relationship was observed with perceived pain relief from opioids. Opioid demand showed acceptable-to-good test-retest reliability (e.g., intensity rxx = .75; elasticity rxx = .63). Temporal reliability was lower for cannabis demand (e.g., intensity rxx = .53; elasticity rxx = .58) and discounting rates (rxx = .42-.61). CONCLUSIONS:Opioid demand was incrementally valid and test-retest reliable as measured by purchase tasks. These findings support behavioral economic demand as a clinically useful measure of drug valuation that is sensitive to individual difference variables.
| S-EPMC6990908 | biostudies-literature
Project description:ContextEmpathy is an essential aspect of clinical care, associated with improved patient satisfaction, increased adherence to treatment, and fewer malpractice complaints. Previous studies suggest that empathy declines during medical training. However, past research relied on a single narrowly operationalised, self-report measure of empathy. As empathy is a complex socio-emotional construct, it is critical to assess changes across its distinct components using multiple measures in order to better understand how it is influenced by medical training.MethodsIn a longitudinal study, medical students completed a series of self-report and behavioural measures twice per year during the first 3 years of their study (2012-2015). These included the previously used Jefferson Scale of Physician Empathy (JSPE), designed to assess empathy in the clinical context, the Questionnaire of Cognitive and Affective Empathy (QCAE), designed to assess overall empathy and its main components, and behavioural measures of sensitivity to others' pain and understanding of others' emotions, both of which are important aspects of empathy. The employment of multiple measures allowed for a more complete assessment of medical students' empathy and related processes.ResultsIn reflection of findings in previous work, students' empathy assessed by the JSPE decreased over training. However, on the QCAE, aspects of students' empathy, specifically overall cognitive empathy and its subcomponent perspective taking, and the emotion contagion subcomponent of affective empathy improved, whereas the remaining subcomponents remained stable. During medical school, students also exhibited comparable growth in their understanding of others' emotions and increased sensitivity to others' pain.ConclusionsChanges in empathy during medical school cannot be simply characterised as representing an overall decline. Indeed, aspects of empathy thought to be valuable in positive physician-patient interactions improve during training. Overall, this study points to the importance of assessing the distinct components of empathy using multiple forms of measurement in order to better understand the mechanisms involved in empathy changes in medical practice.
| S-EPMC5657391 | biostudies-literature