Four tenets of modern validity theory for medical education assessment and evaluation.
Ontology highlight
ABSTRACT:
SUBMITTER: Royal KD
PROVIDER: S-EPMC5546594 | biostudies-other | 2017
REPOSITORIES: biostudies-other
Similar Datasets
Project description:IntroductionThere is an increasing call for developing validity evidence in medical education assessment. The literature lacks a practical resource regarding an actual development process. Our workshop teaches how to apply principles of validity evidence to existing assessment instruments and how to develop new instruments that will yield valid data.MethodsThe literature, consensus findings of curricula and content experts, and principles of adult learning guided the content and methodology of the workshop. The workshop underwent stringent peer review prior to presentation at one international and three national academic conferences. In the interactive workshop, selected domains of validity evidence were taught with sequential cycles of didactics, demonstration, and deliberate practice with facilitated feedback. An exercise guide steered participants through a stepwise approach. Using Likert-scale items and open-response questions, an evaluation form rated the workshop's effectiveness, captured details of how learners reached the objectives, and determined participants' plans for future work.ResultsThe workshop demonstrated generalizability with successful implementation in diverse settings. Sixty-five learners, the majority being clinician-educators, completed evaluations. Learners rated the workshop favorably for each prompt. Qualitative comments corroborated the workshop's effectiveness. The active application and facilitated feedback components allowed learners to reflect in real time as to how they were meeting a particular objective.DiscussionThis feasible and practical educational intervention fills a literature gap by showing the medical educator how to apply validity evidence to both existing and in-development assessment instruments. Thus, it holds the potential to significantly impact learner and, subsequently, patient outcomes.
| S-EPMC6507922 | biostudies-literature
Project description:INTRODUCTION:Assessment in Medical Education fills many roles and is under constant scrutiny. Assessments must be of good quality, and supported by validity evidence. Given the high-stakes consequences of assessment, and the many audiences within medical education (e. g., training level, specialty-specific), we set out to document the breadth, scope, and characteristics of the literature reporting on validation of assessments within medical education. METHOD:Searches in Medline (Ovid), Web of Science, ERIC, EMBASE (Ovid), and PsycINFO (Ovid) identified articles reporting on assessment of learners in medical education published since 1999. Included articles were coded for geographic origin, journal, journal category, targeted assessment, and authors. A map of collaborations between prolific authors was generated. RESULTS:A total of 2,863 articles were included. The majority of articles were from the United States, with Canada producing the most articles per medical school. Most articles were published in journals with medical categorizations (73.1% of articles), but Medical Education was the most represented journal (7.4% of articles). Articles reported on a variety of assessment tools and approaches, and 89 prolific authors were identified, with a total of 228 collaborative links. DISCUSSION:Literature reporting on validation of assessments in medical education is heterogeneous. Literature is produced by a broad array of authors and collaborative networks, reported to a broad audience, and is primarily generated in North American and European contexts. Our findings speak to the heterogeneity of the medical education literature on assessment validation, and suggest that this heterogeneity may stem, at least in part, from differences in constructs measured, assessment purposes, or conceptualizations of validity.
| S-EPMC6002290 | biostudies-literature
Project description:Core competencies have progressively gained importance in medical education. In other contexts, especially personnel selection and development, assessment centers (ACs) are used to assess competencies, but there is only a limited number of studies on competency-based ACs in medical education. To the best of our knowledge, the present study provides the first data on the criterion-related validity of a competency-based AC in medical education.We developed an AC tailored to measure core competencies relevant to medical education (social-ethical, communicative, self, and teaching) and tested its validity in n=30 first-year medical students using 3- to 4-year follow-up measures such as (a) objective structured clinical examinations (OSCE) on basic clinical skills (n=26), (b) OSCE on communication skills (n=21), and (c) peer feedback (n=18). The AC contained three elements: interview, group discussion, and role play. Additionally, a self-report questionnaire was provided as a basis for the interview.Baseline AC average score and teaching competency correlated moderately with the communication OSCE average score (r=0.41, p=0.03, and r=0.38, p=0.04, respectively). Social-ethical competency in the AC showed a very strong convergent association with the communication OSCE average score (r=0.60, p<0.01). The AC total score also showed a moderate correlation with the overall peer feedback score provided in Year 4 (r=0.38, p=0.06). In addition, communicative competency correlated strongly with the overall peer feedback (r=0.50, p=0.02). We found predominantly low and insignificant correlations between the AC and the OSCE on basic clinical skills (r=-0.33 to 0.30, all p's>0.05).The results showed that competency-based ACs can be used at a very early stage of medical training to successfully predict future performance in core competencies.
| S-EPMC4163580 | biostudies-other
Project description:BackgroundThe objective of this study is to reveal on the basis of the item response theory (IRT) the validity and reliability evidence for the data obtained from the scale prepared to determine the satisfaction with distance education in students studying in medical schools.MethodsThis is a quantitative study exploring IRT and measurement invariance evidence in developing a scale. The scale whose IRT evidence was explored was the Distance Education Satisfaction Scale (DESS). The data were obtained from 1332 medical school students who were studying at various universities. The data were analysed using the confirmatory factor analysis (CFA), multidimensional and unidimensional IRT, and measurement invariance.ResultsA 20-item construct with 3 sub-factors was found for DESS. This construct was unable to pass the iteration limit in the multidimensional IRT analysis. A unidimensional IRT was used assuming that the 3 sub-factors were locally independent.ConclusionsThe least informative items were item 23, 24 and 25 in Factor 1, item 3 in Factor 2, and items 13 and 18 in Factor 3. The most informative items of DESS were those that had adaptive, useful expressions that had meaningful content and were able to provide educator support, which are the properties emphasized in the literature with respect to satisfaction with distance education. A measurement invariance test made based on gender revealed that DESS satisfied measurement invariance by meeting the compliance indexes required for configural, metric, scalar and strict invariance as recommended in the literature. The results showed that it is possible to make comparisons on the basis of gender using DESS.
| S-EPMC8831879 | biostudies-literature
Project description:BACKGROUND: Deficiencies in medical education research quality are widely acknowledged. Content, internal structure, and criterion validity evidence support the use of the Medical Education Research Study Quality Instrument (MERSQI) to measure education research quality, but predictive validity evidence has not been explored. OBJECTIVE: To describe the quality of manuscripts submitted to the 2008 Journal of General Internal Medicine (JGIM) medical education issue and determine whether MERSQI scores predict editorial decisions. DESIGN AND PARTICIPANTS: Cross-sectional study of original, quantitative research studies submitted for publication. MEASUREMENTS: Study quality measured by MERSQI scores (possible range 5-18). RESULTS: Of 131 submitted manuscripts, 100 met inclusion criteria. The mean (SD) total MERSQI score was 9.6 (2.6), range 5-15.5. Most studies used single-group cross-sectional (54%) or pre-post designs (32%), were conducted at one institution (78%), and reported satisfaction or opinion outcomes (56%). Few (36%) reported validity evidence for evaluation instruments. A one-point increase in MERSQI score was associated with editorial decisions to send manuscripts for peer review versus reject without review (OR 1.31, 95%CI 1.07-1.61, p = 0.009) and to invite revisions after review versus reject after review (OR 1.29, 95%CI 1.05-1.58, p = 0.02). MERSQI scores predicted final acceptance versus rejection (OR 1.32; 95% CI 1.10-1.58, p = 0.003). The mean total MERSQI score of accepted manuscripts was significantly higher than rejected manuscripts (10.7 [2.5] versus 9.0 [2.4], p = 0.003). CONCLUSIONS: MERSQI scores predicted editorial decisions and identified areas of methodological strengths and weaknesses in submitted manuscripts. Researchers, reviewers, and editors might use this instrument as a measure of methodological quality.
| S-EPMC2517948 | biostudies-literature
Project description:Medical education is a messy tangle of social and material elements. These material entities include tools, like curriculum guides, stethoscopes, cell phones, accreditation standards, and mannequins; natural elements, like weather systems, disease vectors, and human bodies; and, objects, like checklists, internet connections, classrooms, lights, chairs and an endless array of others.We propose that sociomaterial approaches to ethnography can help us explore taken for granted, or under-theorized, elements of a situation under study, thereby enabling us to think differently. In this article, we describe ideas informing Actor-Network Theory approaches, and how these ideas translate into how ethnographic research is designed and conducted. We investigate epistemological (what we can know, and how) positioning of the researcher in an actor-network theory informed ethnography, and describe how we tailor ethnographic methods-document and artefact analysis; observation; and interviews-to align with a sociomaterial worldview.Untangling sociomaterial scenarios can offer a novel perspective on myriad contemporary medical education issues. These issues include examining how novel tools (e.g. accreditation standards, assessment tools, mannequins, videoconferencing technologies) and spaces (e.g. simulation suites, videoconferenced lecture theatres) used in medical education impact how teaching and learning actually happen in these settings.
| S-EPMC6565649 | biostudies-literature
Project description:Fusion validity assessments employ structural equation models to investigate whether an existing scale functions in accordance with theory. Fusion validity parallels criterion validity by depending on correlations with non-scale variables but differs from criterion validity because it requires at least one theorized effect of the scale, and because both the scale and scaled-items are included in the model. Fusion validity, like construct validity, will be most informative if the scale is embedded in as full a substantive context as theory permits. Appropriate scale functioning in a comprehensive theoretical context greatly enhances a scale's validity. Inappropriate scale functioning questions the scale but the scale's theoretical embedding encourages detailed diagnostic investigations potentially challenging specific items, the procedure used to calculate scale values, or aspects of the theory, but also possibly recommends incorporating additional items into the scale. The scaled items should have survived prior content and methodological assessments but the items may or may not reflect a common factor because items having diverse causal backgrounds can sometimes fuse to form a unidimensional entity. Though items reflecting a common cause can be assessed for fusion validity, we illustrate fusion validity in the more challenging context of a scale comprised of diverse items and embedded in a complicated theory. Specifically we consider the Leadership scale from the Alberta Context Tool with care aides working in Canadian long-term care homes.
| S-EPMC6559122 | biostudies-literature
Project description:Cancer is the second leading cause of death in the USA, but there is minimal data on how oncology is taught to medical students. The purpose of this study is to characterize oncology education at US medical schools. An electronic survey was sent between December 2014 and February 2015 to a convenience sample of medical students who either attended the American Society for Radiation Oncology annual meeting or serve as delegates to the American Association of Medical Colleges. Information on various aspects of oncology instruction at participants' medical schools was collected. Seventy-six responses from students in 28 states were received. Among the six most common causes of death in the USA, cancer reportedly received the fourth most curricular time. During the first, second, and third years of medical school, participants most commonly reported 6-10, 16-20, and 6-10 h of oncology teaching, respectively. Participants were less confident in their understanding of cancer treatment than workup/diagnosis or basic science/natural history of cancer (p < 0.01). During the preclinical years, pathologists, scientists/Ph.D.'s, and medical oncologists reportedly performed the majority of teaching, whereas during the clinical clerkships, medical and surgical oncologists reportedly performed the majority of teaching. Radiation oncologists were significantly less involved during both periods (p < 0.01). Most schools did not require any oncology-oriented clerkship. During each mandatory rotation, <20 % of patients had a primary diagnosis of cancer. Oncology education is often underemphasized and fragmented with wide variability in content and structure between medical schools, suggesting a need for reform.
| S-EPMC5533172 | biostudies-literature
Project description:IntroductionPrior research has identified seven elements of a good assessment, but the elements have not been operationalized in the form of a rubric to rate assessment utility. It would be valuable for medical educators to have a systematic way to evaluate the utility of an assessment in order to determine if the assessment used is optimal for the setting.MethodsWe developed and refined an assessment utility rubric using a modified Delphi process. Twenty-nine graduate students pilot-tested the rubric in 2016 with hypothetical data from three examinations, and interrater reliability of rubric scores was measured with interclass correlation coefficients (ICCs).ResultsConsensus for all rubric items was reached after three rounds. The resulting assessment utility rubric includes four elements (equivalence, educational effect, catalytic effect, acceptability) with three items each, one element (validity evidence) with five items, and space to provide four feasibility items relating to time and cost. Rater scores had ICC values greater than .75.DiscussionThe rubric shows promise in allowing educators to evaluate the utility of an assessment specific to their setting. The medical education field needs to give more consideration to how an assessment drives learning forward, how it motivates trainees, and whether it produces acceptable ranges of scores for all stakeholders.
| S-EPMC6338154 | biostudies-literature
Project description:BackgroundSubstance misuse teaching within the undergraduate medical curriculum has been underrepresented compared to more traditional medical topics. In response several national curriculum reviews such as the most recent UK department of health initiative (DOH) have identified deficiencies in substance misuse education and have suggested curriculum interventions for local faculties to implement. The student perspective however has largely been muted during this process and this study aims to explore this using a constructivist grounded theory approach.MethodsEleven final year and intercalating medical students across three separate focus groups participated in this study over a three-month period commencing from March 2018. Time between the audio recorded focus groups allowed for a parallel process of data collection and analysis into more focussed codes and categories to occur, consistent with the grounded theory approach. The qualitative study took place in a single medical school in the UK.ResultsMedical students had a common consensus that substance misuse education was an underperforming subject in their curriculum, from limited teaching hours to curriculum design and organisational problems. Students identified an alternative curriculum is required to not only prepare students for their future clinical duties but also their own personal lives. Students highlighted this proximity to a 'dangerous world' where exposure to substance misuse risks were faced daily. This exposure also provided a source of informal learning experiences which students deemed as being potentially unbalanced and even dangerous. Students also identified unique barriers to curriculum change with reference to a lack of openness due to the impacts of disclosure in substance misuse.ConclusionLarge scale curriculum initiatives appear to correspond to the student voice obtained in this study, providing backing for the implementation of a co-ordinated substance misuse curriculum within medical schools. The student voice however provides an alternative lens by outlining how substance misuse pervades into students' lives and how informal learning is a largely underestimated hidden source of learning with more dangers than benefits. This together with the identification of further barriers to curriculum change, provide space for medical faculties to incorporate and work with students to facilitate local level curriculum changes relating to substance misuse education.
| S-EPMC10068233 | biostudies-literature