Browse
Submit Data
Databases
API
Help

Dataset Information

16 Views

0 Connections

0 Citations

0 Reanalyses

0 Downloads

Omics score: 0

Assessment methods and the validity and reliability of measurement tools in online objective structured clinical examinations: a systematic scoping review

ABSTRACT: The coronavirus disease 2019 (COVID-19) pandemic has required educators to adapt the in-person objective structured clinical examination (OSCE) to online settings in order for it to remain a critical component of the multifaceted assessment of a student’s competency. This systematic scoping review aimed to summarize the assessment methods and validity and reliability of the measurement tools used in current online OSCE (hereafter, referred to as teleOSCE) approaches. A comprehensive literature review was undertaken following the Preferred Reporting Items for Systematic Reviews and Meta-Analyses extension for Scoping Reviews guidelines. Articles were eligible if they reported any form of performance assessment, in any field of healthcare, delivered in an online format. Two reviewers independently screened the results and analyzed relevant studies. Eleven articles were included in the analysis. Pre-recorded videos were used in 3 studies, while observations by remote examiners through an online platform were used in 7 studies. Acceptability as perceived by students was reported in 2 studies. This systematic scoping review identified several insights garnered from implementing teleOSCEs, the components transferable from telemedicine, and the need for systemic research to establish the ideal teleOSCE framework. TeleOSCEs may be able to improve the accessibility and reproducibility of clinical assessments and equip students with the requisite skills to effectively practice telemedicine in the future.

SUBMITTER: Felthun JZ

PROVIDER: S-EPMC8212027 | biostudies-literature |

REPOSITORIES: biostudies-literature

ACCESS DATA

Json Xml

Similar Datasets

A Comparison of Assessment Tools: Is Direct Observation an Improvement Over Objective Structured Clinical Examinations for Communications Skills Evaluation?

Project description:Background :Evaluation of resident physicians' communications skills is a challenging task and is increasingly accomplished with standardized examinations. There exists a need to identify the effective, efficient methods for assessment of communications skills. Objective :We compared objective structured clinical examination (OSCE) and direct observation as approaches for assessing resident communications skills. Methods :We conducted a retrospective cohort analysis of orthopaedic surgery resident physicians at a single tertiary care academic institution, using the Institute for Healthcare Communication "4 Es" model for effective communication. Data were collected between 2011 and 2015. A total of 28 residents, each with OSCE and complete direct observation assessment checklists, were included in the analysis. Residents were included if they had 1 OSCE assessment and 2 or more complete direct observation assessments. Results :There were 28 of a possible 59 residents (47%) included. A total of 89% (25 of 28) of residents passed the communications skills OSCE; only 54% (15 of 28) of residents passed the direct observation communications assessment. There was a positive, moderate correlation between OSCE and direct observation scores overall (r = 0.415, P = .028). There was no agreement between OSCE and direct observation in categorizing residents into passing and failing scores (κ = 0.205, P = .16), after adjusting for chance agreement. Conclusions :Our results suggest that OSCE and direct observation tools provide different insights into resident communications skills (simulation of rare and challenging situations versus real-life daily encounters), and may provide useful perspectives on resident communications skills in different contexts.

| S-EPMC5901804 | biostudies-other

The sights and insights of examiners in objective structured clinical examinations.

Project description:The objective structured clinical examination (OSCE) is considered to be one of the most robust methods of clinical assessment. One of its strengths lies in its ability to minimise the effects of examiner bias due to the standardisation of items and tasks for each candidate. However, OSCE examiners' assessment scores are influenced by several factors that may jeopardise the assumed objectivity of OSCEs. To better understand this phenomenon, the current review aims to determine and describe important sources of examiner bias and the factors affecting examiners' assessments.We performed a narrative review of the medical literature using Medline. All articles meeting the selection criteria were reviewed, with salient points extracted and synthesised into a clear and comprehensive summary of the knowledge in this area.OSCE examiners' assessment scores are influenced by factors belonging to 4 different domains: examination context, examinee characteristics, examinee-examiner interactions, and examiner characteristics. These domains are composed of several factors including halo, hawk/dove and OSCE contrast effects; the examiner's gender and ethnicity; training; lifetime experience in assessing; leadership and familiarity with students; station type; and site effects.Several factors may influence the presumed objectivity of examiners' assessments, and these factors need to be addressed to ensure the objectivity of OSCEs. We offer insights into directions for future research to better understand and address the phenomenon of examiner bias.

| S-EPMC5801428 | biostudies-literature

Improving Pediatric Fellows' Feedback Skills and Confidence Through Objective Structured Examinations.

Project description:Background Medical trainees must learn how to provide effective feedback as an essential communication skill, yet few models exist for training and assessing these skills. Objective To develop an observed structured feedback examination (OSFE) to provide feedback training to pediatric fellows and assess changes in skills and self-reported confidence. Methods This educational study was conducted from 2019 to 2020 at an academic children's hospital. Our team developed the OSFE and trained standardized feedback recipients and faculty. Fellows completed baseline self-assessments (31 items) on prior exposure to feedback training, application of skills, and confidence. They then participated in the OSFE, giving feedback to a standardized recipient using a standardized scenario, and were scored by faculty and recipients using a 15-item checklist for performance. Next, fellows participated in feedback training and received individualized feedback, after which they repeated the OSFE and confidence self-assessment. Three months later, fellows completed self-assessments on confidence and application of skills and another OSFE to assess retention. Descriptive statistics and signed rank sum test were used for analysis. Results Of 60 eligible fellows, 19 participated (32%), with 100% follow-up. After training and individualized feedback, all fellows improved feedback skills as measured by OSFE performance (mean change +0.89). All items, measured on a 5-point Likert scale, were sustained 3 months later (mean change +0.92). All fellows reported improved confidence in feedback knowledge (mean change +2.07 post, +1.67 3 months post). Conclusions Feedback training using simulation and individualized feedback moderately improved fellows' performance, confidence, and 3-month retention of feedback skills.

| S-EPMC10829923 | biostudies-literature

Evaluation of Online Near-Peer Teaching for Penultimate-Year Objective Structured Clinical Examinations in the COVID-19 Era: Longitudinal Study.

Project description:BackgroundThe benefits of near-peer learning are well established in several aspects of undergraduate medical education including preparing students for Objective Structured Clinical Examinations (OSCEs). The COVID-19 pandemic has resulted in a paradigm shift to predominantly online teaching.ObjectiveThis study aims to demonstrate the feasibility and benefits of an exclusively online near-peer OSCE teaching program in a time of significant face-to-face and senior-led teaching shortage.MethodsA teaching program was delivered to penultimate-year students by final-year students at Manchester Medical School. Program development involved compiling a list of salient topics and seeking senior faculty approval. Teachers and students were recruited on Facebook. In total, 22 sessions and 42 talks were attended by 72 students and taught by 13 teachers over a 3-month period. Data collection involved anonymous weekly questionnaires and 2 separate anonymous student and teacher postcourse questionnaires including both quantitative and qualitative components.ResultsOn a scale of 1-10, students rated the quality of the program highly (mean 9.30, SD 1.15) and felt the sessions were highly useful in guiding their revision (mean 8.95, SD 0.94). There was a significant increase in perceived confidence ratings after delivery of the program (P<.001). Teachers felt the program helped them better understand and retain the subject material taught (mean 9.36, SD 0.81) and develop skills to become effective clinical teachers (mean 9.27, SD 0.79).ConclusionsThis is the first study demonstrating the efficacy of a near-peer OSCE teaching program delivered exclusively online. This provides an exemplary framework for how similar programs should be encouraged given their efficacy and logistical viability in supplementing the undergraduate curriculum.

| S-EPMC9185334 | biostudies-literature

Neurology objective structured clinical examination reliability using generalizability theory.

Project description:ObjectivesThis study examines factors affecting reliability, or consistency of assessment scores, from an objective structured clinical examination (OSCE) in neurology through generalizability theory (G theory).MethodsData include assessments from a multistation OSCE taken by 194 medical students at the completion of a neurology clerkship. Facets evaluated in this study include cases, domains, and items. Domains refer to areas of skill (or constructs) that the OSCE measures. G theory is used to estimate variance components associated with each facet, derive reliability, and project the number of cases required to obtain a reliable (consistent, precise) score.ResultsReliability using G theory is moderate (Φ coefficient = 0.61, G coefficient = 0.64). Performance is similar across cases but differs by the particular domain, such that the majority of variance is attributed to the domain. Projections in reliability estimates reveal that students need to participate in 3 OSCE cases in order to increase reliability beyond the 0.70 threshold.ConclusionsThis novel use of G theory in evaluating an OSCE in neurology provides meaningful measurement characteristics of the assessment. Differing from prior work in other medical specialties, the cases students were randomly assigned did not influence their OSCE score; rather, scores varied in expected fashion by domain assessed.

| S-EPMC4642144 | biostudies-literature

Rater training for standardised assessment of Objective Structured Clinical Examinations in rural Tanzania.

Project description:ObjectivesTo describe a simulation-based rater training curriculum for Objective Structured Clinical Examinations (OSCEs) for clinician-based training for frontline staff caring for mothers and babies in rural Tanzania.BackgroundRater training for OSCE evaluation is widely embraced in high-income countries but not well described in low-income and middle-income countries. Helping Babies Breathe, Essential Care for Every Baby and Bleeding after Birth are standardised training programmes that encourage OSCE evaluations. Studies examining the reliability of assessments are rare.MethodsTraining of raters occurred over 3 days. Raters scored selected OSCEs role-played using standardised learners and low-fidelity mannikins, assigning proficiency levels a priori. Researchers used Zabar's criteria to critique rater agreement and mitigate measurement error during score review. Descriptive statistics, Fleiss' kappa and field notes were used to describe results.ResultsSix healthcare providers scored 42 training scenarios. There was moderate rater agreement across all OSCEs (κ=0.508). Kappa values increased with Helping Babies Breathe (κ=0.28-0.48) and Essential Care for Every Baby (κ=0.42-0.77) by day 3 of training, but not with Bleeding after Birth (κ=0.58-0.33). Raters identified average proficiency 50% of the time.ConclusionOur study shows that the in-country raters in this study had a hard time identifying average performance despite moderate rater agreement. Rater training is critical to ensure that the potential of training programmes translates to improved outcomes for mothers and babies; more research into the concepts and training for discernment of competence in this setting is necessary.

| S-EPMC7722831 | biostudies-literature

Validity and reliability of the Standardized Orthopedic Assessment Tool (SOAT): a variation of the traditional objective structured clinical examination.

Project description:CONTEXT: Health care professions have replaced traditional multiple choice tests or essays with structured and practical, performance-based examinations with the hope of eliminating rater bias and measuring clinical competence. OBJECTIVE: To establish the validity and reliability of the Standardized Orthopedic Assessment Tool (SOAT) as a measure of clinical competence of orthopaedic injury evaluation. DESIGN: Descriptive laboratory study. SETTING: University. PATIENTS OR OTHER PARTICIPANTS: A total of 60 undergraduate students and 11 raters from 3 Canadian universities and 1 standardized patient. INTERVENTION(S): Students were required to complete a 30-minute musculoskeletal evaluation in 1 of 2 randomly assigned mock scenarios involving the knee (second-degree medial collateral ligament sprain) or the shoulder (third-degree supraspinatus muscle strain). MAIN OUTCOME MEASURE(S): We measured interreliability with an intraclass correlation coefficient (ICC) (2,k) and stability of the tool with standard error of measurement and confidence intervals. Agreement was measured using Bland-Altman plots. Concurrent validity was measured using a Pearson product moment correlation coefficient whereby the raters' global rating of a student was matched to the cumulative mean grade score. RESULTS: The ICCs were 0.75 and 0.82 for the shoulder and knee cases, respectively. Bland-Altman plots indicated no systematic bias between raters. In addition, Pearson product moment correlation analysis demonstrated a strong relationship between the overall cumulative mean grade score and the global rating score of the examinees' performances. CONCLUSIONS: This study demonstrated good interrater reliability of the SOAT with a standard error of measurement that indicated very modest stability, strong agreement between raters, and correlation indicative of concurrent validity.

| S-EPMC4080604 | biostudies-other

Construct Validity and Generalizability of Simulation-Based Objective Structured Clinical Examination Scenarios.

Project description:BackgroundIt is not known if construct-related validity (progression of scores with different levels of training) and generalizability of Objective Structured Clinical Examination (OSCE) scenarios previously used with non-US graduating anesthesiology residents translate to a US training program.ObjectiveWe assessed for progression of scores with training for a validated high-stakes simulation-based anesthesiology examination.MethodsFifty US anesthesiology residents in postgraduate years (PGYs) 2 to 4 were evaluated in operating room, trauma, and resuscitation scenarios developed for and used in a high-stakes Israeli Anesthesiology Board examination, requiring a score of 70% on the checklist for passing (including all critical items).ResultsThe OSCE error rate was lower for PGY-4 than PGY-2 residents in each field, and for most scenarios within each field. The critical item error rate was significantly lower for PGY-4 than PGY-3 residents in operating room scenarios, and for PGY-4 than PGY-2 residents in resuscitation scenarios. The final pass rate was significantly higher for PGY-3 and PGY-4 than PGY-2 residents in operating room scenarios, and also was significantly higher for PGY-4 than PGY-2 residents overall. PGY-4 residents had a better error rate, total scenarios score, general evaluation score, critical items error rate, and final pass rate than PGY-2 residents.ConclusionsThe comparable error rates, performance grades, and pass rates for US PGY-4 and non-US (Israeli) graduating (PGY-4 equivalent) residents, and the progression of scores among US residents with training level, demonstrate the construct-related validity and generalizability of these high-stakes OSCE scenarios.

| S-EPMC4535213 | biostudies-literature

Evaluating Shared Decision Making in Trial of Labor After Cesarean Counseling Using Objective Structured Clinical Examinations.

Project description:IntroductionAlthough shared decision making (SDM) is optimal for trial of labor after cesarean (TOLAC) counseling, resources to assess residents' clinical competency and communication skills are lacking. We addressed this gap by developing and testing an objective structured clinical examination (OSCE) to evaluate whether learners were able to use SDM in TOLAC counseling.MethodsWe created three simulation scenarios with increasing complexity to assess the skills of residents in their first, second, or third postgraduate year in using SDM in TOLAC counseling. All cases involved a standardized patient requesting a TOLAC consultation. Residents were provided with a medical history and instructed to counsel and develop a care plan. A 10-item scoring rubric was used, and each item was rated 0 (absent), 1 (partial), or 2 (complete). Three coders independently rated the encounters; discrepancies were resolved by consensus.ResultsOver 3 years, 39 residents participated in 60 OSCE encounters. The majority provided complete discussions of the clinical issue (93%), chances of success (72%), and maternal and fetal risks (100% and 85%, respectively) but obtained partial assessments of understanding (78%). Discussions of benefits were typically absent, with the exception of the maternal benefits (47%). More than 40% of residents did not discuss the patient's goals, 53% lacked discussion of uncertainties related to TOLAC, and half failed to explore the patient's preference, with most deferring a decision to a future encounter.DiscussionResidents consistently discussed diagnosis, prognosis, and maternal risks yet infrequently addressed goals and preferences-two critical elements of SDM.

| S-EPMC7182044 | biostudies-literature

Objective structured clinical examinations (OSCE) performance among Quebec urology residents: A retrospective study from 2008-2019.

Project description:IntroductionWe aimed to compare objective structured clinical examinations (OSCE) performance of residents from four Canadian urology programs, based on resident and station characteristics. We also aimed to evaluate OSCE contents by station type and subspecialty.MethodsScores of 109 postgraduate year (PGY)-3 to PGY-5 residents were retrospectively reviewed from 19 OSCEs from May 2008 to February 2019. Scores were grouped by station type/subspecialty, PGY level, medical graduate type (Canadian medical graduate [CMG], international medical graduate [IMG]), sex, and choice of fellowship/practice. Linear mixed modelling was performed to obtain least square means to account for repeated measures.ResultsScore increases from PGY-3 to PGY-5 were significant for all station types and subspecialties (p≤0.001). Scores were similar between male and female residents, and between CMGs and IMGs, except in visual recognition examinations (VREs) (males: 44.3±1.0, females: 39.0±1.6, p=0.005; IMG: 47.3±1.7, CMG: 41.6±0.9, p=0.004). Relative to uro-oncology stations, scores were lower in andrology (p=0.010) and functional urology (p<0.001). More female residents chose pediatric (14.3% vs. 1.5%, p=0.024) and functional urology fellowships (17.9% vs. 2.9%, p=0.021). More male residents chose endourology/robotic fellowships (30.9% vs. 10.7%, p=0.042). No associations between subspecialty scores and choice of fellowship/practice were found. Oral stations and VREs were more frequent than telephone stations. Uro-oncology and pediatric urology were more frequent than other subspecialties.ConclusionsScores improved with higher PGY level. IMGs and male residents scored better in VREs. Scores were lower in functional urology. There was no correlation between subspecialty score and choice of fellowship/practice. Subspecialties and forms of evaluation were not equally represented.

| S-EPMC7492032 | biostudies-literature

OmicsDI is part of the ELIXIR infrastructure

OmicsDI is an Elixir interoperability service. Learn more ›

Tweets

OmicsDI Databases

PRIDE
PeptideAtlas
MassIVE
JPOST Repository
Physiome Model Repository

EGA
EVA
ENA
LINCS
PAXDB
Cell Collective

MetaboLights
Metabolomics Workbench
MetabolomeExpress
GNPS
BioModels
FAIRDOMHub

ArrayExpress
dbGaP
ExpressionAtlas
GEO
NODE

Information

Databases
Help
API
Contact us
Code on GitHub
Terms of use
Submit Data