Virtual Raters for Reproducible and Objective Assessments in Radiology.
Ontology highlight
ABSTRACT:
SUBMITTER: Kleesiek J
PROVIDER: S-EPMC4846987 | biostudies-literature |
REPOSITORIES: biostudies-literature
Similar Datasets
Project description:BACKGROUND:Objective structured clinical examinations (OSCEs) are a useful method to evaluate medical students' performance in the clerkship years. OSCEs are designed to assess skills and knowledge in a standardized clinical setting and through use of a preset standard grading sheet, so that clinical knowledge can be evaluated at a high level and in a reproducible way. OBJECTIVE:This study aimed to present our OSCE assessment tool designed specifically for radiology clerkship medical students, which we called the objective structured radiology examination (OSRE), with the intent to advance the assessment of clerkship medical students by providing an objective, structured, reproducible, and low-cost method to evaluate medical students' radiology knowledge and the reproducibility of this assessment tool. METHODS:We designed 9 different OSRE cases for radiology clerkship classes with participating third- and fourth-year medical students. Each examination comprises 1 to 3 images, a clinical scenario, and structured questions, along with a standardized scoring sheet that allows for an objective and low-cost assessment. Each medical student completed 3 of 9 random examination cases during their rotation. To evaluate for reproducibility of our scoring sheet assessment tool, we used 5 examiners to grade the same students. Reproducibility for each case and consistency for each grader were assessed with a two-way mixed effects intraclass correlation coefficient (ICC). An ICC below 0.4 was deemed poor to fair, an ICC of 0.41 to 0.60 was moderate, an ICC of 0.6 to 0.8 was substantial, and an ICC greater than 0.8 was almost perfect. We also assessed the correlation of scores and the students' clinical experience with a linear regression model and compared mean grades between third- and fourth-year students. RESULTS:A total of 181 students (156 third- and 25 fourth-year students) were included in the study for a full academic year. Moreover, 6 of 9 cases demonstrated average ICCs more than 0.6 (substantial correlation), and the average ICCs ranged from 0.36 to 0.80 (P<.001 for all the cases). The average ICC for each grader was more than 0.60 (substantial correlation). The average grade among the third-year students was 11.9 (SD 4.9), compared with 12.8 (SD 5) among the fourth-year students (P=.005). There was no correlation between clinical experience and OSRE grade (-0.02; P=.48), adjusting for the medical school year. CONCLUSIONS:Our OSRE is a reproducible assessment tool with most of our OSRE cases showing substantial correlation, except for 3 cases. No expertise in radiology is needed to grade these examinations using our scoring sheet. There was no correlation between scores and the clinical experience of the medical students tested.
| S-EPMC7240440 | biostudies-literature
Project description:Improving the reliability and consistency of objective structured clinical examination (OSCE) raters' marking poses a continual challenge in medical education. The purpose of this study was to evaluate an e-Learning training module for OSCE raters who participated in the assessment of third-year medical students at the University of Ottawa, Canada. The effects of online training and those of traditional in-person (face-to-face) orientation were compared. Of the 90 physicians recruited as raters for this OSCE, 60 consented to participate (67.7%) in the study in March 2017. Of the 60 participants, 55 rated students during the OSCE, while the remaining 5 were back-up raters. The number of raters in the online training group was 41, while that in the traditional in-person training group was 19. Of those with prior OSCE experience (n= 18) who participated in the online group, 13 (68%) reported that they preferred this format to the in-person orientation. The total average time needed to complete the online module was 15 minutes. Furthermore, 89% of the participants felt the module provided clarity in the rater training process. There was no significant difference in the number of missing ratings based on the type of orientation that raters received. Our study indicates that online OSCE rater training is comparable to traditional face-to-face orientation.
| S-EPMC6194479 | biostudies-literature
Project description:Rationale and objectivesCOVID-19 has disrupted radiology education and forced a transition from traditional in-person learning to a virtual platform. As a result of hospital and state mandates, our radiology residency program quickly transitioned to a virtual learning platform to continue dissemination of knowledge, maintain resident engagement, and ensure professional development. The goal of this study is to assess the strengths and weaknesses of the virtual learning platform at our institution using resident ratings.Materials and methodsThis institutional IRB-exempt study involved a survey of 17 questions which was electronically distributed to 45 radiology residents using SurveyMonkey. Questions encompassed resident satisfaction with teaching and professional development, scheduling changes, and engagement with the virtual platform. Answers to most questions were submitted on a Likert scale.ResultsA total of 31 of 45 respondents completed the survey (response rate = 69%). Most residents were satisfied with the virtual platform with teaching activities identified as a strength and the incorporation of professional development as a weakness. The most frequent barriers to attending the virtual curriculum were technical difficulties (43%) and childcare (36%). Residents who reported experiencing barriers were less likely to adhere to the virtual curriculum (p = 0.004). Most respondents (81%) reported a desire to maintain elements of the virtual learning practice postpandemic.ConclusionThe majority of residents reported high satisfaction with virtual learning during the COVID-19 pandemic. Teaching activities are a curricular strength. Weaknesses identified include the incorporation of professional development and extrinsic barriers, such as technical difficulties and family obligations, which require further support for trainees.
| S-EPMC7883720 | biostudies-literature
Project description:BackgroundElevated cardiovascular disease risk is observed in patients with COPD. Non-invasive assessments of endothelial dysfunction and arterial stiffness have recently emerged to provide mechanistic insight into cardiovascular disease risk in COPD; however, the reproducibility of endothelial function and arterial stiffness has yet to be investigated in this patient population.ObjectivesThis study sought to examine the within-day and between-day reproducibility of endothelial function and arterial stiffness in patients with COPD.MethodsBaseline diameter, peak diameter, flow-mediated dilation, augmentation index, augmentation index at 75 beats per minute, and pulse wave velocity were assessed three times in 17 patients with COPD (six males, eleven females, age range 47-75 years old; forced expiratory volume in 1 second =51.5% predicted). Session A and B were separated by 3 hours (within-day), whereas session C was conducted at least 7 days following session B (between-day). Reproducibility was assessed by: 1) paired t-tests, 2) coefficients of variation, 3) coefficients of variation prime, 4) intra-class correlation coefficient, 5) Pearson's correlations (r), and 6) Bland-Altman plots. Five acceptable assessments were required to confirm reproducibility.ResultsSix out of six within-day criteria were met for endothelial function and arterial stiffness outcomes. Six out of six between-day criteria were met for baseline and peak diameter, augmentation index and pulse wave velocity, whereas five out of six criteria were met for flow-mediated dilation.ConclusionThe present study provides evidence for within-day and between-day reproducibility of endothelial function and arterial stiffness in patients with COPD.
| S-EPMC4577266 | biostudies-literature
Project description:Many computational approaches estimate the effect of coding variants, but their predictions often disagree with each other. These contradictions confound users and raise questions regarding reliability. Performance assessments can indicate the expected accuracy for each method and highlight advantages and limitations. The Critical Assessment of Genome Interpretation (CAGI) community aims to organize objective and systematic assessments: They challenge predictors on unpublished experimental and clinical data and assign independent assessors to evaluate the submissions. We participated in CAGI experiments as predictors, using the Evolutionary Action (EA) method to estimate the fitness effect of coding mutations. EA is untrained, uses homology information, and relies on a formal equation: The fitness effect equals the functional sensitivity to residue changes multiplied by the magnitude of the substitution. In previous CAGI experiments (between 2011 and 2016), our submissions aimed to predict the protein activity of single mutants. In 2018 (CAGI5), we also submitted predictions regarding clinical associations, folding stability, and matching genomic data with phenotype. For all these diverse challenges, we used EA to predict the fitness effect of variants, adjusted to specifically address each question. Our submissions had consistently good performance, suggesting that EA predicts reliably the effects of genetic variants.
| S-EPMC6900054 | biostudies-literature
Project description:STUDY OBJECTIVES:Patients with postural tachycardia syndrome (POTS) commonly complain of fatigue, unrefreshing sleep, daytime sleepiness, and diminished quality of life. The study objective was to assess objective sleep quality in POTS patients using overnight polysomnography. METHODS:We studied 16 patients with POTS and 15 healthy control subjects performing daytime autonomic functions tests and overnight polysomnography at the Vanderbilt Clinical Research Center. RESULTS:There were no significant differences in the objective sleep parameters including sleep efficiency, sleep onset latency, wake time after sleep onset, REM latency, percentage of time spent in N1, N2, N3, and REM sleep, arousal index, apnea-hypopnea index, or periodic leg movement index in POTS patients as compared with healthy control subjects. There were significant negative correlations between sleep efficiency and the change in HR from supine to stand (rs = -0.527; p = 0.036). CONCLUSIONS:POTS patients do not have significant differences in objective sleep parameters as compared to control subjects based on overnight polysomnograms. Activation of the sympathetic nervous system may contribute significantly to the hyper arousal state and worsening of subjective estimates of sleep quality as previously reported in POTS patients.
| S-EPMC4865560 | biostudies-literature
Project description:Prolonged periods of cognitive workload will cause mental fatigue, but objective, quantitative, and sensitive measurements that reflect long-term, stress-induced mental fatigue have yet to be elucidated. This study aims to apply a potential marker of Rényi entropy to investigate the mental fatigue changes in a long-term, high-level stress condition and compare three different instruments for assessment of mental fatigue: EEG, the oddball task, and self-scoring. We recruited nine individuals who participated in a 5-day intellectually challenging competition. The participants were assessed for mental fatigue each day of the competition using prefrontal cortex electroencephalogram (EEG). Reaction time in an oddball task and self-rated scoring were used comparatively to evaluate the performance of the EEG. Repeated measures ANOVA was utilized to analyze the differences among score, reaction time, and wavelet Rényi entropy. The results demonstrated that both wavelet Rényi entropy extracted from EEG and self-rated scoring revealed significant increases in mental fatigue during the 5 days of competition (P < 0.001). The reaction time of the oddball task did not show significant changes during the five-day competition (P = 0.066). Moreover, the wavelet Rényi entropy analysis of EEG showed greater sensitivity than the self-rated scoring and reaction time of the oddball task for measuring mental fatigue changes. In conclusion, this study shows that mental fatigue accumulates during long-term, high-level stress situations. The study also indicates that EEG wavelet Rényi entropy is an efficient metric to reflect the change of mental fatigue under a long-term stress condition and that EEG is a better method to assess long-term mental fatigue.
| S-EPMC8631328 | biostudies-literature
Project description:Spatial Independent Component Analysis (ICA) decomposes the time by space functional MRI (fMRI) matrix into a set of 1-D basis time courses and their associated 3-D spatial maps that are optimized for mutual independence. When applied to resting state fMRI (rsfMRI), ICA produces several spatial independent components (ICs) that seem to have biological relevance - the so-called resting state networks (RSNs). The ICA problem is well posed when the true data generating process follows a linear mixture of ICs model in terms of the identifiability of the mixing matrix. However, the contrast function used for promoting mutual independence in ICA is dependent on the finite amount of observed data and is potentially non-convex with multiple local minima. Hence, each run of ICA could produce potentially different IC estimates even for the same data. One technique to deal with this run-to-run variability of ICA was proposed by [1] in their algorithm RAICAR which allows for the selection of only those ICs that have a high run-to-run reproducibility. We propose an enhancement to the original RAICAR algorithm that enables us to assign reproducibility p-values to each IC and allows for an objective assessment of both within subject and across subjects reproducibility. We call the resulting algorithm RAICAR-N (N stands for null hypothesis test), and we have applied it to publicly available human rsfMRI data (http://www.nitrc.org). Our reproducibility analyses indicated that many of the published RSNs in rsfMRI literature are highly reproducible. However, we found several other RSNs that are highly reproducible but not frequently listed in the literature.
| S-EPMC3236249 | biostudies-literature
Project description:IntroductionIn virtual bioequivalence (VBE) assessments, pharmacokinetic models informed with in vitro data and verified with small clinical trials' data are used to simulate otherwise unfeasibly large trials. Simulated VBE trials are assessed in a frequentist framework as if they were real despite the unlimited number of virtual subjects they can use. This may adequately control consumer risk but imposes unnecessary risks on producers. We propose a fully Bayesian model-integrated VBE assessment framework that circumvents these limitations.MethodsWe illustrate our approach with a case study on a hypothetical paliperidone palmitate (PP) generic long-acting injectable suspension formulation using a validated population pharmacokinetic model published for the reference formulation. BE testing, study power, type I and type II error analyses or their Bayesian equivalents, and safe-space analyses are demonstrated.ResultsThe fully Bayesian workflow is more precise than the frequentist workflow. Decisions about bioequivalence and safe space analyses in the two workflows can differ markedly because the Bayesian analyses are more accurate.DiscussionA Bayesian framework can adequately control consumer risk and minimize producer risk . It rewards data gathering and model integration to make the best use of prior information. The frequentist approach is less precise but faster to compute, and it can still be used as a first step to narrow down the parameter space to explore in safe-space analyses.
| S-EPMC11319711 | biostudies-literature
Project description:Predicting species potential and future distribution has become a relevant tool in biodiversity monitoring and conservation. In this data article we present the suitability map of a virtual species generated based on two bioclimatic variables, and a dataset containing more than 700,000 random observations at the extent of Europe. The dataset includes spatial attributes such as: distance to roads, protected areas, country codes, and the habitat suitability of two spatially clustered species (grassland and forest species) and a wide-spread species.
| S-EPMC4789307 | biostudies-literature