Dataset Information

Evaluating the reliability of gestalt quality ratings of medical education podcasts: A METRIQ study.

ABSTRACT: INTRODUCTION:Podcasts are increasingly being used for medical education. Studies have found that the assessment of the quality of online resources can be challenging. We sought to determine the reliability of gestalt quality assessment of education podcasts in emergency medicine. METHODS:An international, interprofessional sample of raters was recruited through social media, direct contact, and the extended personal network of the study team. Each participant listened to eight podcasts (selected to include a variety of accents, number of speakers, and topics) and rated the quality of that podcast on a seven-point Likert scale. Phi coefficients were calculated within each group and overall. Decision studies were conducted using a phi of 0.8. RESULTS:A total of 240 collaborators completed all eight surveys and were included in the analysis. Attendings, medical students, and physician assistants had the lowest individual-level variance and thus the lowest number of required raters to reliably evaluate quality (phi >0.80). Overall, 20 raters were required to reliably evaluate the quality of emergency medicine podcasts. DISCUSSION:Gestalt ratings of quality from approximately 20 health professionals are required to reliably assess the quality of a podcast. This finding should inform future work focused on developing and validating tools to support the evaluation of quality in these resources.

SUBMITTER: Woods JM

PROVIDER: S-EPMC7550476 | biostudies-literature | 2020 Oct

REPOSITORIES: biostudies-literature

ACCESS DATA

Publications

Evaluating the reliability of gestalt quality ratings of medical education podcasts: A METRIQ study.

Woods Jason M JM Chan Teresa M TM Roland Damian D Riddell Jeff J Tagg Andrew A Thoma Brent B

Perspectives on medical education 20201001 5

<h4>Introduction</h4>Podcasts are increasingly being used for medical education. Studies have found that the assessment of the quality of online resources can be challenging. We sought to determine the reliability of gestalt quality assessment of education podcasts in emergency medicine.<h4>Methods</h4>An international, interprofessional sample of raters was recruited through social media, direct contact, and the extended personal network of the study team. Each participant listened to eight pod ...[more]

PMID: 32495235

Similar Datasets

Project description:BackgroundIn recent years, the information environment for patients to learn about physician quality is being rapidly changed by Web-based ratings from both commercial and government efforts. However, little is known about how various types of Web-based ratings affect individuals' choice of physicians.ObjectiveThe objective of this research was to measure the relative importance of Web-based quality ratings from governmental and commercial agencies on individuals' choice of primary care physicians.MethodsIn a choice-based conjoint experiment conducted on a sample of 1000 Amazon Mechanical Turk users in October 2016, individuals were asked to choose their preferred primary care physician from pairs of physicians with different ratings in clinical and nonclinical aspects of care provided by governmental and commercial agencies.ResultsThe relative log odds of choosing a physician increases by 1.31 (95% CI 1.26-1.37; P<.001) and 1.32 (95% CI 1.27-1.39; P<.001) units when the government clinical ratings and commercial nonclinical ratings move from 2 to 4 stars, respectively. The relative log odds of choosing a physician increases by 1.12 (95% CI 1.07-1.18; P<.001) units when the commercial clinical ratings move from 2 to 4 stars. The relative log odds of selecting a physician with 4 stars in nonclinical ratings provided by the government is 1.03 (95% CI 0.98-1.09; P<.001) units higher than a physician with 2 stars in this rating. The log odds of selecting a physician with 4 stars in nonclinical government ratings relative to a physician with 2 stars is 0.23 (95% CI 0.13-0.33; P<.001) units higher for females compared with males. Similar star increase in nonclinical commercial ratings increases the relative log odds of selecting the physician by female respondents by 0.15 (95% CI 0.04-0.26; P=.006) units.ConclusionsIndividuals perceive nonclinical ratings provided by commercial websites as important as clinical ratings provided by government websites when choosing a primary care physician. There are significant gender differences in how the ratings are used. More research is needed on whether patients are making the best use of different types of ratings, as well as the optimal allocation of resources in improving physician ratings from the government's perspective.

Project description:BackgroundQuality improvement (QI) is an essential component of modern clinical practice. Front-line professionals offer valuable perspectives on areas for improvement and are motivated to deliver change. In the UK, all junior doctors are expected to participate in QI in order to advance to the next stage of their training. However, UK undergraduates receive no standardized training in QI methods. This is perpetuated within medical schools by a lack of teaching capacity and competing priorities, and may lead to tokenistic engagement with future QI projects.MethodsWe describe a near-peer teaching programme designed to introduce students to QI methods. This pilot study was conceived and delivered in full by junior doctors and used existing resources to ensure high quality teaching content. 111 fifth-year medical students from the University of Cambridge were taught in interactive, participative workshops that encourage them to develop their own QI change ideas and projects. Core topics included the model for improvement, driver diagrams, stakeholder engagement, measurement for improvement and analysing and presenting data. Students completed surveys before and immediately after this intervention to assess their understanding of and confidence in utilizing QI methods. Questionnaires were also completed by junior doctor tutors.ResultsAnalysis of questionnaires completed before and immediately after the intervention revealed statistically significant improvements in students' self-reported understanding of QI (p < 0.05) and confidence in applying techniques to their own work (p < 0.05). Students expressed a preference for QI teaching delivered by junior doctors, citing a relaxed learning environment and greater relevance to their stage of training. Tutors reported increased confidence in using QI techniques and a greater willingness to engage with QI in future.ConclusionsIn this single-centre study, near-peer teaching produced significant improvements in students' self-reported understanding of QI and confidence in applying QI methods. Near-peer teaching may constitute a sustainable means of teaching essential QI skills at undergraduate level. Future work must evaluate objective measures of student engagement with and competence in conducting QI.

Project description:Objectives:The aim of the study was to explore what components of the General Medical Council's (GMC) Quality Assurance Framework work, for whom, in what circumstances and how?Setting:UK undergraduate and postgraduate medical education and training.Participants:We conducted interviews with a stratified sample of 36 individuals. This included those who had direct experiences, as well as those with external insights, representing local, national and international organisations within and outside medicine.Intervention:The GMC quality assure education to protect patient and public safety utilising complex intervention components including meeting standards, institutional visits and monitoring performance. However, the context in which these are implemented matters. We undertook an innovative realist evaluation to test an initial programme theory. Data were analysed using framework analysis.Results:Across components of the intervention, we identified key mechanisms, including transparent reporting to promote quality improvement; dialogical feedback; partnership working facilitating interactions between regulators and providers, and role clarity in conducting proportionate interventions appropriate to risk. The GMC's framework was commended for being comprehensive and enabling a broad understanding of an organisation's performance. Unintended consequences included confusion over roles and boundaries in different contexts which often undermined effectiveness.Conclusions:This realist evaluation substantiates the literature and reveals deeper understandings about quality assuring medical education. While standardised approaches are implemented, interventions need to be contextually proportionate. Routine communication is beneficial to verify data, share concerns and check risk; however, ongoing partnership working can foster assurance. The study provides a modified programme theory to explicate how education providers and regulators can work more effectively together to uphold education quality, and ultimately protect public safety. The findings have influenced the GMC's approach to quality assurance which impacts on all medical students and doctors in training.

Dataset Information

Evaluating the reliability of gestalt quality ratings of medical education podcasts: A METRIQ study.

Publications

Evaluating the reliability of gestalt quality ratings of medical education podcasts: A METRIQ study.

Similar Datasets

OmicsDI is part of the ELIXIR infrastructure

Tweets