Dataset Information

Utility of intermittent online quizzes as an early warning for residents at risk of failing the pediatric board certification examination.

ABSTRACT:

Background

Traditionally, quizzes have been applied as a tool for summative assessment, though literature suggests their use as a formative assessment can improve motivation and content retention. With this premise, we implemented a series of intermittent, online quizzes known as the Board Examination Simulation Exercise (BESE). We sought to demonstrate an association between BESE participation and scores and performance on the American Board of Pediatrics (ABP) Certifying Examination (CE).

Methods

Residents were assigned online quizzes on a single topic at 2 week intervals that consisted of 20 multiple choice questions written by the study authors. This analysis includes graduates of 3 Pediatric and Internal Medicine-Pediatrics residency programs.

Results

Data were available for 329 residents. The overall BESE score weakly correlated with ABP CE score (n = 287; r = 0.39, p < 0.0001). ABP CE pass rates increased from 2009 to 2016 at all programs combined (p = 0.0001). A composite BESE score ≤ 11 had sensitivity of 54% and specificity of 80% for predicting ABP CE failure on the first attempt. There was no difference in ABP CE failure rates or scores by number of completed quizzes.

Conclusion

Intermittent online quizzes implemented at three pediatric residency programs were associated with overall increasing ABP CE pass rates. BESE increased program emphasis on board preparation. Residents with lower BESE scores more often failed ABP CE. Though additional data are needed, BESE is a promising tool for pediatric resident learning and board preparation. It may also aid in earlier identification of residents at higher risk of failing the ABP CE and facilitate targeted interventions.

SUBMITTER: Wallihan R

PROVIDER: S-EPMC6278081 | biostudies-literature | 2018 Dec

REPOSITORIES: biostudies-literature

ACCESS DATA

Publications

Utility of intermittent online quizzes as an early warning for residents at risk of failing the pediatric board certification examination.

Wallihan Rebecca R Smith Keely G KG Hormann Mark D MD Donthi Rajesh R RR Boland Kimberly K Mahan John D JD

BMC medical education 20181204 1

<h4>Background</h4>Traditionally, quizzes have been applied as a tool for summative assessment, though literature suggests their use as a formative assessment can improve motivation and content retention. With this premise, we implemented a series of intermittent, online quizzes known as the Board Examination Simulation Exercise (BESE). We sought to demonstrate an association between BESE participation and scores and performance on the American Board of Pediatrics (ABP) Certifying Examination (C ...[more]

PMID: 30514279

Similar Datasets

Project description:BackgroundArtificial intelligence models can learn from medical literature and clinical cases and generate answers that rival human experts. However, challenges remain in the analysis of complex data containing images and diagrams.ObjectiveThis study aims to assess the answering capabilities and accuracy of ChatGPT-4 Vision (GPT-4V) for a set of 100 questions, including image-based questions, from the 2023 otolaryngology board certification examination.MethodsAnswers to 100 questions from the 2023 otolaryngology board certification examination, including image-based questions, were generated using GPT-4V. The accuracy rate was evaluated using different prompts, and the presence of images, clinical area of the questions, and variations in the answer content were examined.ResultsThe accuracy rate for text-only input was, on average, 24.7% but improved to 47.3% with the addition of English translation and prompts (P<.001). The average nonresponse rate for text-only input was 46.3%; this decreased to 2.7% with the addition of English translation and prompts (P<.001). The accuracy rate was lower for image-based questions than for text-only questions across all types of input, with a relatively high nonresponse rate. General questions and questions from the fields of head and neck allergies and nasal allergies had relatively high accuracy rates, which increased with the addition of translation and prompts. In terms of content, questions related to anatomy had the highest accuracy rate. For all content types, the addition of translation and prompts increased the accuracy rate. As for the performance based on image-based questions, the average of correct answer rate with text-only input was 30.4%, and that with text-plus-image input was 41.3% (P=.02).ConclusionsExamination of artificial intelligence's answering capabilities for the otolaryngology board certification examination improves our understanding of its potential and limitations in this field. Although the improvement was noted with the addition of translation and prompts, the accuracy rate for image-based questions was lower than that for text-based questions, suggesting room for improvement in GPT-4V at this stage. Furthermore, text-plus-image input answers a higher rate in image-based questions. Our findings imply the usefulness and potential of GPT-4V in medicine; however, future consideration of safe use methods is needed.

Project description:BackgroundNarrative analysis and reflection have been found to support professional identity formation (PIF) and resilience among medical students. In the emergency department, students have used reflective practice to process challenging clinical experiences, such as ethical dilemmas or moral distress. An online discussion board, however, has not been described as a curricular component of emergency medicine (EM) rotations. The objective of this educational innovation was to support medical students in an EM clinical rotation via an online discussion board for reflecting on and debriefing clinical experiences with faculty and peers.MethodsFifty-two medical students enrolled in the pass/fail EM elective between May 13, 2019, and October 30, 2020. Each cohort of six students took part in a cohort-specific discussion using the Canvas learning management system. Students were encouraged to post about any observations, reflections, or emotions after their shifts. Faculty course directors responded to each post using concepts of debriefing, coaching, and trauma-informed teaching.ResultsOver 18 months, 49 of 52 (94%) students participated in the discussion board. Of 346 total posts, half were by students, and the other half were faculty responses. Students posted 3.27 times each, on average. Students rarely raised questions about scientific knowledge content, fact-based aspects of patient care, or specific skills. Rather, they often posted about intensely affective reactions to experiences that left them with complex emotions. Upon review of posts by the course directors, the majority (54%) of students' posts contained a range of affective responses. Students appreciated faculty responses and supported each other in their written responses to peers.ConclusionsAn online discussion board can be used successfully for asynchronous reflective practice to debrief clinical experiences during an EM rotation, if designed incorporating faculty and peer support using trauma-informed teaching principles to bolster well-being and PIF.

Project description:BackgroundThe board certification system serves as a quality assurance system for physicians, and its design and operation are important health policy issues. In Japan, board certification was established and operated independently by academic societies and has not been directly linked to reimbursement systems. The phenomenon of younger physicians seeking specialist careers has raised concerns about acceleration of the tendency of fewer physicians working in rural areas and the maldistribution of physicians. Little is known about the associations between physicians' geographical migration patterns and board certification status changes or between the continuation of urban/rural practice and the maintenance of board certification. This study aimed to identify these associations and to discuss their policy implications.MethodsWe analyzed 2012 and 2014 data from the Survey of Physicians, Dentists, and Pharmacists, a national census survey. To analyze geographical migration patterns, transitions in practice location (rural, intermediate, and urban) were analyzed by board certification status change (new, lost, consistently certified, and consistently uncertified). Logistic regression analysis was conducted to assess whether the odds of migrating to more urban/rural municipalities were associated with board certification status changes, adjusting for covariates, and whether practicing in a rural area was associated with maintaining board certification.ResultsAmong 18,726 newly board-certified physicians, 94.9% (13,435/14,150) of those working in urban areas before certification remained in urban areas, whereas 64.6% (393/608) of those working in rural areas stayed in rural areas. Those who were newly certified had higher odds of moving to more urban areas, adjusting for covariates. Those who stayed in rural areas showed lower odds of maintaining board certification, adjusting for covariates.ConclusionsNewly board-certified physicians are more likely to migrate to other types of areas, particularly more urban areas, than other physicians. Allocating more training quotas to rural areas could be one option for leveling the distribution of specialists. It also appeared that those practicing in rural areas have difficulty maintaining their certification, so the need to establish a support system for already-certified physicians in rural areas should be emphasized.

Project description:ContextCertification by an American Board of Medical Specialties (ABMS) member board is emerging as a measure of physician quality.ObjectiveTo identify demographic and educational factors associated with ABMS member board certification of US medical school graduates.Design, setting, and participantsRetrospective study of a national cohort of 1997-2000 US medical school graduates, grouped by specialty choice at graduation and followed up through March 2, 2009. In separate multivariable logistic regression models for each specialty category, factors associated with ABMS member board certification were identified.Main outcome measureABMS member board certification.ResultsOf 42,440 graduates in the study sample, 37,054 (87.3%) were board certified. Graduates in all specialty categories with first-attempt passing scores in the highest tertile (vs first-attempt failing scores) on US Medical Licensing Examination Step 2 Clinical Knowledge were more likely to be board certified; adjusted odds ratios (AORs) varied by specialty category, with the lowest odds for emergency medicine (87.4% vs 73.6%; AOR, 1.82; 95% CI, 1.03-3.20) and highest odds for radiology (98.1% vs 74.9%; AOR, 13.19; 95% CI, 5.55-31.32). In each specialty category except family medicine, graduates self-identified as underrepresented racial/ethnic minorities (vs white) were less likely to be board certified, ranging from 83.5% vs 95.6% in the pediatrics category (AOR, 0.44; 95% CI, 0.33-0.58) to 71.5% vs 83.7% in the other nongeneralist specialties category (AOR, 0.79; 95% CI, 0.64-0.96). With each $50,000 unit increase in debt (vs no debt), graduates choosing obstetrics/gynecology were less likely to be board certified (AOR, 0.89; 95% CI, 0.83-0.96), and graduates choosing family medicine were more likely to be board certified (AOR, 1.13; 95% CI, 1.01-1.26).ConclusionDemographic and educational factors were associated with board certification among US medical school graduates in every specialty category examined; findings varied among specialty categories.

Project description:ImportanceSuccess on the internal medicine (IM) examination is a central requirement of the American Board of Internal Medicine's (ABIM's) Maintenance of Certification program (MOC). Therefore, it is important to understand the degree to which this examination reflects conditions seen in practice, one dimension of content validity, which focuses on the match between content in the discipline and the topics on the examination questions.ObjectiveTo assess whether the frequency of questions on IM-MOC examinations were concordant with the frequency of conditions seen in practice.Design, setting, and participantsThe 2010-2013 IM-MOC examinations were used to calculate the percentage of questions for 186 medical condition categories from the examination blueprint, which balances examination content by considering importance and frequency of conditions seen in practice. Nationally representative estimates of conditions seen in practice by general internists were estimated from the primary diagnosis for 13 832 office visits (2010-2013 National Ambulatory Medical Care Surveys) and 108 472 hospital stays (2010 National Hospital Discharge Survey).ExposuresPrevalence of conditions included on the IM-MOC examination questions.Main outcomes and measuresThe outcome measure was the concordance between the percentages of IM-MOC examination questions and the percentages of conditions seen in practice during either office visits or hospital stays for each of 186 condition categories (eg, diabetes mellitus, ischemic heart disease, liver disease). The concordance thresholds were 0.5 SD of the weighted mean percentages of the applicable 186 conditions seen in practice (0.74% for office visits; 0.51% for hospital stays). If the absolute differences between the percentages of examination questions and the percentages of office visit conditions or hospital stay conditions seen were less than the applicable concordance threshold, then the condition category was judged to be concordant.ResultsDuring the 2010-2013 IM-MOC examination periods, 3600 questions (180 questions per examination form) were administered and 3461 questions (96.1%) were mapped into the 186 study conditions (mean, 18.6 questions per condition). Comparison of the percentages of 186 categories of medical conditions seen in 13 832 office visits and 108 472 hospital stays with the percentages of 3461 questions on IM-MOC examinations revealed that 2389 examination questions (69.0%; 95% CI, 67.5%-70.6% involving 158 conditions) were categorized as concordant. For concordance between questions and office visits only, 2010 questions (58.08%; 95% CI, 56.43%-59.72% of all examination questions) involving 145 conditions were categorized as concordant. For concordance between questions and hospital stays only, 1456 questions (42.07%; 95% CI, 40.42%-43.71% of all examination questions) involving 122 conditions were categorized as concordant.Conclusions and relevanceAmong questions on IM-MOC examinations from 2010-2013, 69% were concordant with conditions seen in general internal medicine practices, although some areas of discordance were identified.

Project description:BackgroundIt is anticipated that many licensing examination centres for pathology will begin fully digitizing the certification examinations. The objective of our study was to test the feasibility of a fully digital examination and to assess the needs, concerns and expectations of pathology residents in moving from a glass slide-based examination to a fully digital examination.MethodsWe conducted a mixed methods study that compared, after randomization, the performance of senior residents (postgraduate years 4 and 5) in 7 accredited anatomical pathology training programs across Canada on a pathology examination using either glass slides or digital whole-slide scanned images of the slides. The pilot examination was followed by a post-test survey. In addition, pathology residents from all levels of training were invited to participate in an online survey.ResultsA total of 100 residents participated in the pilot examination; 49 were given glass slides instead of digital images. We found no significant difference in examination results between the 2 groups of residents (estimated marginal mean 8.23/12, 95% confidence interval [CI] 7.72-8.87, for glass slides; 7.84/12, 95% CI 7.28-8.41, for digital slides). In the post-test survey, most of the respondents expressed concerns with the digital examination, including slowly functioning software, blurring and poor detail of images, particularly nuclear features. All of the respondents of the general survey (n = 179) agreed that additional training was required if the examination were to become fully digital.InterpretationAlthough the performance of residents completing pathology examinations with glass slides was comparable to that of residents using digital images, our study showed that residents were not comfortable with the digital technology, especially given their current level of exposure to it. Additional training may be needed before implementing a fully digital examination, with consideration for a gradual transition.

Dataset Information

Utility of intermittent online quizzes as an early warning for residents at risk of failing the pediatric board certification examination.

Background

Methods

Results

Conclusion

Publications

Utility of intermittent online quizzes as an early warning for residents at risk of failing the pediatric board certification examination.

Similar Datasets

OmicsDI is part of the ELIXIR infrastructure

Tweets