Dataset Information

Scope and Nature of Pain- and Analgesia-Related Content of the United States Medical Licensing Examination (USMLE).

ABSTRACT: Background:"The ongoing opioid crisis lies at the intersection of two substantial public health challenges-reducing the burden of suffering from pain and containing the rising toll of the harms that can result from the use of opioid medications" [1]. Improved pain education for health care providers is an essential component of the multidimensional response to both still-unmet challenges [2,3]. Despite the importance of licensing examinations in assuring competency in health care providers, there has been no prior appraisal of pain and related content within the United States Medical Licensing Examination (USMLE). Methods:An expert panel developed a novel methodology for characterizing USMLE questions based on pain core competencies and topical and public health relevance. Results:Under secure conditions, raters used this methodology to score 1,506 questions, with 28.7% (432) identified as including the word "pain." Of these, 232 questions (15.4% of the 1,506 USMLE questions reviewed) were assessed as being fully or partially related to pain, rather than just mentioning pain but not testing knowledge of its mechanisms and their implications for treatment. The large majority of questions related to pain (88%) focused on assessment rather than safe and effective pain management, or the context of pain. Conclusions:This emphasis on assessment misses other important aspects of safe and effective pain management, including those specific to opioid safety. Our findings inform ways to improve the long-term education of our medical and other graduates, thereby improving the health care of the populations they serve.

SUBMITTER: Fishman SM

PROVIDER: S-EPMC6057520 | biostudies-literature | 2018 Mar

REPOSITORIES: biostudies-literature

ACCESS DATA

Publications

Scope and Nature of Pain- and Analgesia-Related Content of the United States Medical Licensing Examination (USMLE).

Fishman Scott M SM Carr Daniel B DB Hogans Beth B Cheatle Martin M Gallagher Rollin M RM Katzman Joanna J Mackey Sean S Polomano Rosemary R Popescu Adrian A Rathmell James P JP Rosenquist Richard W RW Tauben David D Beckett Laurel L Li Yueju Y Mongoven Jennifer M JM Young Heather M HM

Pain medicine (Malden, Mass.) 20180301 3

<h4>Background</h4>"The ongoing opioid crisis lies at the intersection of two substantial public health challenges-reducing the burden of suffering from pain and containing the rising toll of the harms that can result from the use of opioid medications" [1]. Improved pain education for health care providers is an essential component of the multidimensional response to both still-unmet challenges [2,3]. Despite the importance of licensing examinations in assuring competency in health care provide ...[more]

PMID: 29365160

Similar Datasets

Project description:ObjectiveThe research sought to ascertain medical and dental libraries' collection development policies, evaluation methods, purchase decisions, and issues that relate to print and electronic United States Medical Licensing Examination (USMLE) and National Board Dental Examination (NBDE) preparation materials.MethodsThe investigators surveyed librarians supporting American Association of Medical Colleges (AAMC)-accredited medical schools (n = 58/125) on the USMLE and librarians supporting American Dental Association (ADA)-accredited dental schools (n = 23/56) on the NBDE. The investigators analyzed the data by cross-tabulating and filtering the results using EFM Continuum web survey software. Investigators also surveyed print and electronic USMLE and NBDE preparation materials from 2004-2007 to determine the number of publications and existence of reviews.ResultsA majority of responding AAMC libraries (62%, n = 58) provide at least 1 electronic or online USMLE preparation resource and buy an average of 11.6 print USMLE titles annually. Due to a paucity of NBDE print and electronic resources, ADA libraries bought significantly fewer print resources, and only 1 subscribed to an electronic resource. The most often reported evaluation methods for both populations were feedback from medical or dental students, feedback from medical or dental faculty, and online trials. Some AAMC (10%, n = 58) and ADA libraries (39%, n = 23) libraries reported that no evaluation of these materials occured at their libraries.ConclusionsFrom 2004-2007, publishers produced 45 USMLE preparation resources (total n = 546) to every 1 NBDE preparation resource (total n = 12). Users' needs, institutional missions and goals, financial status, and official collection policies most often underlie decisions to collect or not collect examination preparation materials. Evaluating the quality of examination preparation materials can be problematic due to lack of published reviews, lack of usability testing by libraries, and librarians' and library users' unfamiliarity with the actual content of examinations. Libraries must integrate faculty and students into the purchase process to make sure examination preparation resources of the highest quality are purchased.

Project description:BackgroundChat Generative Pre-trained Transformer (ChatGPT) is a 175-billion-parameter natural language processing model that can generate conversation-style responses to user input.ObjectiveThis study aimed to evaluate the performance of ChatGPT on questions within the scope of the United States Medical Licensing Examination Step 1 and Step 2 exams, as well as to analyze responses for user interpretability.MethodsWe used 2 sets of multiple-choice questions to evaluate ChatGPT's performance, each with questions pertaining to Step 1 and Step 2. The first set was derived from AMBOSS, a commonly used question bank for medical students, which also provides statistics on question difficulty and the performance on an exam relative to the user base. The second set was the National Board of Medical Examiners (NBME) free 120 questions. ChatGPT's performance was compared to 2 other large language models, GPT-3 and InstructGPT. The text output of each ChatGPT response was evaluated across 3 qualitative metrics: logical justification of the answer selected, presence of information internal to the question, and presence of information external to the question.ResultsOf the 4 data sets, AMBOSS-Step1, AMBOSS-Step2, NBME-Free-Step1, and NBME-Free-Step2, ChatGPT achieved accuracies of 44% (44/100), 42% (42/100), 64.4% (56/87), and 57.8% (59/102), respectively. ChatGPT outperformed InstructGPT by 8.15% on average across all data sets, and GPT-3 performed similarly to random chance. The model demonstrated a significant decrease in performance as question difficulty increased (P=.01) within the AMBOSS-Step1 data set. We found that logical justification for ChatGPT's answer selection was present in 100% of outputs of the NBME data sets. Internal information to the question was present in 96.8% (183/189) of all questions. The presence of information external to the question was 44.5% and 27% lower for incorrect answers relative to correct answers on the NBME-Free-Step1 (P<.001) and NBME-Free-Step2 (P=.001) data sets, respectively.ConclusionsChatGPT marks a significant improvement in natural language processing models on the tasks of medical question answering. By performing at a greater than 60% threshold on the NBME-Free-Step-1 data set, we show that the model achieves the equivalent of a passing score for a third-year medical student. Additionally, we highlight ChatGPT's capacity to provide logic and informational context across the majority of answers. These facts taken together make a compelling case for the potential applications of ChatGPT as an interactive medical education tool to support learning.

Project description:BackgroundChatGPT (OpenAI) has gained considerable attention because of its natural and intuitive responses. ChatGPT sometimes writes plausible-sounding but incorrect or nonsensical answers, as stated by OpenAI as a limitation. However, considering that ChatGPT is an interactive AI that has been trained to reduce the output of unethical sentences, the reliability of the training data is high and the usefulness of the output content is promising. Fortunately, in March 2023, a new version of ChatGPT, GPT-4, was released, which, according to internal evaluations, was expected to increase the likelihood of producing factual responses by 40% compared with its predecessor, GPT-3.5. The usefulness of this version of ChatGPT in English is widely appreciated. It is also increasingly being evaluated as a system for obtaining medical information in languages other than English. Although it does not reach a passing score on the national medical examination in Chinese, its accuracy is expected to gradually improve. Evaluation of ChatGPT with Japanese input is limited, although there have been reports on the accuracy of ChatGPT's answers to clinical questions regarding the Japanese Society of Hypertension guidelines and on the performance of the National Nursing Examination.ObjectiveThe objective of this study is to evaluate whether ChatGPT can provide accurate diagnoses and medical knowledge for Japanese input.MethodsQuestions from the National Medical Licensing Examination (NMLE) in Japan, administered by the Japanese Ministry of Health, Labour and Welfare in 2022, were used. All 400 questions were included. Exclusion criteria were figures and tables that ChatGPT could not recognize; only text questions were extracted. We instructed GPT-3.5 and GPT-4 to input the Japanese questions as they were and to output the correct answers for each question. The output of ChatGPT was verified by 2 general practice physicians. In case of discrepancies, they were checked by another physician to make a final decision. The overall performance was evaluated by calculating the percentage of correct answers output by GPT-3.5 and GPT-4.ResultsOf the 400 questions, 292 were analyzed. Questions containing charts, which are not supported by ChatGPT, were excluded. The correct response rate for GPT-4 was 81.5% (237/292), which was significantly higher than the rate for GPT-3.5, 42.8% (125/292). Moreover, GPT-4 surpassed the passing standard (>72%) for the NMLE, indicating its potential as a diagnostic and therapeutic decision aid for physicians.ConclusionsGPT-4 reached the passing standard for the NMLE in Japan, entered in Japanese, although it is limited to written questions. As the accelerated progress in the past few months has shown, the performance of the AI will improve as the large language model continues to learn more, and it may well become a decision support system for medical professionals by providing more accurate information.

Dataset Information

Scope and Nature of Pain- and Analgesia-Related Content of the United States Medical Licensing Examination (USMLE).

Publications

Scope and Nature of Pain- and Analgesia-Related Content of the United States Medical Licensing Examination (USMLE).

Similar Datasets

OmicsDI is part of the ELIXIR infrastructure

Tweets