Development and Validation of a Test for Competence in Evidence-Based Medicine.
Ontology highlight
ABSTRACT: Background
Medical educators need valid, reliable, and efficient tools to assess evidence-based medicine (EBM) knowledge and skills. Available EBM assessment tools either do not assess skills or are laborious to grade.Objective
To validate a multiple-choice-based EBM test-the Resident EBM Skills Evaluation Tool (RESET).Design
Cross-sectional study.Participants
A total of 304 medicine residents from five training programs and 33 EBM experts comprised the validation cohort.Main measures
Internal reliability, item difficulty, and item discrimination were assessed. Construct validity was assessed by comparing mean total scores of trainees to experts. Experts were also asked to rate importance of each test item to assess content validity.Key results
Experts had higher total scores than trainees (35.6 vs. 29.4, P < 0.001) and also scored significantly higher than residents on 11/18 items. Cronbach's alpha was 0.6 (acceptable), and no items had a low item-total correlation. Item difficulty ranged from 7 to 86%. All items were deemed "important" by > 50% of experts.Conclusions
The proposed EBM assessment tool is a reliable and valid instrument to assess competence in EBM. It is easy to administer and grade and could be used to guide and assess interventions in EBM education.
SUBMITTER: Patell R
PROVIDER: S-EPMC7210361 | biostudies-literature |
REPOSITORIES: biostudies-literature
Similar Datasets
Project description:ObjectiveTo describe the development and validation of a test of knowledge and skills in evidence based medicine.DesignCross sectional study.SettingFamily practice residency programme in California; a list server for those who teach evidence based medicine; and an evidence based medicine seminar series.ParticipantsFamily practice residents and faculty members (n=43); volunteers self identified as experts in evidence based medicine (n=53); family practice teachers (19) beginning a seminar series on evidence based medicine.InterventionThe Fresno test is a performance based measure for use in medical education that assesses a wide range of evidence based medicine skills. Open ended questions are scored with standardised grading rubrics. Calculation skills are assessed by fill in the blank questions.Main outcome measuresInter-rater reliability, internal reliability, item analyses, and construct validity.ResultsInter-rater correlations ranged from 0.76 to 0.98 for individual items. Cronbach's alpha was 0.88. Item difficulties ranged from moderate to difficult, all with positive and strong ability to discriminate between candidates. Experts scored consistently higher than novices. On the 212 point test, the novice mean was 95.6 and the expert mean was 147.5 (P<0.001). On individual items, a higher proportion of experts than novices earned passing scores on 15 of the 17 items.ConclusionThe Fresno test is a reliable and valid test for detecting the effect of instruction in evidence based medicine. Its use in other settings requires further exploration.
| S-EPMC143529 | biostudies-literature
Project description:Teachers' social-emotional competence is considered important in order to master the social and emotional challenges inherent in their profession and to build positive teacher-student relationships. In turn, this is key to both teachers' occupational well-being and positive student development. Nonetheless, an instrument assessing the profession-specific knowledge and skills that teachers need to master the social and emotional demands in the classroom is still lacking. Therefore, we developed the Test of Regulation in and Understanding of Social Situations in Teaching (TRUST), which is a theory-based situational judgment test measuring teachers' knowledge about strategies for emotion regulation and relationship management in emotionally and socially challenging situations with students. Results from three studies (N = 166 in-service teachers, N = 73 in-service teachers, N = 107 pre-service teachers) showed satisfactory internal consistency for both the emotion regulation and relationship management subtests. Furthermore, confirmatory factor analyses supported the differentiation between the two facets of social-emotional competence. Regarding convergent validity, results from Study 3 revealed a positive association between the profession-specific TRUST and pre-service teachers' general emotional intelligence. Furthermore, small to moderate correlations with the Big Five personality traits provided evidence for the discriminant validity of TRUST. In Studies 1 and 2, we found evidence for a correlation with external criteria, that is, teachers with higher test scores reported providing more emotional support for students and having better teacher-student relationships. For teachers' occupational well-being, we found a link with symptoms of depersonalization and job satisfaction, but none for emotional exhaustion. We will discuss the use of TRUST in research, for the evaluation of interventions, in teacher education, and professional development and will illustrate ideas for enhancing the tool.
| S-EPMC7273885 | biostudies-literature
Project description:BackgroundWhile a variety of instruments have been developed to assess knowledge and skills in evidence based medicine (EBM), few assess all aspects of EBM - including knowledge, skills attitudes and behaviour - or have been psychometrically evaluated. The aim of this study was to develop and validate an instrument that evaluates medical trainees' competency in EBM across knowledge, skills and attitude.MethodsThe 'Assessing Competency in EBM' (ACE) tool was developed by the authors, with content and face validity assessed by expert opinion. A cross-sectional sample of 342 medical trainees representing 'novice', 'intermediate' and 'advanced' EBM trainees were recruited to complete the ACE tool. Construct validity, item difficulty, internal reliability and item discrimination were analysed.ResultsWe recruited 98 EBM-novice, 108 EBM-intermediate and 136 EBM-advanced participants. A statistically significant difference in the total ACE score was observed and corresponded to the level of training: on a 0-15-point test, the mean ACE scores were 8.6 for EBM-novice; 9.5 for EBM-intermediate; and 10.4 for EBM-advanced (p?<?0.0001). Individual item discrimination was excellent (Item Discrimination Index ranging from 0.37 to 0.84), with internal reliability consistent across all but three items (Item Total Correlations were all positive ranging from 0.14 to 0.20).ConclusionThe 15-item ACE tool is a reliable and valid instrument to assess medical trainees' competency in EBM. The ACE tool provides a novel assessment that measures user performance across the four main steps of EBM. To provide a complete suite of instruments to assess EBM competency across various patient scenarios, future refinement of the ACE instrument should include further scenarios across harm, diagnosis and prognosis.
| S-EPMC4062508 | biostudies-literature
Project description:BackgroundEvidence-Based Medicine (EBM) integrates best available evidence from literature and patients' values, which then informs clinical decision making. However, there is a lack of validated instruments to assess the knowledge, practice and barriers of primary care physicians in the implementation of EBM. This study aimed to develop and validate an Evidence-Based Medicine Questionnaire (EBMQ) in Malaysia.MethodsThe EBMQ was developed based on a qualitative study, literature review and an expert panel. Face and content validity was verified by the expert panel and piloted among 10 participants. Primary care physicians with or without EBM training who could understand English were recruited from December 2015 to January 2016. The EBMQ was administered at baseline and two weeks later. A higher score indicates better knowledge, better practice of EBM and less barriers towards the implementation of EBM. We hypothesized that the EBMQ would have three domains: knowledge, practice and barriers.ResultsThe final version of the EBMQ consists of 80 items: 62 items were measured on a nominal scale, 22 items were measured on a 5 point Likert-scale. Flesch reading ease was 61.2. A total of 343 participants were approached; of whom 320 agreed to participate (response rate?=?93.2%). Factor analysis revealed that the EBMQ had eight domains after 13 items were removed: "EBM websites", "evidence-based journals", "types of studies", "terms related to EBM", "practice", "access", "patient preferences" and "support". Cronbach alpha for the overall EBMQ was 0.909, whilst the Cronbach alpha for the individual domain ranged from 0.657-0.940. The EBMQ was able to discriminate between doctors with and without EBM training for 24 out of 42 items. At test-retest, kappa values ranged from 0.155 to 0.620.ConclusionsThe EBMQ was found to be a valid and reliable instrument to assess the knowledge, practice and barriers towards the implementation of EBM among primary care physicians in Malaysia.
| S-EPMC6015654 | biostudies-literature
Project description:PurposeContrast sensitivity (CS) testing is an important measure of visual function reflecting variations in everyday visual experience in different conditions and helps to identify more subtle vision loss. However, it is only infrequently used. To make this more accessible, we have developed and validated a smartphone-based CS test.MethodsA new tumbling-E smartphone-based CS test was developed, Peek Contrast Sensitivity (PeekCS). This was field tested and refined through several iterations. Reference standard was a tumbling-E Pelli-Robson CS test (PRCS). The validation study was conducted in community clinics in Ethiopia. Test-retest variability was measured for both PRCS and PeekCS. PRCS and PeekCS were then compared. Correlation coefficients and 95% confidence intervals (CIs) were calculated; 95% limits of agreement were calculated and displayed on Bland-Altman plots.ResultsPeekCS showed strong repeatability (correlation coefficient: 0.93; 95% CI: 0.91-0.95), which was comparable with PRCS (correlation coefficient: 0.96; 95% CI: 0.95-0.97). The 95% limit of agreement for test-retest variability of PRCS and PeekCS were -0.20 to 0.21 and -0.31 to 0.29, respectively. PRCS and PeekCS were highly correlated: 0.94 (95% CI: 0.93-0.95); 95% limits of agreement -0.27 to 0.29; and mean difference 0.010 (95% CI: -0.001 to 0.022). PeekCS had a faster testing time (44.6 seconds) than PRCS (48.6 seconds): mean difference -3.98 (95% CI: -5.38 to -2.58); P < 0.001.ConclusionsThe smartphone-based PeekCS is a repeatable and rapid test, providing results that are highly comparable with the commonly used PRCS test.Translational relevancePeekCS provides an accessible and easy to perform alternative for CS testing, particularly in the community setting.
| S-EPMC6743644 | biostudies-literature
Project description: Not available
| S-EPMC102329 | biostudies-literature
Project description:Data are available such that choice of Helicobacter pylori therapy for an individual patient can be reliably predicted. Here, treatment success is defined as a cure rate of 90% or greater. Treatment outcome in a population or a patient can be calculated based on the effectiveness of a regimen for infections with susceptible and with resistant strains coupled with the knowledge of the prevalence of resistance (ie, based on formal measurement, clinical experience, or both). We provide the formula for predicting outcome and we illustrate the calculations. Because clarithromycin-containing triple therapy and 10-day sequential therapy are now only effective in special populations, they are considered obsolete; neither should continue to be used as empiric therapies (ie, 7- and 14-day triple therapies fail when clarithromycin resistance exceeds 5% and 15%, respectively, and 10-day sequential therapy fails when metronidazole resistance exceeds 20%). Therapy should be individualized based on prior history and whether the patient is in a high-risk group for resistance. The preferred choices for Western countries are 14-day concomitant therapy, 14-day bismuth quadruple therapy, and 14-day hybrid sequential-concomitant therapy. We also provide details regarding the successful use of fluoroquinolone-, rifabutin-, and furazolidone-containing therapies. Finally, we provide recommendations for the efficient development (ie, identification and optimization) of new regimens, as well as how to prevent or minimize failures. The trial-and-error approach for identifying and testing regimens frequently resulted in poor treatment success. The described approach allows outcome to be predicted and should simplify treatment and drug development.
| S-EPMC3830667 | biostudies-literature
Project description:There are professional expectations for public health nurses to develop competencies in evidence-informed decision-making (EIDM) due to its potential for improved client outcomes. Robust tools to assess EIDM competence can encourage increased EIDM engagement and uptake. This study aimed to develop and validate the content of a measure to assess EIDM competence among public health nurses. A four-stage process, based on measure development principles and the Standards for Educational and Psychological Testing, was used to develop and refine items for a new EIDM competence measure: a) content coverage assessment of existing measures; b) identification of existing measures for use and development of items; c) validity assessment based on content; d) validity assessment based on response process. An EIDM competence measurement tool consisting of EIDM knowledge, skills, attitudes/beliefs, and behaviour items was developed using conceptual literature and existing measures (Evidence-Based Practice Competency Tool and Evidence-Based Practice Beliefs Scale) to address limitations of existing EIDM tools identified from the content coverage assessment. Item content validity index ratings ranged from 0.64-1.00. Qualitative themes from validity assessment based on content and response process included word changes to improve clarity, reducing item redundancy, separating multi-component items, and ensuring items reflect nursing role expectations. Upon determining its reliability and validity, there is potential for the EIDM competence measure to be used in: public health nursing practice to identify competence gaps and strengths to facilitate professional development activities; in research to support development of strategies to build EIDM capacity; and for curriculum planning and development across nursing education programs.
| S-EPMC7946311 | biostudies-literature
Project description:The oocytes presenting SERa are considered dysmorphic and have been associated with reduced embryological outcome and an increased risk of congenital anomalies. Therefore, in 2011 the Alpha/ESHRE Consensus strongly recommended not to inseminate MII oocytes displaying SERa (SERa+). Not all Centers follow this recommendation and some Authors reported that SERa+ can lead to healthy newborns. Moreover, no experimental data about the effect of SERa presence on the oocyte molecular status are available. To date the question whether or not SERa+ oocytes should be discarded as well as the opportunity to review or not the ESHRE consensus are still open for debate. This study included 28 women, < 38 years aged, without any ovarian pathology, undergoing IVF treatment. Supernumerary MII oocytes with no sign of morphological alterations and SERa+ oocytes were donated after written informed consent. A total of 31 oocytes without SERa (SERa-) and 24 SERa+ oocytes were analyzed. Pools of ≈10 oocytes for both group were prepared. Total RNA was extracted from each pool, amplified, labelled and hybridized on oligonucleotide microarrays. Analyses were performed by R software using the limma package. The expression profiles of SERa+ oocytes significantly differed from those of SERa- oocytes in 488 probe sets corresponding to 102 down-regulated and 283 up-regulated unique transcripts. Gene Ontology analysis by DAVID bioinformatics resource disclosed that three main biological processes were significantly down-regulated in SERa+ oocytes respect to SERa- oocytes: i) genes involved in cell and mitotic nuclear division, spindle assembly, chromosome partition and G2/M transition of mitotic cell cycle; ii) organization of cytoskeleton and microtubules; iii) mitochondrial structure and activity. Among the transcripts up-regulated in SERa+ oocytes, the most significantly (P= 0.002) enriched GO term was “GoLoco motif” including RAP1GAP, GPSM3, and GPSM1 genes. Data validation in a larger cohort of samples would be beneficial, although we applied stringent criteria for gene selection (fold-change >3 or <1/3 and FDR<0.1). Surveys on clinical outcomes, malformation rates, and follow-up of babies born after transfer of embryos from SERa+ oocytes are necessary. For the first time we provide information on molecular status of SERa+ oocytes, highlighting possible correlations among presence of SERa, altered oocyte physiology and reduced developmental competence. Based on our findings, we believe that it is far too early for an eventual revision of the 2011 Alpha/ESHRE Consensus on the issue and presence of SERa should be still a “red flag” in clinical IVF practices.
2018-05-10 | GSE106222 | GEO
Project description:ObjectiveTo develop and validate an instrument for measuring knowledge and skills in evidence based medicine and to investigate whether short courses in evidence based medicine lead to a meaningful increase in knowledge and skills.DesignDevelopment and validation of an assessment instrument and before and after study.SettingVarious postgraduate short courses in evidence based medicine in Germany.ParticipantsThe instrument was validated with experts in evidence based medicine, postgraduate doctors, and medical students. The effect of courses was assessed by postgraduate doctors from medical and surgical backgrounds.InterventionIntensive 3 day courses in evidence based medicine delivered through tutor facilitated small groups.Main outcome measureIncrease in knowledge and skills.ResultsThe questionnaire distinguished reliably between groups with different expertise in evidence based medicine. Experts attained a threefold higher average score than students. Postgraduates who had not attended a course performed better than students but significantly worse than experts. Knowledge and skills in evidence based medicine increased after the course by 57% (mean score before course 6.3 (SD 2.9) v 9.9 (SD 2.8), P<0.001). No difference was found among experts or students in absence of an intervention.ConclusionsThe instrument reliably assessed knowledge and skills in evidence based medicine. An intensive 3 day course in evidence based medicine led to a significant increase in knowledge and skills.
| S-EPMC137813 | biostudies-literature