Evaluation of MCQs from MOOCs for common item writing flaws.
Ontology highlight
ABSTRACT:
SUBMITTER: Costello E
PROVIDER: S-EPMC6276217 | biostudies-literature | 2018 Dec
REPOSITORIES: biostudies-literature
Publications
Evaluation of MCQs from MOOCs for common item writing flaws.
Costello Eamon E Holland Jane C JC Kirwan Colette C
BMC research notes 20181203 1
<h4>Objective</h4>There is a dearth of research into the quality of assessments based on Multiple Choice Question (MCQ) items in Massive Open Online Courses (MOOCs). This dataset was generated to determine whether MCQ item writing flaws existed in a selection of MOOC assessments, and to evaluate their prevalence if so. Hence, researchers reviewed MCQs from a sample of MOOCs, using an evaluation protocol derived from the medical health education literature, which has an extensive evidence-base wi ...[more]
Similar Datasets
Project description:BackgroundThis study investigated the impact of addressing item writing flaws, testing at low cognitive level and non-functioning distractors (< 5 % selection frequency) in multiple-choice assessment in preclinical medical education.MethodMultiple-choice questions with too high or too low difficulty (difficulty index < 0.4 or > 0.8) and insufficient discriminatory ability (point-biserial correlation < 0.2) on previous administration were identified. Items in Experimental Subgroup A underwent removal of item writing flaws along with enhancement of tested cognitive level (21 multiple-choice questions), while Experimental Subgroup B underwent replacement or removal of non-functioning distractors (11 multiple-choice questions). A control group of items (Group C) did not undergo any intervention (23 multiple-choice questions).ResultPost-intervention, the average number of functioning distractors (≥ 5 % selection frequency) per multiple-choice question increased from 0.67 to 0.81 in Subgroup A and from 0.91 to 1.09 in Subgroup B; a statistically significant increase in the number of multiple-choice questions with sufficient point-biserial correlation was also noted. No significant changes were noted in psychometric characteristics of the control group of items.ConclusionCorrection of item flaws, removal or replacement of non-functioning distractors, and enhancement of tested cognitive level positively impact the discriminatory ability of multiple-choice questions. This helps prevent construct-irrelevant variance from affecting the evidence of validity of scores obtained in multiple-choice questions.
| S-EPMC4602009 | biostudies-literature
Project description:BackgroundThe implementation of the National Genomic Medicine Service in the UK has increased patient access to germline genomic testing. Increased testing leads to more genetic diagnoses but does result in the identification of genomic variants of uncertain significance (VUS). The rigorous process of interpreting these variants requires multi-disciplinary, highly trained healthcare professionals (HCPs). To meet this training need, we designed two Massive Open Online Courses (MOOCs) for HCPs involved in germline genomic testing pathways: Fundamental Principles (FP) and Inherited Cancer Susceptibility (ICS).MethodsAn evaluation cohort of HCPs involved in genomic testing were recruited, with additional data also available from anonymous self-registered learners to both MOOCs. Pre- and post-course surveys and in-course quizzes were used to assess learner satisfaction, confidence and knowledge gained in variant interpretation. In addition, granular feedback was collected on the complexity of the MOOCs to iteratively improve the resources.ResultsA cohort of 92 genomics HCPs, including clinical scientists, and non-genomics clinicians (clinicians working in specialties outside of genomics) participated in the evaluation cohort. Between baseline and follow-up, total confidence scores improved by 38% (15.2/40.0) (95% confidence interval [CI] 12.4-18.0) for the FP MOOC and 54% (18.9/34.9) (95%CI 15.5-22.5) for the ICS MOOC (p < 0.0001 for both). Of those who completed the knowledge assessment through six summative variant classification quizzes (V1-6), a mean of 79% of respondents classified the variants such that correct clinical management would be undertaken (FP: V1 (73/90) 81% Likely Pathogenic/Pathogenic [LP/P]; V2 (55/78) 70% VUS; V3 (59/75) 79% LP/P; V4 (62/72) 86% LP/LP. ICS: V5 (66/91) 73% VUS; V6 (76/88) 86% LP/P). A non-statistically significant higher attrition rate was seen amongst the non-genomics workforce when compared to genomics specialists for both courses. More participants from the non-genomics workforce rated the material as "Too Complex" (FP n = 2/7 [29%], ICS n = 1/5 [20%]) when compared to the specialist genomics workforce (FP n = 1/43 [2%], ICS n = 0/35 [0%]).ConclusionsAfter completing one or both MOOCs, self-reported confidence in genomic variant interpretation significantly increased, and most respondents could correctly classify variants such that appropriate clinical management would be instigated. Genomics HCPs reported higher satisfaction with the level of content than the non-genomics clinicians. The MOOCs provided foundational knowledge and improved learner confidence, but should be adapted for different workforces to maximise the benefit for clinicians working in specialties outside of genetics.
| S-EPMC10386229 | biostudies-literature
Project description:Massive open online courses (MOOCs) generate learners' performance data that can be used to understand learners' proficiency and to improve their efficiency. However, the approaches currently used, such as assessing the proportion of correct responses in assessments, are oversimplified and may lead to poor conclusions and decisions because they do not account for additional information on learner, content, and context. There is a need for theoretically grounded data-driven explainable educational measurement approaches for MOOCs. In this conceptual paper, we try to establish a connection between psychometrics, a scientific discipline concerned with techniques for educational and psychological measurement, and MOOCs. First, we describe general principles of traditional measurement of learners' proficiency in education. Second, we discuss qualities of MOOCs which hamper direct application of approaches based on these general principles. Third, we discuss recent developments in measuring proficiency that may be relevant for analyzing MOOC data. Finally, we draw directions in psychometric modeling that might be interesting for future MOOC research.
| S-EPMC7243831 | biostudies-literature
Project description:Aims and method The mean delay for bipolar disorder diagnosis is 10 years. Identification of patients with previous hypomania is challenging, sometimes resulting in misdiagnosis. The aims of this study were: (a) to estimate the proportion of primary care patients with depression currently taking antidepressants who have undiagnosed bipolar disorder and (b) to compare a brief 3-item manic features questionnaire with the Hypomania Checklist (HCL-13). The sample comprised patients with a recorded diagnosis of depression, either on long-term antidepressant therapy or with previous multiple courses of antidepressants. Results Of 149 participants assessed, 24 (16.1%) satisfied criteria for bipolar disorder. Areas under the curve (AUC) for the 3-item questionnaire and the HCL-13 were similar (0.79 and 0.72, respectively) but positive predictive values (PPV) were low. Clinical implications Bipolar disorder may be underdiagnosed in primary care. A 3-item questionnaire could be used by general practitioners to screen for bipolar disorder in their patients with depression.
| S-EPMC5537571 | biostudies-other
Project description:Research related to the fit evaluation at the item level involving cognitive diagnosis models (CDMs) has been scarce. According to the parsimony principle, balancing goodness of fit against model complexity is necessary. General CDMs require a larger sample size to be estimated reliably, and can lead to worse attribute classification accuracy than the appropriate reduced models when the sample size is small and the item quality is poor, which is typically the case in many empirical applications. The main purpose of this study was to systematically examine the statistical properties of four inferential item-fit statistics: S-X2 , the likelihood ratio (LR) test, the Wald (W) test, and the Lagrange multiplier (LM) test. To evaluate the performance of the statistics, a comprehensive set of factors, namely, sample size, correlational structure, test length, item quality, and generating model, is systematically manipulated using Monte Carlo methods. Results show that the S-X2 statistic has unacceptable power. Type I error and power comparisons favor LR and W tests over the LM test. However, all the statistics are highly affected by the item quality. With a few exceptions, their performance is only acceptable when the item quality is high. In some cases, this effect can be ameliorated by an increase in sample size and test length. This implies that using the above statistics to assess item fit in practical settings when the item quality is low remains a challenge.
| S-EPMC5978477 | biostudies-literature
Project description:This study examined middle school students' perceptions of an automated writing evaluation (AWE) system, MI Write. We summarize students' perceptions of MI Write's usability, usefulness, and desirability both quantitatively and qualitatively. We then estimate hierarchical entry regression models that account for district context, classroom climate, demographic factors (i.e., gender, special education status, limited English proficiency status, socioeconomic status, grade), students' writing-related beliefs and affect, and students' writing proficiency as predictors of students' perceptions. Controlling for districts, students reporting more optimal classroom climate also reported higher usability, usefulness, and desirability for MI Write. Also, model results revealed that eighth graders, students with limited English proficiency, and students of lower socioeconomic status perceived MI Write relatively more useable; students with lower socioeconomic status also perceived MI Write relatively more useful and desirable. Students who liked writing more and more strongly believed that writing is a recursive process viewed MI Write as more useable, useful, and desirable. Students with greater writing proficiency viewed MI Write as less useable and useful; writing proficiency was not related to desirability perceptions. We conclude with a discussion of implications and future directions.
| S-EPMC10839244 | biostudies-literature
Project description:Linking of two forms is an important task when using item response theory, particularly when two forms are administered to nonequivalent groups. When linking with characteristic curve methods, the ability distribution and weights associated with that distribution can be used to weight observations differently. These are commonly specified as equally spaced intervals from -4 to 4, but other options or distributional forms can be specified. The use of these different distributions and weights of the ability distributions will be explored with a Monte Carlo simulation. Primary simulation conditions will include sample size, number of items, number of common items, ability distribution, and randomly varying population transformation constants. Study results show that the linking weights have little impact on the estimation of the linking constants; however, the underlying ability distribution of examinees does have significant impact. Implications for applied researchers will be discussed.
| S-EPMC5978475 | biostudies-literature
Project description:Pregnancy, birth and adjusting to a new baby is a potentially stressful time that can negatively affect the health of women. There is some evidence that expressive writing can have positive effects on psychological and physical health, particularly during stressful periods. The current study aimed to evaluate whether expressive writing would improve women's postpartum health. A randomized controlled trial was conducted with three conditions: expressive writing (n = 188), a control writing task (n = 213), or normal care (n = 163). Measures of psychological health, physical health and quality of life were measured at baseline (6-12 weeks postpartum), 1 and 6 months later. Ratings of stress were taken before and after the expressive writing task. Intent-to-treat analyses showed no significant differences between women in the expressive writing, control writing and normal care groups on measures of physical health, anxiety, depression, mood or quality of life at 1 and 6 months. Uptake and adherence to the writing tasks was low. However, women in the expressive writing group rated their stress as significantly reduced after completing the task. Cost analysis suggest women who did expressive writing had the lowest costs in terms of healthcare service use and lowest cost per unit of improvement in quality of life. Results suggest expressive writing is not effective as a universal intervention for all women 6-12 weeks postpartum. Future research should examine expressive writing as a targeted intervention for women in high-risk groups, such as those with mild or moderate depression, and further examine cost-effectiveness.Clinical trial registration number ISRCTN58399513 www.isrctn.com.
| S-EPMC6209049 | biostudies-literature
Project description:Rapid changes in technology force Higher Education Institutions (HEIs) to generate policies and permanent digital adaptations in their exercise of forming professionals through university professors. HEIs -in their permanent desire to qualify teaching faculty and graduate high-level professionals-develop continuous training events to strengthen and update techno-pedagogical skills that allow giving concrete responses to the needs of a globalized society during a human-educational crisis that arises from the COVID-19 pandemic. This study aims at analyzing whether nano-MOOCs improve digital teaching competences in university professors since in the scientific literature, this topic does not show with certainty the effectiveness of these types of courses in teacher training. By conducting a quantitative descriptive-inferential, comparative quasi-experimental research (pre-test and post-test) and with a sample made up of 297 faculty members from Universidad Técnica del Norte (UTN, Ibarra-Ecuador) belonging to the five academic units that compose it, it was identified that the teaching staff has limitations in two of the areas of competence that are articulated by INTEF Common Framework: creation of digital content and security; nevertheless, they did show optimal skills in the areas of information and information literacy, communication and collaboration, and problem solving. The findings also determined that online training based on a nano-MOOC format becomes a successful alternative for university faculty training, 83.84% of the participants under study improved their level of digital competence. These results show that an efficient customizable training can be achieved in less time and adjusted to the needs and characteristics of the professors. The criteria of various authors in this field are ratified with this research, it is, therefore, relevant to evaluate the level of digital competence of teachers and, based on that, be able to plan a personalized training program.
| S-EPMC9163514 | biostudies-literature
Project description:BackgroundThere has recently been growing interest in the roles of inflammation in contributing to the development of anxiety in people with immune-mediated inflammatory diseases (IMID). Patient-reported outcome measures can facilitate the assessment of physical and psychological functioning. The National Institutes of Health (NIH)'s Patient-Reported Outcomes Measurement Information System (PROMIS®) is a set of Patient-Reported Outcomes (PROs) that cover physical appearance, mental health, and social health. The PROMIS has been built through an Item Response Theory approach (IRT), a model-based measurement in which trait level estimates depend on both persons' responses and on the properties of the items that were administered. The aim of this study is to test the psychometric properties of an Italian custom four-item Short Form of the PROMIS Anxiety item bank in a cohort of outpatients with IMIDs.MethodsWe selected four items from the Italian standard Short Form Anxiety 8a and administered them to consecutive outpatients affected by Inflammatory Bowel disease (n = 246), rheumatological (n = 100) and dermatological (n = 43) diseases, and healthy volunteers (n = 280). Data was analyzed through an Item Response Theory (IRT) analysis in order to evaluate the psychometric properties of the Italian adaptation of the PROMIS anxiety short form.ResultsTaken together, Confirmatory Factor Analysis and Exploratory Factor analysis suggest that the unidimensionality assumption of the instrument holds. The instrument has excellent reliability from a Classical Theory of Test (CTT) standpoint (Cronbach's α = 0.93, McDonald's ω = 0.92). The 2PL Graded Response Model (GRM) model provided showed a better goodness of fit as compared to the 1PL GRM model, and local independence assumption appears to be met overall. We did not find signs of differential item functioning (DIF) for age and gender, but evidence for uniform (but not non-uniform) DIF was found in three out of four items for the patient vs. control group. Analysis of the test reliability curve suggested that the instrument is most reliable for higher levels of the latent trait of anxiety. The groups of patients exhibited higher levels of anxiety as compared to the control group (ps < 0.001, Bonferroni-corrected). The groups of patients were not different between themselves (p = 1, Bonferroni-corrected). T-scores based on estimated latent trait and raw scores were highly correlated (Pearson's r = 0.98) and led to similar results.DiscussionThe Italian custom four-item short form from the PROMIS anxiety form 8a shows acceptable psychometric properties both from a CTT and an IRT standpoint. The Test Reliability Curve shows that this instrument is mostly informative for people with higher levels of anxiety, making it particularly suitable for clinical populations such as IMID patients.
| S-EPMC8556715 | biostudies-literature