Dataset Information

Multiple-Choice Item Distractor Development Using Topic Modeling Approaches.

ABSTRACT: Writing a high-quality, multiple-choice test item is a complex process. Creating plausible but incorrect options for each item poses significant challenges for the content specialist because this task is often undertaken without implementing a systematic method. In the current study, we describe and demonstrate a systematic method for creating plausible but incorrect options, also called distractors, based on students' misconceptions. These misconceptions are extracted from the labeled written responses. One thousand five hundred and fifteen written responses from an existing constructed-response item in Biology from Grade 10 students were used to demonstrate the method. Using a topic modeling procedure commonly used with machine learning and natural language processing called latent dirichlet allocation, 22 plausible misconceptions from students' written responses were identified and used to produce a list of plausible distractors based on students' responses. These distractors, in turn, were used as part of new multiple-choice items. Implications for item development are discussed.

SUBMITTER: Shin J

PROVIDER: S-EPMC6524712 | biostudies-literature | 2019

REPOSITORIES: biostudies-literature

ACCESS DATA

Publications

Multiple-Choice Item Distractor Development Using Topic Modeling Approaches.

Shin Jinnie J Guo Qi Q Gierl Mark J MJ

Frontiers in psychology 20190425

Writing a high-quality, multiple-choice test item is a complex process. Creating plausible but incorrect options for each item poses significant challenges for the content specialist because this task is often undertaken without implementing a systematic method. In the current study, we describe and demonstrate a systematic method for creating plausible but incorrect options, also called distractors, based on students' misconceptions. These misconceptions are extracted from the labeled written r ...[more]

PMID: 31133911

Similar Datasets

Project description:BackgroundYin deficiency (YD) is a pathological condition characterized by emaciation, afternoon fever, dry mouth, and night sweats. The incidence of YD is 23.3%. A 27-item Yin Deficiency Scale (YDS) was developed to estimate the clinical severity of YD. This study aimed to develop three short-form YDS versions to reduce the burden of response time, using three item-reduction approaches: Rasch, equidiscriminatory item-total correlation (EITC), and factor-based analyses.MethodsTwo datasets were analyzed from previous studies (169 outpatients from May to June 2009 and 237 healthy college students from January to April 2016). The optimal response category was examined using Rasch analysis. Items with higher item-total correlations were determined using the EITC. Using a factor-based approach, the items were reduced, while maintaining the original YDS construct. Reliability was estimated using the person separation index (PSI) and Cronbach's α values. The predictive accuracy was examined using the area under the curve (AUC). Finally, the relationship between YD and dysfunctional breathing (DB) was examined using factor scores from the YDS and the Korean version of the Nijmegen Questionnaire (KNQ).ResultsWe developed two 14-item YDS versions using the Rasch and EITC approaches, and a 16-item YDS version using a factor-based approach. Rasch analysis suggested an optimal response category of five points. The PSI of Rasch and Cronbach's α of the EITC and factor-based versions were 2.19, 0.855, and 0.827. The AUCs of the three short-form YDS were 0.812, 0.811, and 0.818. The sensitivity of the EITC-YDS was 0.632, which was lower than its specificity of 0.875. The fatigue-related scores of the factor-based YDS were fairly correlated with the factor scores of the KNQ estimating the DB (r = 0.349-0.499).ConclusionThe 14-item Rasch- and 16-item factor-based YDS may replace the original YDS during YD's primary screening, epidemiological surveys, and health checkups.

Project description:BACKGROUND: The Food and Drug Administration (FDA) approved drug labels contain a broad array of information, ranging from adverse drug reactions (ADRs) to drug efficacy, risk-benefit consideration, and more. However, the labeling language used to describe these information is free text often containing ambiguous semantic descriptions, which poses a great challenge in retrieving useful information from the labeling text in a consistent and accurate fashion for comparative analysis across drugs. Consequently, this task has largely relied on the manual reading of the full text by experts, which is time consuming and labor intensive. METHOD: In this study, a novel text mining method with unsupervised learning in nature, called topic modeling, was applied to the drug labeling with a goal of discovering "topics" that group drugs with similar safety concerns and/or therapeutic uses together. A total of 794 FDA-approved drug labels were used in this study. First, the three labeling sections (i.e., Boxed Warning, Warnings and Precautions, Adverse Reactions) of each drug label were processed by the Medical Dictionary for Regulatory Activities (MedDRA) to convert the free text of each label to the standard ADR terms. Next, the topic modeling approach with latent Dirichlet allocation (LDA) was applied to generate 100 topics, each associated with a set of drugs grouped together based on the probability analysis. Lastly, the efficacy of the topic modeling was evaluated based on known information about the therapeutic uses and safety data of drugs. RESULTS: The results demonstrate that drugs grouped by topics are associated with the same safety concerns and/or therapeutic uses with statistical significance (P<0.05). The identified topics have distinct context that can be directly linked to specific adverse events (e.g., liver injury or kidney injury) or therapeutic application (e.g., antiinfectives for systemic use). We were also able to identify potential adverse events that might arise from specific medications via topics. CONCLUSIONS: The successful application of topic modeling on the FDA drug labeling demonstrates its potential utility as a hypothesis generation means to infer hidden relationships of concepts such as, in this study, drug safety and therapeutic use in the study of biomedical documents.

Dataset Information

Multiple-Choice Item Distractor Development Using Topic Modeling Approaches.

Publications

Multiple-Choice Item Distractor Development Using Topic Modeling Approaches.

Similar Datasets

OmicsDI is part of the ELIXIR infrastructure

Tweets