Robust Neural Automated Essay Scoring Using Item Response Theory
Ontology highlight
ABSTRACT:
SUBMITTER: Bittencourt I
PROVIDER: S-EPMC7334153 | biostudies-literature | 2020 Jun
REPOSITORIES: biostudies-literature
Similar Datasets
Project description:Assessment in the Education system plays a significant role in judging student performance. The present evaluation system is through human assessment. As the number of teachers' student ratio is gradually increasing, the manual evaluation process becomes complicated. The drawback of manual evaluation is that it is time-consuming, lacks reliability, and many more. This connection online examination system evolved as an alternative tool for pen and paper-based methods. Present Computer-based evaluation system works only for multiple-choice questions, but there is no proper evaluation system for grading essays and short answers. Many researchers are working on automated essay grading and short answer scoring for the last few decades, but assessing an essay by considering all parameters like the relevance of the content to the prompt, development of ideas, Cohesion, and Coherence is a big challenge till now. Few researchers focused on Content-based evaluation, while many of them addressed style-based assessment. This paper provides a systematic literature review on automated essay scoring systems. We studied the Artificial Intelligence and Machine Learning techniques used to evaluate automatic essay scoring and analyzed the limitations of the current studies and research trends. We observed that the essay evaluation is not done based on the relevance of the content and coherence.Supplementary informationThe online version contains supplementary material available at 10.1007/s10462-021-10068-2.
| S-EPMC8460059 | biostudies-literature
Project description:Composite assessments aim to combine different aspects of a disease in a single score and are utilized in a variety of therapeutic areas. The data arising from these evaluations are inherently discrete with distinct statistical properties. This tutorial presents the framework of the item response theory (IRT) for the analysis of this data type in a pharmacometric context. The article considers both conceptual (terms and assumptions) and practical questions (modeling software, data requirements, and model building).
| S-EPMC5915608 | biostudies-literature
Project description:Studying the biology of sleep requires the accurate assessment of the state of experimental subjects, and manual analysis of relevant data is a major bottleneck. Recently, deep learning applied to electroencephalogram and electromyogram data has shown great promise as a sleep scoring method, approaching the limits of inter-rater reliability. As with any machine learning algorithm, the inputs to a sleep scoring classifier are typically standardized in order to remove distributional shift caused by variability in the signal collection process. However, in scientific data, experimental manipulations introduce variability that should not be removed. For example, in sleep scoring, the fraction of time spent in each arousal state can vary between control and experimental subjects. We introduce a standardization method, mixture z-scoring, that preserves this crucial form of distributional shift. Using both a simulated experiment and mouse in vivo data, we demonstrate that a common standardization method used by state-of-the-art sleep scoring algorithms introduces systematic bias, but that mixture z-scoring does not. We present a free, open-source user interface that uses a compact neural network and mixture z-scoring to allow for rapid sleep scoring with accuracy that compares well to contemporary methods. This work provides a set of computational tools for the robust automation of sleep scoring.
| S-EPMC6910668 | biostudies-literature
Project description:Responses to survey data are determined not only by item characteristics and respondents' trait standings but also by response styles. Recently, methods for modeling response style with personality and attitudinal data have turned toward the use of anchoring vignettes, which provide fixed rating targets. Although existing research is promising, a few outstanding questions remain. First, it is not known how many vignettes and vignette ratings are necessary to identify response style parameters. Second, the comparative accuracy of these models is largely unexplored. Third, it remains unclear whether correcting for response style improves criterion validity. Both simulated data and data observed from a population-representative sample responding to a measure of personality pathology (the Personality Inventory for DSM-5 [PID-5]) were modeled using an array of response style models. In simulations, most models estimating response styles outperformed the graded response model (GRM), and in observed data, all response style models were superior to the GRM. Correcting for response style had a small, but in some cases significant, effect on the prediction of self-reported social dysfunction.
| S-EPMC6297915 | biostudies-literature
Project description:Background: Subjective well-being refers to the extent to which a person believes or feels that her life is going well. It is considered as one of the best available proxies for a broader, more canonical form of well-being. For over 30 years, one important distinction in the conceptualization of subjective well-being is the contrast between more affective evaluations of biological emotional reactions and more cognitive evaluations of one's life in relation to a psychologically self-imposed ideal. More recently, researchers have suggested the addition of harmony in life, comprising behavioral evaluations of how one is doing in a social context. Since measures used to assess subjective well-being are self-reports, often validated only using Classical Test Theory, our aim was to focus on the psychometric properties of the measures using Item Response Theory. Method: A total of 1000 participants responded to the Positive Affect Negative Affect Schedule. At random, half of the participants answered to the Satisfaction with Life Scale or to the Harmony in life Scale. First, we evaluate and provide enough evidence of unidimensionality for each scale. Next, we conducted graded response models to validate the psychometric properties of the subjective well-being scales. Results: All scales showed varied frequency item distribution, high discrimination values (Alphas), and had different difficulty parameters (Beta) on each response options. For example, we identified items that respondents found difficult to endorse at the highest/lowest point of the scales (e.g., "Proud" for positive affect; item 5, "If I could live my life over, I would change almost nothing," for life satisfaction; and item 3, "I am in harmony," for harmony in life). In addition, all scales could cover a good portion of the range of subjective well-being (Theta): -2.50 to 2.30 for positive affect, -1.00 to 3.50 for negative affect, -2.40 to 2.50 for life satisfaction, and -2.40 to 2.50 for harmony in life. Importantly, for all scales, there were weak reliability for respondents with extreme latent scores of subjective well-being. Conclusion: The affective component, especially low levels of negative affect, were less accurately measured, while both the cognitive and social component were covered to an equal degree. There was less reliability for respondents with extreme latent scores of subjective well-being. Thus, to improve reliability at the level of the scale, at the item level and at the level of the response scale for each item, we point out specific items that need to be modified or added. Moreover, the data presented here can be used as normative data for each of the subjective well-being constructs.
| S-EPMC6987476 | biostudies-literature
Project description:BackgroundTraumatic Antecedents Questionnaire (TAQ) is a traumatic experience scale that measures exposure to traumatic events across four age periods. Although the TAQ has good convergent validity with other traumatic scales, the classification of the domains and the psychometric properties of the scale has not been verified.MethodsA total of 290 young adults completed the TAQ, and 156 participated in the retest. The number of trauma domains was determined using principal component analysis. Rasch model was used for verifying the items that each domain might represent in one common measurement.ResultsWhen scores were transformed as binary a code 0 and 1 from the original 4 categories, 8 domains were established consisting of Domestic violence, Sexual/other rare trauma, Incompetence, Caring family, Accidents to close person, Unstable caring environment, Safe environment, and Lack of sexual/extreme trauma. Most domains had acceptable psychometric properties with a mean-square fit value within the range of 0.7-1.3. The Bland and Altman analysis suggest 98.7% of difference scores between test and retest were within ±2 standard deviations from the mean. TAQ severity showed a significant relationship with the multiplicity score of the Maltreatment and Abuse Chronology of Exposure scale (r = 0.677).ConclusionsThe newly proposed scoring system and 8 domains for the TAQ demonstrated excellent internal consistency, test-retest reliability, and validity. Further studies are needed to develop new items in domains with less than 5 items to improve the psychometric properties of the scale and to create a maltreatment domain that includes bullying items.
| S-EPMC7749547 | biostudies-literature
Project description:Multidimensional item response theory (MIRT) models for response style (e.g., Bolt, Lu, & Kim, 2014, Psychological Methods, 19, 528; Falk & Cai, 2016, Psychological Methods, 21, 328) provide flexibility in accommodating various response styles, but often present difficulty in isolating the effects of response style(s) from the intended substantive trait(s). In the presence of such measurement limitations, we consider several ways in which MIRT models are nevertheless useful in lending insight into how response styles may interfere with measurement for a given test instrument. Such a study can also inform whether alternative design considerations (e.g., anchoring vignettes, self-report items of heterogeneous content) that seek to control for response style effects may be helpful. We illustrate several aspects of an MIRT approach using real and simulated analyses.
| S-EPMC6765459 | biostudies-literature
Project description:Current use of multidimensional computerized adaptive testing (MCAT) has been developed in conjunction with compensatory multidimensional item response theory (MIRT) models rather than with non-compensatory ones. In recognition of the usefulness of MCAT and the complications associated with non-compensatory data, this study aimed to develop MCAT algorithms using non-compensatory MIRT models and to evaluate their performance. For the purpose of the study, three item selection methods were adapted and compared, namely, the Fisher information method, the mutual information method, and the Kullback-Leibler information method. The results of a series of simulations showed that the Fisher information and mutual information methods performed similarly, and both outperformed the Kullback-Leibler information method. In addition, it was found that the more stringent the termination criterion and the higher the correlation between the latent traits, the higher the resulting measurement precision and test reliability. Test reliability was very similar across the dimensions, regardless of the correlation between the latent traits and termination criterion. On average, the difficulties of the administered items were found to be at a lower level than the examinees' abilities, which shed light on item bank construction for non-compensatory items.
| S-EPMC6696872 | biostudies-literature
Project description:BackgroundOur study aims to provide validity evidence for the EuroQol five dimensions questionnaire (EQ-5D) in the National Health Interview Survey of Taiwan in the 2013 wave and further interpret the EQ-5D scores for patients with chronic diseases. Another goal of the study was to use item response theory (IRT) to identify items that are informative for assessing quality of life using EQ-5D.MethodsThere were 17,260 participants, aged 12-64, who completed the interviews in our study. Psychometric methods, including factor analysis and the IRT model known as the Graded Response Model (GRM), were used to assess the unidimensionality of EQ-5D and its item properties. Correlation analysis was used to assess whether EQ-5D scores are associated with scores from the 36-Item Short Form Survey (SF-36).ResultsThe EQ-5D scores have moderate internal consistency (Cronbach's alpha: 0.60) and a scree plot suggests that the EQ-5D measure is unidimensional. The item information function analysis from the IRT model demonstrates that the first 3 items, "mobility," "self-care," and "usual activities" are the most informative items for patients who have chronic diseases and health-related quality of life below the 10th percentile. The EQ-5D scores have a moderate correlation (r: 0.61) with SF-36 scores.ConclusionsThe EQ-5D scale shows promise for use in the general population. The IRT model informs our interpretation of the EQ-5D scores. Given the time constraints in clinical settings, we suggest using the first three items in EQ-5D to measure the health-related quality of life for patients with chronic diseases.
| S-EPMC8684691 | biostudies-literature
Project description:This article describes the item parameters analysis result of the space-relations subtest measurement. This subtest is part of the differential aptitude test (DAT) instrument. The item parameters are characteristic psychometric refer quality of the item. The Item parameters than analyzed in this instrument are item fit model, item difficulty, item discrimination, pseudo-guessing, item information curves, and test information function. The data was collected through documentation technique from the space-relation test conducted at Biro Psikologi (Psychology Bureau) UNY, amounting to 1046 students from Yogyakarta, Indonesia. Data were analyzed using item response analysis with the assistance of BILOG program.
| S-EPMC6141381 | biostudies-literature