Dataset Information

Directions in abusive language training data, a systematic review: Garbage in, garbage out.

ABSTRACT: Data-driven and machine learning based approaches for detecting, categorising and measuring abusive content such as hate speech and harassment have gained traction due to their scalability, robustness and increasingly high performance. Making effective detection systems for abusive content relies on having the right training datasets, reflecting a widely accepted mantra in computer science: Garbage In, Garbage Out. However, creating training datasets which are large, varied, theoretically-informed and that minimize biases is difficult, laborious and requires deep expertise. This paper systematically reviews 63 publicly available training datasets which have been created to train abusive language classifiers. It also reports on creation of a dedicated website for cataloguing abusive language data hatespeechdata.com. We discuss the challenges and opportunities of open science in this field, and argue that although more dataset sharing would bring many benefits it also poses social and ethical risks which need careful consideration. Finally, we provide evidence-based recommendations for practitioners creating new abusive content training datasets.

SUBMITTER: Vidgen B

PROVIDER: S-EPMC7769249 | biostudies-literature | 2020

REPOSITORIES: biostudies-literature

ACCESS DATA

Publications

Directions in abusive language training data, a systematic review: Garbage in, garbage out.

Vidgen Bertie B Derczynski Leon L

PloS one 20201228 12

Data-driven and machine learning based approaches for detecting, categorising and measuring abusive content such as hate speech and harassment have gained traction due to their scalability, robustness and increasingly high performance. Making effective detection systems for abusive content relies on having the right training datasets, reflecting a widely accepted mantra in computer science: Garbage In, Garbage Out. However, creating training datasets which are large, varied, theoretically-inform ...[more]

PMID: 33370298

Similar Datasets

Project description:ImportanceTraining parents to implement strategies to support child language development is crucial to support long-term outcomes, given that as many as 2 of 5 children younger than 5 years have difficulty learning language.ObjectiveTo examine the association between parent training and language and communication outcomes in young children.Data sourcesSearches of ERIC, Academic Search Complete, PsycINFO, and PsycARTICLES were conducted on August 11, 2014; August 18, 2016; January 23, 2018; and October 30, 2018.Study selectionStudies included in this review and meta-analysis were randomized or nonrandomized clinical trials that evaluated a language intervention that included parent training with children with a mean age of less than 6 years. Studies were excluded if the parent was not the primary implementer of the intervention, the study included fewer than 10 participants, or the study did not report outcomes related to language or communication.Data extraction and synthesisPreferred Reporting Items for Systematic Reviews and Meta-analyses (PRISMA) guidelines were applied to a total of 31 778 articles identified for screening, with the full text of 723 articles reviewed and 76 total studies ultimately included.Main outcomes and measuresMain outcomes included language and communication skills in children with primary or secondary language impairment and children at risk for language impairment.ResultsThis meta-analysis included 59 randomized clinical trials and 17 nonrandomized clinical trials including 5848 total participants (36.4 female [20.8%]; mean [SD] age, 3.5 [3.9] years). The intervention approach in 63 studies was a naturalistic teaching approach, and 16 studies used a primarily dialogic reading approach. There was a significant moderate association between parent training and child communication, engagement, and language outcomes (mean [SE] Hedges g, -0.33 [0.06]; P < .001). The association between parent training and parent use of language support strategies was large (mean [SE] Hedges g, 0.55 [0.11], P < .001). Children with developmental language disorder had the largest social communication outcomes (mean [SE] Hedges g, 0.37 [0.17]); large and significant associations were observed for receptive (mean [SE] Hedges g, 0.92 [0.30]) and expressive language (mean [SE] Hedges g, 0.83 [0.20]). Children at risk for language impairments had moderate effect sizes across receptive language (mean [SE] Hedges g, 0.28 [0.15]) and engagement outcomes (mean [SE] Hedges g, 0.36 [0.17]).Conclusions and relevanceThe findings suggest that training parents to implement language and communication intervention techniques is associated with improved outcomes for children and increased parent use of support strategies. These findings may have direct implications on intervention and prevention.

Project description:Background: Research on predictors of working memory training responsiveness, which could help tailor cognitive interventions individually, is a timely topic in healthy aging. However, the findings are highly heterogeneous, reporting partly conflicting results following a broad spectrum of methodological approaches to answer the question "who benefits most" from working memory training. Objective: The present systematic review aimed to systematically investigate prognostic factors and models for working memory training responsiveness in healthy older adults. Method: Four online databases were searched up to October 2019 (MEDLINE Ovid, Web of Science, CENTRAL, and PsycINFO). The inclusion criteria for full texts were publication in a peer-reviewed journal in English/German, inclusion of healthy older individuals aged ≥55 years without any neurological and/or psychiatric diseases including cognitive impairment, and the investigation of prognostic factors and/or models for training responsiveness after targeted working memory training in terms of direct training effects, near-transfer effects to verbal and visuospatial working memory as well as far-transfer effects to other cognitive domains and behavioral variables. The study design was not limited to randomized controlled trials. Results: A total of 16 studies including n = 675 healthy older individuals with a mean age of 63.0-86.8 years were included in this review. Within these studies, five prognostic model approaches and 18 factor finding approaches were reported. Risk of bias was assessed using the Quality in Prognosis Studies checklist, indicating that important information, especially regarding the domains study attrition, study confounding, and statistical analysis and reporting, was lacking throughout many of the investigated studies. Age, education, intelligence, and baseline performance in working memory or other cognitive domains were frequently investigated predictors across studies. Conclusions: Given the methodological shortcomings of the included studies, no clear conclusions can be drawn, and emerging patterns of prognostic effects will have to survive sound methodological replication in future attempts to promote precision medicine approaches in the context of working memory training. Methodological considerations are discussed, and our findings are embedded to the cognitive aging literature, considering, for example, the cognitive reserve framework and the compensation vs. magnification account. The need for personalized cognitive prevention and intervention methods to counteract cognitive decline in the aging population is high and the potential enormous. Registration: PROSPERO, ID CRD42019142750.

Dataset Information

Directions in abusive language training data, a systematic review: Garbage in, garbage out.

Publications

Directions in abusive language training data, a systematic review: Garbage in, garbage out.

Similar Datasets

OmicsDI is part of the ELIXIR infrastructure