Dataset Information

Directions in abusive language training data, a systematic review: Garbage in, garbage out.

ABSTRACT: Data-driven and machine learning based approaches for detecting, categorising and measuring abusive content such as hate speech and harassment have gained traction due to their scalability, robustness and increasingly high performance. Making effective detection systems for abusive content relies on having the right training datasets, reflecting a widely accepted mantra in computer science: Garbage In, Garbage Out. However, creating training datasets which are large, varied, theoretically-informed and that minimize biases is difficult, laborious and requires deep expertise. This paper systematically reviews 63 publicly available training datasets which have been created to train abusive language classifiers. It also reports on creation of a dedicated website for cataloguing abusive language data hatespeechdata.com. We discuss the challenges and opportunities of open science in this field, and argue that although more dataset sharing would bring many benefits it also poses social and ethical risks which need careful consideration. Finally, we provide evidence-based recommendations for practitioners creating new abusive content training datasets.

SUBMITTER: Vidgen B

PROVIDER: S-EPMC7769249 | biostudies-literature | 2020

REPOSITORIES: biostudies-literature

ACCESS DATA

Publications

Directions in abusive language training data, a systematic review: Garbage in, garbage out.

Vidgen Bertie B Derczynski Leon L

PloS one 20201228 12

Data-driven and machine learning based approaches for detecting, categorising and measuring abusive content such as hate speech and harassment have gained traction due to their scalability, robustness and increasingly high performance. Making effective detection systems for abusive content relies on having the right training datasets, reflecting a widely accepted mantra in computer science: Garbage In, Garbage Out. However, creating training datasets which are large, varied, theoretically-inform ...[more]

PMID: 33370298

Similar Datasets

Project description:BackgroundSocial media have transformed the communications landscape. People increasingly obtain news and health information online and via social media. Social media platforms also serve as novel sources of rich observational data for health research (including infodemiology, infoveillance, and digital disease detection detection). While the number of studies using social data is growing rapidly, very few of these studies transparently outline their methods for collecting, filtering, and reporting those data. Keywords and search filters applied to social data form the lens through which researchers may observe what and how people communicate about a given topic. Without a properly focused lens, research conclusions may be biased or misleading. Standards of reporting data sources and quality are needed so that data scientists and consumers of social media research can evaluate and compare methods and findings across studies.ObjectiveWe aimed to develop and apply a framework of social media data collection and quality assessment and to propose a reporting standard, which researchers and reviewers may use to evaluate and compare the quality of social data across studies.MethodsWe propose a conceptual framework consisting of three major steps in collecting social media data: develop, apply, and validate search filters. This framework is based on two criteria: retrieval precision (how much of retrieved data is relevant) and retrieval recall (how much of the relevant data is retrieved). We then discuss two conditions that estimation of retrieval precision and recall rely on--accurate human coding and full data collection--and how to calculate these statistics in cases that deviate from the two ideal conditions. We then apply the framework on a real-world example using approximately 4 million tobacco-related tweets collected from the Twitter firehose.ResultsWe developed and applied a search filter to retrieve e-cigarette-related tweets from the archive based on three keyword categories: devices, brands, and behavior. The search filter retrieved 82,205 e-cigarette-related tweets from the archive and was validated. Retrieval precision was calculated above 95% in all cases. Retrieval recall was 86% assuming ideal conditions (no human coding errors and full data collection), 75% when unretrieved messages could not be archived, 86% assuming no false negative errors by coders, and 93% allowing both false negative and false positive errors by human coders.ConclusionsThis paper sets forth a conceptual framework for the filtering and quality evaluation of social data that addresses several common challenges and moves toward establishing a standard of reporting social data. Researchers should clearly delineate data sources, how data were accessed and collected, and the search filter building process and how retrieval precision and recall were calculated. The proposed framework can be adapted to other public social media platforms.

Project description:ImportanceTraining parents to implement strategies to support child language development is crucial to support long-term outcomes, given that as many as 2 of 5 children younger than 5 years have difficulty learning language.ObjectiveTo examine the association between parent training and language and communication outcomes in young children.Data sourcesSearches of ERIC, Academic Search Complete, PsycINFO, and PsycARTICLES were conducted on August 11, 2014; August 18, 2016; January 23, 2018; and October 30, 2018.Study selectionStudies included in this review and meta-analysis were randomized or nonrandomized clinical trials that evaluated a language intervention that included parent training with children with a mean age of less than 6 years. Studies were excluded if the parent was not the primary implementer of the intervention, the study included fewer than 10 participants, or the study did not report outcomes related to language or communication.Data extraction and synthesisPreferred Reporting Items for Systematic Reviews and Meta-analyses (PRISMA) guidelines were applied to a total of 31 778 articles identified for screening, with the full text of 723 articles reviewed and 76 total studies ultimately included.Main outcomes and measuresMain outcomes included language and communication skills in children with primary or secondary language impairment and children at risk for language impairment.ResultsThis meta-analysis included 59 randomized clinical trials and 17 nonrandomized clinical trials including 5848 total participants (36.4 female [20.8%]; mean [SD] age, 3.5 [3.9] years). The intervention approach in 63 studies was a naturalistic teaching approach, and 16 studies used a primarily dialogic reading approach. There was a significant moderate association between parent training and child communication, engagement, and language outcomes (mean [SE] Hedges g, -0.33 [0.06]; P < .001). The association between parent training and parent use of language support strategies was large (mean [SE] Hedges g, 0.55 [0.11], P < .001). Children with developmental language disorder had the largest social communication outcomes (mean [SE] Hedges g, 0.37 [0.17]); large and significant associations were observed for receptive (mean [SE] Hedges g, 0.92 [0.30]) and expressive language (mean [SE] Hedges g, 0.83 [0.20]). Children at risk for language impairments had moderate effect sizes across receptive language (mean [SE] Hedges g, 0.28 [0.15]) and engagement outcomes (mean [SE] Hedges g, 0.36 [0.17]).Conclusions and relevanceThe findings suggest that training parents to implement language and communication intervention techniques is associated with improved outcomes for children and increased parent use of support strategies. These findings may have direct implications on intervention and prevention.

Project description:Background: Research on predictors of working memory training responsiveness, which could help tailor cognitive interventions individually, is a timely topic in healthy aging. However, the findings are highly heterogeneous, reporting partly conflicting results following a broad spectrum of methodological approaches to answer the question "who benefits most" from working memory training. Objective: The present systematic review aimed to systematically investigate prognostic factors and models for working memory training responsiveness in healthy older adults. Method: Four online databases were searched up to October 2019 (MEDLINE Ovid, Web of Science, CENTRAL, and PsycINFO). The inclusion criteria for full texts were publication in a peer-reviewed journal in English/German, inclusion of healthy older individuals aged ≥55 years without any neurological and/or psychiatric diseases including cognitive impairment, and the investigation of prognostic factors and/or models for training responsiveness after targeted working memory training in terms of direct training effects, near-transfer effects to verbal and visuospatial working memory as well as far-transfer effects to other cognitive domains and behavioral variables. The study design was not limited to randomized controlled trials. Results: A total of 16 studies including n = 675 healthy older individuals with a mean age of 63.0-86.8 years were included in this review. Within these studies, five prognostic model approaches and 18 factor finding approaches were reported. Risk of bias was assessed using the Quality in Prognosis Studies checklist, indicating that important information, especially regarding the domains study attrition, study confounding, and statistical analysis and reporting, was lacking throughout many of the investigated studies. Age, education, intelligence, and baseline performance in working memory or other cognitive domains were frequently investigated predictors across studies. Conclusions: Given the methodological shortcomings of the included studies, no clear conclusions can be drawn, and emerging patterns of prognostic effects will have to survive sound methodological replication in future attempts to promote precision medicine approaches in the context of working memory training. Methodological considerations are discussed, and our findings are embedded to the cognitive aging literature, considering, for example, the cognitive reserve framework and the compensation vs. magnification account. The need for personalized cognitive prevention and intervention methods to counteract cognitive decline in the aging population is high and the potential enormous. Registration: PROSPERO, ID CRD42019142750.

Dataset Information

Directions in abusive language training data, a systematic review: Garbage in, garbage out.

Publications

Directions in abusive language training data, a systematic review: Garbage in, garbage out.

Similar Datasets

OmicsDI is part of the ELIXIR infrastructure

Tweets