Dataset Information

Caveat emptor, computational social science: Large-scale missing data in a widely-published Reddit corpus.

ABSTRACT: As researchers use computational methods to study complex social behaviors at scale, the validity of this computational social science depends on the integrity of the data. On July 2, 2015, Jason Baumgartner published a dataset advertised to include "every publicly available Reddit comment" which was quickly shared on Bittorrent and the Internet Archive. This data quickly became the basis of many academic papers on topics including machine learning, social behavior, politics, breaking news, and hate speech. We have discovered substantial gaps and limitations in this dataset which may contribute to bias in the findings of that research. In this paper, we document the dataset, substantial missing observations in the dataset, and the risks to research validity from those gaps. In summary, we identify strong risks to research that considers user histories or network analysis, moderate risks to research that compares counts of participation, and lesser risk to machine learning research that avoids making representative claims about behavior and participation on Reddit.

SUBMITTER: Gaffney D

PROVIDER: S-EPMC6034852 | biostudies-literature | 2018

REPOSITORIES: biostudies-literature

ACCESS DATA

Publications

Caveat emptor, computational social science: Large-scale missing data in a widely-published Reddit corpus.

Gaffney Devin D Matias J Nathan JN

PloS one 20180706 7

As researchers use computational methods to study complex social behaviors at scale, the validity of this computational social science depends on the integrity of the data. On July 2, 2015, Jason Baumgartner published a dataset advertised to include "every publicly available Reddit comment" which was quickly shared on Bittorrent and the Internet Archive. This data quickly became the basis of many academic papers on topics including machine learning, social behavior, politics, breaking news, and ...[more]

PMID: 29979741

Similar Datasets

Project description:BackgroundUsers increasingly use social networking services (SNSs) to share their feelings and emotions. For those with mental disorders, SNSs can also be used to seek advice on mental health issues. One available SNS is Reddit, in which users can freely discuss such matters on relevant health diagnostic subreddits.ObjectiveIn this study, we analyzed the distinctive linguistic characteristics in users' posts on specific mental disorder subreddits (depression, anxiety, bipolar disorder, borderline personality disorder, schizophrenia, autism, and mental health) and further validated their distinctiveness externally by comparing them with posts of subreddits not related to mental illness. We also confirmed that these differences in linguistic formulations can be learned through a machine learning process.MethodsReddit posts uploaded by users were collected for our research. We used various statistical analysis methods in Linguistic Inquiry and Word Count (LIWC) software, including 1-way ANOVA and subsequent post hoc tests, to see sentiment differences in various lexical features within mental health-related subreddits and against unrelated ones. We also applied 3 supervised and unsupervised clustering methods for both cases after extracting textual features from posts on each subreddit using bidirectional encoder representations from transformers (BERT) to ensure that our data set is suitable for further machine learning or deep learning tasks.ResultsWe collected 3,133,509 posts of 919,722 Reddit users. The results using the data indicated that there are notable linguistic differences among the subreddits, consistent with the findings of prior research. The findings from LIWC analyses revealed that patients with each mental health issue show significantly different lexical and semantic patterns, such as word count or emotion, throughout their online social networking activities, with P<.001 for all cases. Furthermore, distinctive features of each subreddit group were successfully identified through supervised and unsupervised clustering methods, using the BERT embeddings extracted from textual posts. This distinctiveness was reflected in the Davies-Bouldin scores ranging from 0.222 to 0.397 and the silhouette scores ranging from 0.639 to 0.803 in the former case, with scores of 1.638 and 0.729, respectively, in the latter case.ConclusionsBy taking a multifaceted approach, analyzing textual posts related to mental health issues using statistical, natural language processing, and machine learning techniques, our approach provides insights into aspects of recent lexical usage and information about the linguistic characteristics of patients with specific mental health issues, which can inform clinicians about patients' mental health in diagnostic terms to aid online intervention. Our findings can further promote research areas involving linguistic analysis and machine learning approaches for patients with mental health issues by identifying and detecting mentally vulnerable groups of people online.

Dataset Information

Caveat emptor, computational social science: Large-scale missing data in a widely-published Reddit corpus.

Publications

Caveat emptor, computational social science: Large-scale missing data in a widely-published Reddit corpus.

Similar Datasets

OmicsDI is part of the ELIXIR infrastructure

Tweets