Dataset Information

Comparing Amazon Mechanical Turk with unpaid internet resources in online clinical trials.

ABSTRACT: Internet interventions face significant challenges in recruitment and attrition rates are typically high and problematic. Finding innovative yet scientifically valid avenues for attaining and retaining participants is therefore of considerable importance. The main goal of this study was to compare recruitment process and participants characteristics between two similar randomized control trials of mood management interventions. One of the trials (Bunge et al., 2016) was conducted with participants recruited from Amazon's Mechanical Turk (AMT), and the other trial recruited via Unpaid Internet Resources (UIR). Methods:The AMT sample (Bunge et al., 2016) consisted of 765 adults, and the UIR sample (recruited specifically for this study) consisted of 329 adult US residents. Participants' levels of depression, anxiety, confidence, motivation, and perceived usefulness of the intervention were assessed. The AMT sample was financially compensated whereas the UIR was not. Results:AMT yielded higher recruitment rates per month (p?

SUBMITTER: Bunge E

PROVIDER: S-EPMC6096331 | biostudies-literature | 2018 Jun

REPOSITORIES: biostudies-literature

ACCESS DATA

Publications

Comparing Amazon Mechanical Turk with unpaid internet resources in online clinical trials.

Bunge Eduardo E Cook Haley M HM Bond Melissa M Williamson Rachel E RE Cano Monique M Barrera Alinne Z AZ Leykin Yan Y Muñoz Ricardo F RF

Internet interventions 20180415

Internet interventions face significant challenges in recruitment and attrition rates are typically high and problematic. Finding innovative yet scientifically valid avenues for attaining and retaining participants is therefore of considerable importance. The main goal of this study was to compare recruitment process and participants characteristics between two similar randomized control trials of mood management interventions. One of the trials (Bunge et al., 2016) was conducted with participan ...[more]

PMID: 30135770

Similar Datasets

Project description:BackgroundObtaining a diagnosis of neuropsychiatric disorders such as autism requires long waiting times that can exceed a year and can be prohibitively expensive. Crowdsourcing approaches may provide a scalable alternative that can accelerate general access to care and permit underserved populations to obtain an accurate diagnosis.ObjectiveWe aimed to perform a series of studies to explore whether paid crowd workers on Amazon Mechanical Turk (AMT) and citizen crowd workers on a public website shared on social media can provide accurate online detection of autism, conducted via crowdsourced ratings of short home video clips.MethodsThree online studies were performed: (1) a paid crowdsourcing task on AMT (N=54) where crowd workers were asked to classify 10 short video clips of children as "Autism" or "Not autism," (2) a more complex paid crowdsourcing task (N=27) with only those raters who correctly rated ?8 of the 10 videos during the first study, and (3) a public unpaid study (N=115) identical to the first study.ResultsFor Study 1, the mean score of the participants who completed all questions was 7.50/10 (SD 1.46). When only analyzing the workers who scored ?8/10 (n=27/54), there was a weak negative correlation between the time spent rating the videos and the sensitivity (?=-0.44, P=.02). For Study 2, the mean score of the participants rating new videos was 6.76/10 (SD 0.59). The average deviation between the crowdsourced answers and gold standard ratings provided by two expert clinical research coordinators was 0.56, with an SD of 0.51 (maximum possible SD is 3). All paid crowd workers who scored 8/10 in Study 1 either expressed enjoyment in performing the task in Study 2 or provided no negative comments. For Study 3, the mean score of the participants who completed all questions was 6.67/10 (SD 1.61). There were weak correlations between age and score (r=0.22, P=.014), age and sensitivity (r=-0.19, P=.04), number of family members with autism and sensitivity (r=-0.195, P=.04), and number of family members with autism and precision (r=-0.203, P=.03). A two-tailed t test between the scores of the paid workers in Study 1 and the unpaid workers in Study 3 showed a significant difference (P<.001).ConclusionsMany paid crowd workers on AMT enjoyed answering screening questions from videos, suggesting higher intrinsic motivation to make quality assessments. Paid crowdsourcing provides promising screening assessments of pediatric autism with an average deviation <20% from professional gold standard raters, which is potentially a clinically informative estimate for parents. Parents of children with autism likely overfit their intuition to their own affected child. This work provides preliminary demographic data on raters who may have higher ability to recognize and measure features of autism across its wide range of phenotypic manifestations.

Dataset Information

Comparing Amazon Mechanical Turk with unpaid internet resources in online clinical trials.

Publications

Comparing Amazon Mechanical Turk with unpaid internet resources in online clinical trials.

Similar Datasets

OmicsDI is part of the ELIXIR infrastructure