Dataset Information

Upon Repeated Reflection: Consequences of Frequent Exposure to the Cognitive Reflection Test for Mechanical Turk Participants.

ABSTRACT: Participants from public participant panels, such as Amazon Mechanical Turk, are shared across many labs and participate in many studies during their panel tenure. Here, I demonstrate direct and indirect downstream consequences of frequent exposure in three studies (N _1-3 = 3, 660), focusing on the cognitive reflection test (CRT), one of the most frequently used cognitive measures in online research. Study 1 explored several variants of the signature bat-and-ball item in samples recruited from Mechanical Turk. Panel tenure was shown to impact responses to both the original and merely similar items. Solution rates were not found to be higher than in a commercial online panel with less exposure to the CRT (Qualtrics panels, n = 1, 238). In Study 2, an alternative test with transformed numeric values showed higher correlations with validation measures than the original test. Finally, Study 3 investigated sources of item familiarity and measured performance on novel lure items.

SUBMITTER: Woike JK

PROVIDER: S-EPMC6909056 | biostudies-literature | 2019

REPOSITORIES: biostudies-literature

ACCESS DATA

Publications

Upon Repeated Reflection: Consequences of Frequent Exposure to the Cognitive Reflection Test for Mechanical Turk Participants.

Woike Jan K JK

Frontiers in psychology 20191206

Participants from public participant panels, such as Amazon Mechanical Turk, are shared across many labs and participate in many studies during their panel tenure. Here, I demonstrate direct and indirect downstream consequences of frequent exposure in three studies (<i>N</i> <sub>1-3</sub> = 3, 660), focusing on the cognitive reflection test (CRT), one of the most frequently used cognitive measures in online research. Study 1 explored several variants of the signature bat-and-ball item in sample ...[more]

PMID: 31866890

Similar Datasets

Project description:Internet interventions face significant challenges in recruitment and attrition rates are typically high and problematic. Finding innovative yet scientifically valid avenues for attaining and retaining participants is therefore of considerable importance. The main goal of this study was to compare recruitment process and participants characteristics between two similar randomized control trials of mood management interventions. One of the trials (Bunge et al., 2016) was conducted with participants recruited from Amazon's Mechanical Turk (AMT), and the other trial recruited via Unpaid Internet Resources (UIR). Methods:The AMT sample (Bunge et al., 2016) consisted of 765 adults, and the UIR sample (recruited specifically for this study) consisted of 329 adult US residents. Participants' levels of depression, anxiety, confidence, motivation, and perceived usefulness of the intervention were assessed. The AMT sample was financially compensated whereas the UIR was not. Results:AMT yielded higher recruitment rates per month (p?<?.05). At baseline, the AMT sample reported significantly lower depression and anxiety scores (p?<?.001 and p?<?.005, respectively) and significantly higher mood, motivation, and confidence (all p?<?.001) compared to the UIR sample. AMT participants spent significantly less time on the site (p?<?.05) and were more likely to complete follow-ups than the UIR sample (p?<?.05). Both samples reported a significant increase in their level of confidence and motivation from pre- to post-intervention. AMT participants showed a significant increase in perceived usefulness of the intervention (p?<?.0001), whereas the UIR sample did not (p?=?.1642). Conclusions:By using AMT, researchers can recruit very rapidly and obtain higher retention rates; however, these participants may not be representative of the general online population interested in clinical interventions. Considering that AMT and UIR participants differed in most baseline variables, data from clinical studies resulting from AMT samples should be interpreted with caution.

Dataset Information

Upon Repeated Reflection: Consequences of Frequent Exposure to the Cognitive Reflection Test for Mechanical Turk Participants.

Publications

Upon Repeated Reflection: Consequences of Frequent Exposure to the Cognitive Reflection Test for Mechanical Turk Participants.

Similar Datasets

OmicsDI is part of the ELIXIR infrastructure

Tweets