{"id":680152,"date":"2025-07-16T09:45:34","date_gmt":"2025-07-16T06:45:34","guid":{"rendered":"https:\/\/buradabiliyorum.com\/en\/research-highlights-unreliable-responses-from-most-amazon-mturk-users-except-for-master-workers\/"},"modified":"2025-07-16T09:45:34","modified_gmt":"2025-07-16T06:45:34","slug":"research-highlights-unreliable-responses-from-most-amazon-mturk-users-except-for-master-workers","status":"publish","type":"post","link":"https:\/\/buradabiliyorum.com\/en\/research-highlights-unreliable-responses-from-most-amazon-mturk-users-except-for-master-workers\/","title":{"rendered":"Research highlights unreliable responses from most Amazon MTurk users, except for ‘master’ workers"},"content":{"rendered":"
\n
\n
\n
\n \"online\"
\n Credit: CC0 Public Domain
\n <\/figcaption><\/figure>\n<\/p><\/div>\n<\/div>\n

A new study led by Dr. Vadim Axelrod, of the Gonda (Goldschmied) Multidisciplinary Brain Research Center at Bar-Ilan University, has revealed serious concerns about the quality of data collected on Amazon Mechanical Turk’s (MTurk)\u2014a platform widely used for behavioral and psychological research.<\/p>\n

MTurk, an online crowdsourcing marketplace where individuals complete small tasks for payment, has served as a key resource for researchers for over 15 years. Despite previous concerns about participant quality, the platform remains popular within the academic community. Dr. Axelrod’s team set out to rigorously assess the current quality of data produced by MTurk participants.<\/p>\n

The study, involving over 1,300 participants across main and replication experiments, employed a straightforward but powerful method: repeating identical questionnaire items to measure response consistency. “If a participant is reliable, their answers to repeated questions should be consistent,” added Dr. Axelrod. In addition, the study included different types of “attentional catch” questions that should be easily answered by any attentive respondent.<\/p>\n

The findings, just published in Royal Society Open Science<\/a><\/i>, were stark: the majority of participants from MTurk’s general<\/a> worker pool failed the attention checks and demonstrated highly inconsistent responses, even when the sample was limited to users with a 95% or higher app<\/a>roval rating.<\/p>\n

“It’s hard to trust the data of someone who claims a runner isn’t tired after completing a marathon in extremely hot weather or that a cancer diagnosis would make someone glad,” Dr. Axelrod noted.<\/p>\n

“The participants did not lack the knowledge to answer such attentional catch questions\u2014they just weren’t paying sufficient attention. The implication is that their responses to the main questionnaire may be equally random.”<\/p>\n

By contrast, Amazon’s elite “Master” workers\u2014selected by Amazon based on high performance across previous tasks\u2014consistently produced high-quality data. The authors recommend using Master workers for future research, taking into consideration that these participants are much more experienced and far fewer in number.<\/p>\n

“Reliable data is the foundation of any empirical science<\/a>,” said Dr. Axelrod. “Researchers need to be fully informed about the reliability of their participant pool. Our findings suggest that caution is warranted when using MTurk’s general pool for behavioral research.”<\/p>\n

\n

More information:<\/strong>
\n Assessing the quality and reliability of the Amazon Mechanical Turk (MTurk) data in 2024, Royal Society Open Science<\/i> (2025). DOI: 10.1098\/rsos.250361<\/a>. royalsocietypublishing.org\/doi\/10.1098\/rsos.250361<\/a><\/p>\n<\/p><\/div>\n

\n\t\t\t\t\t\t\t\t\t\t\t\tProvided by
\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\tBar-Ilan University
\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t
\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t<\/svg>
\n\t\t\t\t\t\t\t\t\t\t\t\t\t<\/a><\/p><\/div>\n

<\/p>\n

\n

\n Citation<\/strong>:
\n Research highlights unreliable responses from most Amazon MTurk users, except for ‘master’ workers (2025, July 15)
\n retrieved 16 July 2025
\n from https:\/\/medicalxpress.com\/news<\/a>\/2025-07-highlights-unreliable-responses-amazon-mturk.html\n <\/p>\n

\n This document is subject to copyright. Apart from any fair dealing for the purpose of private study or research, no
\n part may be reproduced without the written permission. The content is provided for information purposes only.\n <\/p>\n<\/p><\/div>\n<\/p><\/div>\n