Artificial Artificial Artificial Intelligence: Crowd Workers Widely Use Large Language Models for Text Production Tasks¶
Arxiv Link - 2023-06-13 16:46:24
Abstract¶
Large language models (LLMs) are remarkable data annotators. They can be used to generate high-fidelity supervised training data, as well as survey and experimental data. With the widespread adoption of LLMs, human gold--standard annotations are key to understanding the capabilities of LLMs and the validity of their results. However, crowdsourcing, an important, inexpensive way to obtain human annotations, may itself be impacted by LLMs, as crowd workers have financial incentives to use LLMs to increase their productivity and income. To investigate this concern, we conducted a case study on the prevalence of LLM usage by crowd workers. We reran an abstract summarization task from the literature on Amazon Mechanical Turk and, through a combination of keystroke detection and synthetic text classification, estimate that 33-46% of crowd workers used LLMs when completing the task. Although generalization to other, less LLM-friendly tasks is unclear, our results call for platforms, researchers, and crowd workers to find new ways to ensure that human data remain human, perhaps using the methodology proposed here as a stepping stone. Code/data: https://github.com/epfl-dlab/GPTurk
Socials¶
| X | |
|---|---|
| 🚀 Exciting insights into the impact of Large Language Models (LLMs) on crowdsourcing! 🤖💬 A recent study delved into the prevalence of LLM usage by crowd workers in data annotation tasks. The findings showed that 33-46% of crowd workers leveraged LLMs to enhance their productivity and earnings. This raises important considerations for ensuring the integrity of human-generated data in the era of advanced AI technologies. For a detailed overview of the study and its implications, check out the full paper here: http://arxiv.org/abs/2306.07899v1 #LLMs #Crowdsourcing #AI #DataAnnotation #TechResearch #ArtificialIntelligence #NLP Code and data from the study are available at: https://github.com/epfl-dlab/GPTurk Let's keep exploring the intersection of human intelligence and AI advancements! 🌐💡 #TechInnovation #Research #DataScience #MachineLearning #SocialMediaExpert #LinkedInPost |
🚀 New research alert! Learn how Large Language Models impact crowd workers and human annotations in AI tasks. A case study found that 33-46% of workers used LLMs on Amazon Mechanical Turk. Check out the study here: http://arxiv.org/abs/2306.07899v1 #AI #LLMs #NLP #Research Code/data available at: https://github.com/epfl-dlab/GPTurk |