The Game of Prompts – Identifying Auxiliary Constructions with LLMs and Human Participants in a Language Game

Bajzát, Tímea Borbála and Indig, Balázs (2025) The Game of Prompts – Identifying Auxiliary Constructions with LLMs and Human Participants in a Language Game. In: Human Language Technologies as a Challenge for Computer Science and Linguistics – 2025. Uniwersytet im. Adama Mickiewicza w Poznaniu, Poznan, pp. 13-17. ISBN 9788323245407; 9788323245414

Preview

Text 2_vetulani_human_language_2025_amup_oa.pdf - Published Version Download (336kB) | Preview

Abstract

Recent work in theoretical linguistics has begun to integrate large language models (LLMs) with traditional corpus analysis to address limitations such as annotator fatigue and ethical constraints in human-subject research. While both methods rely on the statistical distribution of linguistic elements, LLMs offer a unique, scalable means of simulating human-like linguistic behaviour. In this paper, we examine Hungarian auxiliary constructions to investigate how closely LLMs align with human linguistic intuition. We compare responses from a gamified cloze test administered to human participants with outputs from multiple versions of GPT. Our results show that, although GPT-4o can achieve a human-level accuracy, it does so by using a high-recall, low-diversity strategy. In contrast, human participants demonstrate higher precision and more varied lexical choices. Although LLMs can mimic human-like semantic and syntactic intuitions, the qualitative differences in their frequency-driven strategies highlight their complementary role in linguistic research, rather than their ability to replace human judgement.

Actions (login required)

View Item