Benchmarking Source-Sensitive Reasoning in Turkish: Humans and LLMs under Evidential Trust Manipulation
TLDR
Turkish speakers use evidential morphology based on source trustworthiness, a nuanced linguistic ability that current LLMs largely fail to replicate.
Key contributions
- Turkish speakers robustly use -DI for high-trust sources and -mIs for low-trust sources in past-domain contexts.
- LLMs show inconsistent and often reversed trust effects, heavily dependent on model and prompting paradigm.
- Reveals a significant human-LLM gap in source-sensitive evidential reasoning in Turkish.
Why it matters
This paper reveals a critical gap between human and LLM understanding of source trustworthiness in language. It's crucial for developing LLMs that can handle nuanced linguistic cues related to information reliability, especially in languages with rich evidential systems like Turkish.
Original Abstract
This paper investigates whether source trustworthiness shapes Turkish evidential morphology and whether large language models (LLMs) track this sensitivity. We study the past-domain contrast between -DI and -mIs in controlled cloze contexts where the information source is overtly external, while only its perceived reliability is manipulated (High-Trust vs. Low-Trust). In a human production experiment, native speakers of Turkish show a robust trust effect: High-Trust contexts yield relatively more -DI, whereas Low-Trust contexts yield relatively more -mIs, with the pattern remaining stable across sensitivity analyses. We then evaluate 10 LLMs in three prompting paradigms (open gap-fill, explicit past-tense gap-fill, and forced-choice A/B selection). LLM behavior is highly model- and prompt-dependent: some models show weak or local trust-consistent shifts, but effects are generally unstable, often reversed, and frequently overshadowed by output-compliance problems and strong base-rate suffix preferences. The results provide new evidence for a trust-/commitment-based account of Turkish evidentiality and reveal a clear human-LLM gap in source-sensitive evidential reasoning.
📬 Weekly AI Paper Digest
Get the top 10 AI/ML arXiv papers from the week — summarized, scored, and delivered to your inbox every Monday.