What Kind of Language is Easy to Language-Model Under Curriculum Learning?
Nadine El-Naggar, Tatsuki Kuribayashi, Ted Briscoe
TLDR
This paper investigates how curriculum learning affects language models' ability to learn typological language patterns.
Key contributions
- Explores how curriculum learning (CL) impacts language model (LM) inductive bias for typological patterns.
- Investigates if LMs can reproduce typological language feature configurations under CL.
- Introduces a simple CL variant, starting with simpler sentences, to LM-based typological studies.
- Demonstrates that CL substantially alters the apparent inductive bias of language models.
Why it matters
This paper is important as it reveals how curriculum learning significantly impacts language models' inductive bias when learning typological patterns. Understanding this interaction is crucial for designing more effective and developmentally-inspired AI language acquisition. It helps predict typological tendencies.
Original Abstract
Many of the thousands of attested languages share common configurations of features, creating a spectrum from typologically very rare (e.g., object-verb-subject word order) or impossible languages to very common combinations of features (e.g., subject-object-verb word order). One central question is under what conditions such typological tendencies can be predicted, and specifically whether the learning bias of language models (LMs) is sufficient to reproduce such patterns. In this study, we add one dimensionality to such analysis -- the learning scenario for LMs -- to explore its interaction with the inductive bias of LMs. Specifically, as a first study, we examine the effect of curriculum learning (CL), as a developmentally motivated learning scenario, i.e., starting with simpler sentences rather than randomly-ordered input. We expand existing LM-based exploration (El-Naggar et al., 2025a,b) with a simple CL variant and find that CL substantially impacts the apparent inductive bias of LMs.
📬 Weekly AI Paper Digest
Get the top 10 AI/ML arXiv papers from the week — summarized, scored, and delivered to your inbox every Monday.