Exploring Language-Agnosticity in Function Vectors: A Case Study in Machine Translation
Nurkhan Laiyk, Gerard I. Gállego, Javier Ferrando, Fajri Koto
TLDR
This paper shows that function vectors extracted for machine translation are language-agnostic, improving translation across unseen languages.
Key contributions
- Function vectors (FVs) for machine translation are language-agnostic across multilingual LLMs.
- FVs extracted from English-to-Target improve translation quality for multiple unseen target languages.
- Removing FVs degrades translation performance across languages, confirming their specific role.
- Base-model FVs transfer to instruction-tuned models and partially generalize from word to sentence translation.
Why it matters
This research demonstrates that function vectors can generalize across languages in machine translation. This finding is crucial for developing more efficient multilingual LLMs, potentially reducing the need for extensive language-specific fine-tuning and improving cross-lingual task transfer. Understanding FV properties is key for robust AI.
Original Abstract
Function vectors (FVs) are vector representations of tasks extracted from model activations during in-context learning. While prior work has shown that multilingual model representations can be language-agnostic, it remains unclear whether the same holds for function vectors. We study whether FVs exhibit language-agnosticity, using machine translation as a case study. Across three decoder-only multilingual LLMs, we find that translation FVs extracted from a single English$\rightarrow$Target direction transfer to other target languages, consistently improving the rank of correct translation tokens across multiple unseen languages. Ablation results show that removing the FV degrades translation across languages with limited impact on unrelated tasks. We further show that base-model FVs transfer to instruction-tuned variants and partially generalize from word-level to sentence-level translation.
📬 Weekly AI Paper Digest
Get the top 10 AI/ML arXiv papers from the week — summarized, scored, and delivered to your inbox every Monday.