Extending Context Window of Large Language Models via Positional Interpolation
Shouyuan Chen, Sherman Wong, Liangjian Chen, Yuandong Tian
TLDR
Position Interpolation (PI) enables RoPE-based large language models to handle context windows up to 32,768 tokens with minimal fine-tuning while maintaining performance on both extended and original context lengths.
Key contributions
- Introduces Position Interpolation to linearly downscale position indices, avoiding unstable extrapolation in attention mechanisms.
- Extends context window size of pretrained LLaMA models from original limits up to 32,768 tokens with under 1000 fine-tuning steps.
- Demonstrates strong empirical results on long-context tasks like passkey retrieval, language modeling, and long document summarization.
Why it matters
This paper addresses a critical limitation in large language models—their fixed and relatively short context windows—by proposing a simple yet effective method to extend these windows significantly without architectural changes or heavy retraining. This advancement enables LLMs to better handle long documents and tasks requiring extensive context, broadening their applicability and improving performance in real-world scenarios where long-range dependencies are essential.
Original Abstract
We present Position Interpolation (PI) that extends the context window sizes of RoPE-based pretrained LLMs such as LLaMA models to up to 32768 with minimal fine-tuning (within 1000 steps), while demonstrating strong empirical results on various tasks that require long context, including passkey retrieval, language modeling, and long document summarization from LLaMA 7B to 65B. Meanwhile, the extended model by Position Interpolation preserve quality relatively well on tasks within its original context window. To achieve this goal, Position Interpolation linearly down-scales the input position indices to match the original context window size, rather than extrapolating beyond the trained context length which may lead to catastrophically high attention scores that completely ruin the self-attention mechanism. Our theoretical study shows that the upper bound of interpolation is at least $\sim 600 \times$ smaller than that of extrapolation, further demonstrating its stability. Models extended via Position Interpolation retain its original architecture and can reuse most pre-existing optimization and infrastructure.
📬 Weekly AI Paper Digest
Get the top 10 AI/ML arXiv papers from the week — summarized, scored, and delivered to your inbox every Monday.