StraTA: Incentivizing Agentic Reinforcement Learning with Strategic Trajectory Abstraction
Xiangyuan Xue, Yifan Zhou, Zidong Wang, Shengji Tang, Philip Torr + 3 more
TLDR
StraTA introduces strategic trajectory abstraction to agentic RL, improving LLM performance in long-horizon tasks by enhancing exploration and credit assignment.
Key contributions
- Introduces StraTA, a framework that adds explicit trajectory-level strategies to agentic RL for LLMs.
- Uses a hierarchical GRPO-style rollout, diverse strategy sampling, and self-judgment for joint strategy/action training.
- Achieves state-of-the-art success rates: 93.1% on ALFWorld, 84.2% on WebShop, and 63.5% on SciWorld.
Why it matters
This paper addresses a key challenge in LLM agents: long-horizon decision-making. By introducing strategic planning, StraTA significantly boosts performance and sample efficiency. It demonstrates a novel approach to make LLMs more effective in complex, interactive environments, even outperforming advanced commercial models.
Original Abstract
Large language models (LLMs) are increasingly used as interactive agents, but optimizing them for long-horizon decision making remains difficult because current methods are largely purely reactive, which weakens both exploration and credit assignment over extended trajectories. In this work, we present Strategic Trajectory Abstraction (StraTA), a simple framework that introduces an explicit trajectory-level strategy into agentic reinforcement learning (RL). StraTA samples a compact strategy from the initial task state, conditions subsequent actions on that strategy, and trains strategy generation and action execution jointly with a hierarchical GRPO-style rollout design, further enhanced by diverse strategy rollout and critical self-judgment. Experiments on ALFWorld, WebShop, and SciWorld show that StraTA consistently improves both sample efficiency and final performance over strong baselines. StraTA reaches success rates of 93.1% on ALFWorld and 84.2% on WebShop. On SciWorld, StraTA attains a 63.5% overall score, outperforming frontier closed-source models.
📬 Weekly AI Paper Digest
Get the top 10 AI/ML arXiv papers from the week — summarized, scored, and delivered to your inbox every Monday.