Task-Aware LLM Routing with Multi-Level Task-Profile-Guided Data Synthesis for Cold-Start Scenarios
Hui Liu, Bin Zou, Kecheng Chen, Jie Liu, Wenya Wang + 1 more
TLDR
This paper introduces a data synthesis framework and TRouter for effective LLM routing in cold-start scenarios, improving performance and cost efficiency.
Key contributions
- Proposes a multi-level task-profile-guided data synthesis framework for cold-start LLM routing.
- Constructs a hierarchical task taxonomy and generates diverse question-answer pairs to approximate query distribution.
- Introduces TRouter, a task-type-aware router modeling query-conditioned cost and performance.
- Enhances routing utility in both cold-start and in-domain settings through prior regularization.
Why it matters
LLM routing is crucial for managing performance and cost, but existing systems struggle with new tasks lacking training data (cold-start). This work provides a novel solution through data synthesis and a new router, significantly enhancing LLM routing utility in challenging cold-start environments. This improves efficiency and adaptability.
Original Abstract
Large language models (LLMs) exhibit substantial variability in performance and computational cost across tasks and queries, motivating routing systems that select models to meet user-specific cost-performance trade-offs. However, existing routers generalize poorly in cold-start scenarios where in-domain training data is unavailable. We address this limitation with a multi-level task-profile-guided data synthesis framework that constructs a hierarchical task taxonomy and produces diverse question-answer pairs to approximate the test-time query distribution. Building on this, we introduce TRouter, a task-type-aware router approach that models query-conditioned cost and performance via latent task-type variables, with prior regularization derived from the synthesized task taxonomy. This design enhances TRouter's routing utility under both cold-start and in-domain settings. Across multiple benchmarks, we show that our synthesis framework alleviates cold-start issues and that TRouter delivers effective LLM routing.
📬 Weekly AI Paper Digest
Get the top 10 AI/ML arXiv papers from the week — summarized, scored, and delivered to your inbox every Monday.