Adaptivity Under Realizability Constraints: Comparing In-Context and Agentic Learning
Anastasis Kratsios, A. Martina Neuman, Philipp Petersen
TLDR
This paper shows how representational constraints, like ReLU networks, profoundly alter the benefits of adaptive learning over fixed-query methods.
Key contributions
- Compares in-context (fixed queries) and agentic (adaptive queries) learning for task approximation.
- Examines adaptivity's performance under both unrestricted and ReLU neural network realizability constraints.
- Finds adaptivity's advantage can change dramatically when moving from unrestricted to realizable settings.
- Identifies four distinct scenarios where adaptivity's benefit varies with representational constraints.
Why it matters
This research highlights that adaptivity's benefits are not universal, changing significantly under representational constraints. Understanding this interaction is crucial for designing more effective and realistic AI learning systems.
Original Abstract
We compare in-context learning with fixed queries and agentic learning with adaptive queries for uniform approximation of task families. We consider two settings: an unrestricted regime, where querying and approximation are arbitrary functions, and a realizable regime, where we require these operations to be implemented by ReLU neural networks. In both settings, adaptivity never hinders approximation performance. However, this advantage can change when one passes from the unrestricted regime to the realizable regime. We identify four distinct approximation scenarios, each witnessed by an explicit task family: (a) no advantage of adaptivity; (b) an advantage in the unrestricted regime that persists under ReLU realizability; (c) an advantage that arises only under realizability; and (d) an advantage that disappears under realizability. This demonstrates that representational constraints interact profoundly with the effect of adaptivity.
📬 Weekly AI Paper Digest
Get the top 10 AI/ML arXiv papers from the week — summarized, scored, and delivered to your inbox every Monday.