Model-Based Reinforcement Learning for Control under Time-Varying Dynamics
Klemens Iten, Bruce Lee, Chenhao Li, Lenart Treven, Andreas Krause + 1 more
TLDR
This paper introduces a model-based RL algorithm with adaptive data buffers to handle time-varying system dynamics, improving control performance.
Key contributions
- Analyzes continual model-based RL under time-varying dynamics using Gaussian processes.
- Identifies that limiting outdated data is crucial for calibrated uncertainty and regret in non-stationary settings.
- Proposes an optimistic model-based RL algorithm with adaptive data buffer mechanisms.
- Demonstrates improved performance on continuous control benchmarks with non-stationary dynamics.
Why it matters
Real-world control systems often face changing dynamics, which traditional RL struggles with. This paper provides a principled approach to address non-stationarity in model-based RL. Its insights on data management and the proposed algorithm offer a robust solution for practical applications.
Original Abstract
Learning-based control methods typically assume stationary system dynamics, an assumption often violated in real-world systems due to drift, wear, or changing operating conditions. We study reinforcement learning for control under time-varying dynamics. We consider a continual model-based reinforcement learning setting in which an agent repeatedly learns and controls a dynamical system whose transition dynamics evolve across episodes. We analyze the problem using Gaussian process dynamics models under frequentist variation-budget assumptions. Our analysis shows that persistent non-stationarity requires explicitly limiting the influence of outdated data to maintain calibrated uncertainty and meaningful dynamic regret guarantees. Motivated by these insights, we propose a practical optimistic model-based reinforcement learning algorithm with adaptive data buffer mechanisms and demonstrate improved performance on continuous control benchmarks with non-stationary dynamics.
📬 Weekly AI Paper Digest
Get the top 10 AI/ML arXiv papers from the week — summarized, scored, and delivered to your inbox every Monday.