Uncertainty-Aware Predictive Safety Filters for Probabilistic Neural Network Dynamics
Bernd Frauenknecht, Lukas Kesper, Daniel Mayfrank, Henrik Hose, Sebastian Trimpe
TLDR
UPSi introduces an uncertainty-aware predictive safety filter using probabilistic neural networks for rigorous safety in deep reinforcement learning.
Key contributions
- Introduces UPSi, a predictive safety filter using probabilistic ensemble neural networks.
- Provides rigorous safety predictions by formulating future outcomes as reachable sets.
- Integrates an explicit certainty constraint to prevent model exploitation during exploration.
- Achieves substantial improvements in exploration safety on standard safe RL benchmarks.
Why it matters
This paper addresses the critical gap between scalable model-based RL and rigorous safety guarantees. UPSi enables the use of complex neural network dynamics in safety filters, leading to safer and more generalizable deep reinforcement learning systems. It paves the way for deploying RL in safety-critical applications.
Original Abstract
Predictive safety filters (PSFs) leverage model predictive control to enforce constraint satisfaction during deep reinforcement learning (RL) exploration, yet their reliance on first-principles models or Gaussian processes limits scalability and broader applicability. Meanwhile, model-based RL (MBRL) methods routinely employ probabilistic ensemble (PE) neural networks to capture complex, high-dimensional dynamics from data with minimal prior knowledge. However, existing attempts to integrate PEs into PSFs lack rigorous uncertainty quantification. We introduce the Uncertainty-Aware Predictive Safety Filter (UPSi), a PSF that provides rigorous safety predictions using PE dynamics models by formulating future outcomes as reachable sets. UPSi introduces an explicit certainty constraint that prevents model exploitation and integrates seamlessly into common MBRL frameworks. We evaluate UPSi within Dyna-style MBRL on standard safe RL benchmarks and report substantial improvements in exploration safety over prior neural network PSFs while maintaining performance on par with standard MBRL. UPSi bridges the gap between the scalability and generality of modern MBRL and the safety guarantees of predictive safety filters.
📬 Weekly AI Paper Digest
Get the top 10 AI/ML arXiv papers from the week — summarized, scored, and delivered to your inbox every Monday.