Lucid-XR: An Extended-Reality Data Engine for Robotic Manipulation
Yajvan Ravan, Adam Rashid, Alan Yu, Kai McClennen, Gio Huh + 6 more
TLDR
Lucid-XR is an XR-based generative data engine for training robotic systems, enabling zero-shot transfer to real-world manipulation tasks.
Key contributions
- Introduces Lucid-XR, a generative data engine creating diverse, realistic multi-modal data for robot training.
- Features `vuer`, a web-based physics simulation running on XR headsets for immersive, latency-free interactions.
- Integrates on-device physics simulation, human-to-robot pose retargeting, and NL-steerable video generation.
- Demonstrates zero-shot transfer of robot visual policies trained on synthetic data to unseen real-world environments.
Why it matters
Training robots requires vast amounts of diverse, realistic data, which is often costly and time-consuming to collect. Lucid-XR offers an innovative solution by leveraging XR technology to generate high-quality synthetic data efficiently. This enables robust robot training and impressive zero-shot transfer capabilities, accelerating progress in robotic manipulation.
Original Abstract
We introduce Lucid-XR, a generative data engine for creating diverse and realistic-looking multi-modal data to train real-world robotic systems. At the core of Lucid-XR is vuer, a web-based physics simulation environment that runs directly on the XR headset, enabling internet-scale access to immersive, latency-free virtual interactions without requiring specialized equipment. The complete system integrates on-device physics simulation with human-to-robot pose retargeting. Data collected is further amplified by a physics-guided video generation pipeline steerable via natural language specifications. We demonstrate zero-shot transfer of robot visual policies to unseen, cluttered, and badly lit evaluation environments, after training entirely on Lucid-XR's synthetic data. We include examples across dexterous manipulation tasks that involve soft materials, loosely bound particles, and rigid body contact. Project website: https://lucidxr.github.io
📬 Weekly AI Paper Digest
Get the top 10 AI/ML arXiv papers from the week — summarized, scored, and delivered to your inbox every Monday.