Seeing Without Eyes: 4D Human-Scene Understanding from Wearable IMUs
Hao-Yu Hsu, Tianhang Cheng, Jing Wen, Alexander G. Schwing, Shenlong Wang
TLDR
This paper introduces IMU-to-4D, a framework that reconstructs 4D human motion and scene layouts using only wearable IMU sensors, bypassing cameras.
Key contributions
- Introduces IMU-to-4D, a novel framework for 4D human-scene understanding without cameras.
- Leverages wearable IMU sensors (earbuds, watches, smartphones) for spatiotemporal perception.
- Repurposes large language models for non-visual reconstruction of human motion and scene structure.
- Achieves more coherent and stable 4D human-scene understanding than SoTA cascaded pipelines.
Why it matters
This paper addresses critical limitations of camera-based perception by enabling robust 4D human-scene understanding using only privacy-preserving wearable IMUs. It opens new avenues for pervasive, energy-efficient, and scalable context-aware AI.
Original Abstract
Understanding human activities and their surrounding environments typically relies on visual perception, yet cameras pose persistent challenges in privacy, safety, energy efficiency, and scalability. We explore an alternative: 4D perception without vision. Its goal is to reconstruct human motion and 3D scene layouts purely from everyday wearable sensors. For this we introduce IMU-to-4D, a framework that repurposes large language models for non-visual spatiotemporal understanding of human-scene dynamics. IMU-to-4D uses data from a few inertial sensors from earbuds, watches, or smartphones and predicts detailed 4D human motion together with coarse scene structure. Experiments across diverse human-scene datasets show that IMU-to-4D yields more coherent and temporally stable results than SoTA cascaded pipelines, suggesting wearable motion sensors alone can support rich 4D understanding.
📬 Weekly AI Paper Digest
Get the top 10 AI/ML arXiv papers from the week — summarized, scored, and delivered to your inbox every Monday.