SS3D: End2End Self-Supervised 3D from Web Videos
Marwane Hariat, Gianni Franchi, David Filliat, Antoine Manzanera
TLDR
SS3D is a self-supervised pipeline for end-to-end 3D estimation from monocular web videos, jointly predicting depth, ego-motion, and intrinsics.
Key contributions
- Introduces SS3D, an end-to-end self-supervised pipeline for 3D estimation from monocular web videos.
- Jointly predicts depth, ego-motion, and camera intrinsics in a single forward pass.
- Utilizes an intrinsics-first schedule and multi-view signal proxy (MVS) to handle unconstrained web video.
- Achieves strong cross-domain zero-shot transfer and improved fine-tuning after pretraining on YouTube-8M.
Why it matters
This paper introduces a robust method for self-supervised 3D reconstruction from diverse web videos, a challenging domain. By jointly predicting multiple 3D components and using novel stabilization techniques, it significantly advances monocular 3D estimation. Its strong transfer capabilities make it valuable for various downstream tasks.
Original Abstract
We present SS3D, a web-scale SfM-based self-supervision pretraining pipeline for feed-forward 3D estimation from monocular video. Our model jointly predicts depth, ego-motion, and intrinsics in a single forward pass and is trained/evaluated as a coherent end-to-end 3D estimator. To stabilize joint learning, we use an intrinsics-first two-stage schedule and a unified single-checkpoint evaluation protocol. Scaling SfM self-supervision to unconstrained web video is challenging due to weak multi-view observability and strong corpus heterogeneity; we address these with a multi-view signal proxy (MVS) used for filtering and curriculum sampling, and with expert training distilled into a single student. Pretraining on YouTube-8M (~100M frames after filtering) yields strong cross-domain zero-shot transfer and improved fine-tuning performance over prior self-supervised baselines. We release the pretrained checkpoint and code.
📬 Weekly AI Paper Digest
Get the top 10 AI/ML arXiv papers from the week — summarized, scored, and delivered to your inbox every Monday.