Pi-HOC: Pairwise 3D Human-Object Contact Estimation
Sravan Chittupalli, Ayush Jain, Dong Huang
TLDR
Pi-HOC is a new framework for dense 3D semantic contact prediction between all human-object pairs, outperforming SOTA in accuracy and speed.
Key contributions
- Introduces Pi-HOC, a single-pass, instance-aware framework for dense 3D semantic contact prediction.
- Uses dedicated human-object tokens and an InteractionFormer for refining pairwise contact.
- Achieves significant accuracy and localization improvements with 20x higher throughput over SOTA.
- Enhances 3D image-to-mesh reconstruction and enables language-based referential contact prediction.
Why it matters
This paper addresses the critical challenge of disentangling fine-grained human-object contact in complex multi-human scenes. Pi-HOC offers a robust and efficient solution, significantly advancing 3D interaction understanding. Its high throughput and versatility open new avenues for applications in robotics and AR/VR.
Original Abstract
Resolving real-world human-object interactions in images is a many-to-many challenge, in which disentangling fine-grained concurrent physical contact is particularly difficult. Existing semantic contact estimation methods are either limited to single-human settings or require object geometries (e.g., meshes) in addition to the input image. Current state-of-the-art leverages powerful VLM for category-level semantics but struggles with multi-human scenarios and scales poorly in inference. We introduce Pi-HOC, a single-pass, instance-aware framework for dense 3D semantic contact prediction of all human-object pairs. Pi-HOC detects instances, creates dedicated human-object (HO) tokens for each pair, and refines them using an InteractionFormer. A SAM-based decoder then predicts dense contact on SMPL human meshes for each human-object pair. On the MMHOI and DAMON datasets, Pi-HOC significantly improves accuracy and localization over state-of-the-art methods while achieving 20x higher throughput. We further demonstrate that predicted contacts improve SAM-3D image-to-mesh reconstruction via a test-time optimization algorithm and enable referential contact prediction from language queries without additional training.
📬 Weekly AI Paper Digest
Get the top 10 AI/ML arXiv papers from the week — summarized, scored, and delivered to your inbox every Monday.