Perceptual Flow Network for Visually Grounded Reasoning
Yangfu Li, Yuning Gong, Hongjian Zhan, Teng Li, Yuanhuiyi Lyu + 6 more
TLDR
PFlowNet enhances visually grounded reasoning in LVLMs by decoupling perception and using RL with geometric shaping, achieving new SOTA results.
Key contributions
- Decouples perception from reasoning in LVLMs for self-conditioned visual generation.
- Uses variational RL with multi-dimensional rewards and geometric shaping for reasoning-oriented perception.
- Mitigates language bias and hallucination in visually grounded reasoning tasks.
- Sets new SOTA records on V* Bench (90.6%) and MME-RealWorld-lite (67.0%).
Why it matters
LVLMs often suffer from language bias and hallucination in visual reasoning. PFlowNet introduces a novel approach by decoupling perception and using reinforcement learning to achieve more reliable and interpretable visual reasoning. This significantly improves the trustworthiness and performance of vision-language models, setting new benchmarks.
Original Abstract
Despite the success of Large-Vision Language Models (LVLMs), general optimization objectives (e.g., standard MLE) fail to constrain visual trajectories, leading to language bias and hallucination. To mitigate this, current methods introduce geometric priors from visual experts as additional supervision. However, we observe that such supervision is typically suboptimal: it is biased toward geometric precision and offers limited reasoning utility. To bridge this gap, we propose Perceptual Flow Network (PFlowNet), which eschews rigid alignment with the expert priors and achieves interpretable yet more effective visual reasoning. Specifically, PFlowNet decouples perception from reasoning to establish a self-conditioned generation process. Based on this, it integrates multi-dimensional rewards with vicinal geometric shaping via variational reinforcement learning, thereby facilitating reasoning-oriented perceptual behaviors while preserving visual reliability. PFlowNet delivers a provable performance guarantee and competitive empirical results, particularly setting new SOTA records on V* Bench (90.6%) and MME-RealWorld-lite (67.0%).
📬 Weekly AI Paper Digest
Get the top 10 AI/ML arXiv papers from the week — summarized, scored, and delivered to your inbox every Monday.