OmniShotCut: Holistic Relational Shot Boundary Detection with Shot-Query Transformer
Boyang Wang, Guangyi Xu, Zhipeng Tang, Jiahui Zhang, Zezhou Cheng
TLDR
OmniShotCut introduces a Transformer-based method for Shot Boundary Detection, addressing limitations with a relational prediction approach and a new synthetic benchmark.
Key contributions
- Proposes OmniShotCut, a shot query-based Transformer for holistic relational Shot Boundary Detection.
- Formulates SBD as structured relational prediction, jointly estimating shot ranges and relations.
- Develops a fully synthetic transition synthesis pipeline for precise, diverse SBD data.
- Introduces OmniShotCutBench, a modern, wide-domain benchmark for diagnostic SBD evaluation.
Why it matters
Existing SBD methods struggle with accuracy and interpretability due to noisy data and outdated benchmarks. OmniShotCut offers a novel relational prediction approach using a Transformer. It also provides a synthetic data pipeline and a new benchmark, significantly advancing the field.
Original Abstract
Shot Boundary Detection (SBD) aims to automatically identify shot changes and divide a video into coherent shots. While SBD was widely studied in the literature, existing state-of-the-art methods often produce non-interpretable boundaries on transitions, miss subtle yet harmful discontinuities, and rely on noisy, low-diversity annotations and outdated benchmarks. To alleviate these limitations, we propose OmniShotCut to formulate SBD as structured relational prediction, jointly estimating shot ranges with intra-shot relations and inter-shot relations, by a shot query-based dense video Transformer. To avoid imprecise manual labeling, we adopt a fully synthetic transition synthesis pipeline that automatically reproduces major transition families with precise boundaries and parameterized variants. We also introduce OmniShotCutBench, a modern wide-domain benchmark enabling holistic and diagnostic evaluation.
📬 Weekly AI Paper Digest
Get the top 10 AI/ML arXiv papers from the week — summarized, scored, and delivered to your inbox every Monday.