GeoStack: A Framework for Quasi-Abelian Knowledge Composition in VLMs
Pranav Mantini, Shishir K. Shah
TLDR
GeoStack is a modular framework for VLMs that composes independently trained experts, mitigating catastrophic forgetting with constant-time inference.
Key contributions
- Introduces GeoStack, a modular framework for composing independently trained VLM domain experts.
- Preserves base model knowledge by imposing geometric and structural constraints on the adapter manifold.
- Achieves constant-time inference (O(1)) via a novel weight-folding property.
- Effectively mitigates catastrophic forgetting in multi-domain adaptation and incremental learning.
Why it matters
Catastrophic forgetting is a critical challenge in scaling VLMs. GeoStack provides an efficient and effective solution for long-term knowledge composition, allowing models to continuously learn and integrate new expertise across domains without performance degradation or increased inference cost.
Original Abstract
We address the challenge of knowledge composition in Vision-Language Models (VLMs), where accumulating expertise across multiple domains or tasks typically leads to catastrophic forgetting. We introduce GeoStack (Geometric Stacking), a modular framework that allows independently trained domain experts to be composed into a unified model. By imposing geometric and structural constraints on the adapter manifold, GeoStack ensures the foundational knowledge of the base model is preserved. Furthermore, we mathematically demonstrate a weight-folding property that achieves constant-time inference complexity ($O(1)$), regardless of the number of integrated experts. Experimental results across multi-domain adaptation and class-incremental learning show that GeoStack provides an efficient mechanism for long-term knowledge composition while significantly mitigating catastrophic forgetting. Code is available at https://github.com/QuantitativeImagingLaboratory/GeoStack.
📬 Weekly AI Paper Digest
Get the top 10 AI/ML arXiv papers from the week — summarized, scored, and delivered to your inbox every Monday.