A foundation model of vision, audition, and language for in-silico neuroscience
Stéphane d'Ascoli, Jérémy Rapin, Yohann Benchetrit, Teon Brooks, Katelyn Begany + 3 more
TLDR
TRIBE v2 is a tri-modal AI foundation model that accurately predicts human brain activity across vision, audition, and language, enabling in-silico neuroscience.
Key contributions
- Introduces TRIBE v2, a tri-modal (vision, audio, language) foundation model for brain activity.
- Predicts high-resolution fMRI responses with several-fold accuracy improvement over prior models.
- Enables in-silico experiments, replicating established neuroscience findings across paradigms.
- Reveals fine-grained topography of multisensory integration using interpretable latent features.
Why it matters
This paper introduces a unified AI framework for understanding the human brain, overcoming the fragmentation of specialized models. TRIBE v2's ability to accurately predict brain activity and enable in-silico experiments could revolutionize cognitive neuroscience research.
Original Abstract
Cognitive neuroscience is fragmented into specialized models, each tailored to specific experimental paradigms, hence preventing a unified model of cognition in the human brain. Here, we introduce TRIBE v2, a tri-modal (video, audio and language) foundation model capable of predicting human brain activity in a variety of naturalistic and experimental conditions. Leveraging a unified dataset of over 1,000 hours of fMRI across 720 subjects, we demonstrate that our model accurately predicts high-resolution brain responses for novel stimuli, tasks and subjects, superseding traditional linear encoding models, delivering several-fold improvements in accuracy. Critically, TRIBE v2 enables in silico experimentation: tested on seminal visual and neuro-linguistic paradigms, it recovers a variety of results established by decades of empirical research. Finally, by extracting interpretable latent features, TRIBE v2 reveals the fine-grained topography of multisensory integration. These results establish artificial intelligence as a unifying framework for exploring the functional organization of the human brain.
📬 Weekly AI Paper Digest
Get the top 10 AI/ML arXiv papers from the week — summarized, scored, and delivered to your inbox every Monday.