Working Memory Constraints Scaffold Learning in Transformers under Data Scarcity
Pranava Madhyastha, Dagmar Adamcova
TLDR
Integrating human-like working memory constraints into Transformers, especially fixed-width attention, significantly improves grammatical accuracy with scarce data.
Key contributions
- Integrated human-like working memory constraints (fixed-width, temporal decay) into Transformers.
- Trained modified GPT-2 models from scratch on small, developmentally plausible datasets.
- Fixed-width attention significantly improves grammatical accuracy, especially with scarce data.
- Constrained models show stronger alignment with human reading time processing metrics.
Why it matters
This research shows that human-like working memory constraints act as a beneficial inductive bias for Transformers. They guide models towards more robust linguistic representations, particularly in data-scarce environments. This is crucial for developing efficient and human-aligned AI.
Original Abstract
We investigate the integration of human-like working memory constraints into the Transformer architecture and implement several cognitively inspired attention variants, including fixed-width windows based and temporal decay based attention mechanisms. Our modified GPT-2 models are trained from scratch on developmentally plausible datasets (10M and 100M words). Performance is evaluated on grammatical judgment tasks (BLiMP) and alignment with human reading time data. Our results indicate that these cognitively-inspired constraints, particularly fixed-width attention, can significantly improve grammatical accuracy especially when training data is scarce. These constrained models also tend to show a stronger alignment with human processing metrics. The findings suggest that such constraints may serve as a beneficial inductive bias, guiding models towards more robust linguistic representations, especially in data-limited settings.
📬 Weekly AI Paper Digest
Get the top 10 AI/ML arXiv papers from the week — summarized, scored, and delivered to your inbox every Monday.