GPT-NeoX-20B: An Open-Source Autoregressive Language Model
Sid Black, Stella Biderman, Eric Hallahan, Quentin Anthony, Leo Gao + 12 more
TLDR
GPT-NeoX-20B is a 20 billion parameter open-source autoregressive language model that demonstrates strong few-shot reasoning abilities and outperforms comparable models in multi-shot settings.
Key contributions
- Introduces GPT-NeoX-20B, the largest publicly available dense autoregressive language model at 20B parameters.
- Demonstrates superior few-shot and five-shot performance compared to similarly sized GPT-3 and FairSeq models.
- Provides fully open-source model weights, training, and evaluation code under a permissive license.
Why it matters
This paper matters because it democratizes access to large-scale language models by releasing one of the biggest and most capable autoregressive models openly, enabling researchers and developers to build upon state-of-the-art technology without restrictive licenses. Its strong few-shot reasoning performance highlights advancements in model capabilities, fostering further innovation in natural language understanding and generation.
Original Abstract
We introduce GPT-NeoX-20B, a 20 billion parameter autoregressive language model trained on the Pile, whose weights will be made freely and openly available to the public through a permissive license. It is, to the best of our knowledge, the largest dense autoregressive model that has publicly available weights at the time of submission. In this work, we describe \model{}'s architecture and training and evaluate its performance on a range of language-understanding, mathematics, and knowledge-based tasks. We find that GPT-NeoX-20B is a particularly powerful few-shot reasoner and gains far more in performance when evaluated five-shot than similarly sized GPT-3 and FairSeq models. We open-source the training and evaluation code, as well as the model weights, at https://github.com/EleutherAI/gpt-neox.
📬 Weekly AI Paper Digest
Get the top 10 AI/ML arXiv papers from the week — summarized, scored, and delivered to your inbox every Monday.