Large Language Models: A Survey
Shervin Minaee, Tomas Mikolov, Narjes Nikzad, Meysam Chenaghlu, Richard Socher + 2 more
TLDR
This survey comprehensively reviews the development, capabilities, and evaluation of large language models (LLMs) like GPT, LLaMA, and PaLM, highlighting their strengths, limitations, and future research directions.
Key contributions
- Detailed overview of prominent LLM families, their architectures, and training methodologies.
- Summary of datasets, fine-tuning techniques, and evaluation metrics used for LLM development and benchmarking.
- Discussion of current challenges and potential future directions in LLM research.
Why it matters
As LLMs have rapidly transformed natural language processing by enabling powerful general-purpose language understanding and generation, this paper provides a crucial synthesis of the state-of-the-art models, training strategies, and evaluation practices. It serves as a foundational resource for researchers and practitioners aiming to understand the landscape, compare models, and identify open problems to advance the field.
Original Abstract
Large Language Models (LLMs) have drawn a lot of attention due to their strong performance on a wide range of natural language tasks, since the release of ChatGPT in November 2022. LLMs' ability of general-purpose language understanding and generation is acquired by training billions of model's parameters on massive amounts of text data, as predicted by scaling laws \cite{kaplan2020scaling,hoffmann2022training}. The research area of LLMs, while very recent, is evolving rapidly in many different ways. In this paper, we review some of the most prominent LLMs, including three popular LLM families (GPT, LLaMA, PaLM), and discuss their characteristics, contributions and limitations. We also give an overview of techniques developed to build, and augment LLMs. We then survey popular datasets prepared for LLM training, fine-tuning, and evaluation, review widely used LLM evaluation metrics, and compare the performance of several popular LLMs on a set of representative benchmarks. Finally, we conclude the paper by discussing open challenges and future research directions.
📬 Weekly AI Paper Digest
Get the top 10 AI/ML arXiv papers from the week — summarized, scored, and delivered to your inbox every Monday.