Can AI Be a Good Peer Reviewer? A Survey of Peer Review Process, Evaluation, and the Future
Sihong Wu, Owen Jiang, Yilun Zhao, Tiansheng Hu, Yiling Ma + 3 more
TLDR
This survey explores how AI, particularly LLMs, can assist or automate the multi-stage peer review process, covering generation, post-review tasks, and evaluation.
Key contributions
- Synthesizes techniques for AI-driven peer review generation, including fine-tuning and agent-based systems.
- Covers AI methods for post-review tasks like rebuttals, meta-reviews, and revision alignment.
- Details various evaluation methods for AI in peer review, from human-centered to LLM-based approaches.
- Catalogs datasets, compares modeling choices, and discusses ethical concerns and future directions.
Why it matters
This survey offers a comprehensive overview of AI's role in the peer review process, from review generation to post-review tasks and evaluation. It provides practical guidance for researchers and developers aiming to build and integrate LLM systems into the academic publishing workflow.
Original Abstract
Peer review is a multi-stage process involving reviews, rebuttals, meta-reviews, final decisions, and subsequent manuscript revisions. Recent advances in large language models (LLMs) have motivated methods that assist or automate different stages of this pipeline. In this survey, we synthesize techniques for (i) peer review generation, including fine-tuning strategies, agent-based systems, RL-based methods, and emerging paradigms to enhance generation; (ii) after-review tasks including rebuttals, meta-review and revision aligned to reviews; and (iii) evaluation methods spanning human-centered, reference-based, LLM-based and aspect-oriented. We catalog datasets, compare modeling choices, and discuss limitations, ethical concerns, and future directions. The survey aims to provide practical guidance for building, evaluating, and integrating LLM systems across the full peer review workflow.
📬 Weekly AI Paper Digest
Get the top 10 AI/ML arXiv papers from the week — summarized, scored, and delivered to your inbox every Monday.