WebGPT: Browser-assisted question-answering with human feedback
Reiichiro Nakano, Jacob Hilton, Suchir Balaji, Jeff Wu, Long Ouyang + 13 more
TLDR
WebGPT fine-tunes GPT-3 to answer complex questions by browsing the web and using human feedback to improve factual accuracy and answer quality.
Key contributions
- Introduces a text-based web-browsing environment enabling GPT-3 to search and navigate the internet for information.
- Trains models using imitation learning from human demonstrations and optimizes with human preference feedback.
- Achieves answers preferred over human demonstrators 56% of the time and surpasses top Reddit answers 69% of the time.
Why it matters
This paper advances the capability of language models to generate more accurate, evidence-backed long-form answers by integrating web browsing and human feedback, addressing challenges in factual correctness and interpretability. It demonstrates a scalable approach to improving AI question-answering systems, making them more reliable and useful for real-world applications.
Original Abstract
We fine-tune GPT-3 to answer long-form questions using a text-based web-browsing environment, which allows the model to search and navigate the web. By setting up the task so that it can be performed by humans, we are able to train models on the task using imitation learning, and then optimize answer quality with human feedback. To make human evaluation of factual accuracy easier, models must collect references while browsing in support of their answers. We train and evaluate our models on ELI5, a dataset of questions asked by Reddit users. Our best model is obtained by fine-tuning GPT-3 using behavior cloning, and then performing rejection sampling against a reward model trained to predict human preferences. This model's answers are preferred by humans 56% of the time to those of our human demonstrators, and 69% of the time to the highest-voted answer from Reddit.
📬 Weekly AI Paper Digest
Get the top 10 AI/ML arXiv papers from the week — summarized, scored, and delivered to your inbox every Monday.