research
∙
10/19/2022
Scaling Laws for Reward Model Overoptimization
In reinforcement learning from human feedback, it is common to optimize ...
share
research
∙
10/12/2022
EleutherAI: Going Beyond "Open Science" to "Science in the Open"
Over the past two years, EleutherAI has established itself as a radicall...
research
∙
04/14/2022
GPT-NeoX-20B: An Open-Source Autoregressive Language Model
We introduce GPT-NeoX-20B, a 20 billion parameter autoregressive languag...
research
∙
01/13/2022
Datasheet for the Pile
This datasheet describes the Pile, a 825 GiB dataset of human-authored t...
research
∙
10/15/2021
Multitask Prompted Training Enables Zero-Shot Task Generalization
Large language models have recently been shown to attain reasonable zero...
research
∙
09/02/2021
An Empirical Exploration in Quality Filtering of Text Data
While conventional wisdom suggests that more aggressively filtering data...
research
∙
12/31/2020
The Pile: An 800GB Dataset of Diverse Text for Language Modeling
Recent work has demonstrated that increased training dataset diversity i...
research
∙
11/20/2020