MiniRBT: A Two-stage Distilled Small Chinese Pre-trained Model

04/03/2023
by   Xin Yao, et al.
0

In natural language processing, pre-trained language models have become essential infrastructures. However, these models often suffer from issues such as large size, long inference time, and challenging deployment. Moreover, most mainstream pre-trained models focus on English, and there are insufficient studies on small Chinese pre-trained models. In this paper, we introduce MiniRBT, a small Chinese pre-trained model that aims to advance research in Chinese natural language processing. MiniRBT employs a narrow and deep student model and incorporates whole word masking and two-stage distillation during pre-training to make it well-suited for most downstream tasks. Our experiments on machine reading comprehension and text classification tasks reveal that MiniRBT achieves 94 speedup, demonstrating its effectiveness and efficiency.

READ FULL TEXT

Please sign up or login with your details

Forgot password? Click here to reset