Teaching the Pre-trained Model to Generate Simple Texts for Text Simplification

05/21/2023
by   Renliang Sun, et al.
0

Randomly masking text spans in ordinary texts in the pre-training stage hardly allows models to acquire the ability to generate simple texts. It can hurt the performance of pre-trained models on text simplification tasks. In this paper, we propose a new continued pre-training strategy to teach the pre-trained model to generate simple texts. We continue pre-training BART, a representative model, to obtain SimpleBART. It consistently and significantly improves the results on lexical simplification, sentence simplification, and document-level simplification tasks over BART. At the end, we compare SimpleBART with several representative large language models (LLMs).

READ FULL TEXT

Please sign up or login with your details

Forgot password? Click here to reset