Language Representation in Multilingual BERT and its applications to improve Cross-lingual Generalization

10/20/2020
by   Chi-Liang Liu, et al.
21

A token embedding in multilingual BERT (m-BERT) contains both language and semantic information. We find that representation of a language can be obtained by simply averaging the embeddings of the tokens of the language. With the language representation, we can control the output languages of multilingual BERT by manipulating the token embeddings and achieve unsupervised token translation. We further propose a computationally cheap but effective approach to improve the cross-lingual ability of m-BERT based on the observation.

READ FULL TEXT

Please sign up or login with your details

Forgot password? Click here to reset