research
∙
09/20/2023
Kosmos-2.5: A Multimodal Literate Model
We present Kosmos-2.5, a multimodal literate model for machine reading o...
share
research
∙
09/11/2023
Large Language Model for Science: A Study on P vs. NP
In this work, we use large language models (LLMs) to augment and acceler...
research
∙
08/08/2023
Evil Operation: Breaking Speaker Recognition with PaddingBack
Machine Learning as a Service (MLaaS) has gained popularity due to advan...
research
∙
07/17/2023
Retentive Network: A Successor to Transformer for Large Language Models
In this work, we propose Retentive Network (RetNet) as a foundation arch...
research
∙
07/05/2023
LongNet: Scaling Transformers to 1,000,000,000 Tokens
Scaling sequence length has become a critical demand in the era of large...
research
∙
12/20/2022
Why Can GPT Learn In-Context? Language Models Secretly Perform Gradient Descent as Meta-Optimizers
Large pretrained language models have shown surprising In-Context Learni...
research
∙
12/20/2022
A Length-Extrapolatable Transformer
Position modeling plays a critical role in Transformers. In this paper, ...
research
∙
12/19/2022
Optimizing Prompts for Text-to-Image Generation
Well-designed prompts can guide text-to-image models to generate amazing...
research
∙
12/01/2022
Extensible Prompts for Language Models
We propose eXtensible Prompt (X-Prompt) for prompting a large language m...
research
∙
11/23/2022
TorchScale: Transformers at Scale
Large Transformers have achieved state-of-the-art performance across man...
research
∙
08/22/2022
Image as a Foreign Language: BEiT Pretraining for All Vision and Vision-Language Tasks
A big convergence of language, vision, and multimodal pretraining is eme...
research
∙
08/12/2022
BEiT v2: Masked Image Modeling with Vector-Quantized Visual Tokenizers
Masked image modeling (MIM) has demonstrated impressive results in self-...
research
∙
06/13/2022
Language Models are General-Purpose Interfaces
Foundation models have received much attention due to their effectivenes...
research
∙
05/20/2022
Visually-Augmented Language Modeling
Human language is grounded on multimodal knowledge including visual know...
research
∙
01/15/2022
Kformer: Knowledge Injection in Transformer Feed-Forward Layers
Knowledge-Enhanced Model have developed a diverse set of techniques for ...
research
∙
11/18/2021
Swin Transformer V2: Scaling Up Capacity and Resolution
We present techniques for scaling Swin Transformer up to 3 billion param...
research
∙
06/15/2021
BEiT: BERT Pre-Training of Image Transformers
We introduce a self-supervised vision representation model BEiT, which s...
research
∙
04/18/2021
Knowledge Neurons in Pretrained Transformers
Large-scale pretrained language models are surprisingly good at recallin...
research
∙
04/18/2021
mT6: Multilingual Pretrained Text-to-Text Transformer with Translation Pairs
Multilingual T5 (mT5) pretrains a sequence-to-sequence model on massive ...
research
∙
02/28/2020
UniLMv2: Pseudo-Masked Language Models for Unified Language Model Pre-Training
We propose to pre-train a unified language model for both autoencoding a...
research
∙
02/25/2020
MiniLM: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained Transformers
Pre-trained language models (e.g., BERT (Devlin et al., 2018) and its va...
research
∙
05/08/2019
