research
∙
09/18/2023
Multimodal Foundation Models: From Specialists to General-Purpose Assistants
This paper presents a comprehensive survey of the taxonomy and evolution...
share
research
∙
08/26/2023
ORES: Open-vocabulary Responsible Visual Synthesis
Avoiding synthesizing specific visual concepts is an essential challenge...
research
∙
07/27/2023
Spatial-Frequency U-Net for Denoising Diffusion Probabilistic Models
In this paper, we study the denoising diffusion probabilistic model (DDP...
research
∙
06/30/2023
DisCo: Disentangled Control for Referring Human Dance Generation in Real World
Generative AI has made significant strides in computer vision, particula...
research
∙
06/26/2023
Mitigating Hallucination in Large Multi-Modal Models via Robust Instruction Tuning
Despite the promising progress in multi-modal tasks, current large multi...
research
∙
06/07/2023
MultiSum: A Dataset for Multimodal Summarization and Thumbnail Generation of Videos
Multimodal summarization with multimodal output (MSMO) has emerged as a ...
research
∙
05/07/2023
Neural Voting Field for Camera-Space 3D Hand Pose Estimation
We present a unified framework for camera-space 3D hand pose estimation ...
research
∙
04/28/2023
An Empirical Study of Multimodal Model Merging
Model merging (e.g., via interpolation or task arithmetic) fuses multipl...
research
∙
04/16/2023
The Contour integral method for Feynman-Kac equation with two internal states
We develop the contour integral method for numerically solving the Feynm...
research
∙
04/13/2023
Diagnostic Benchmark and Iterative Inpainting for Layout-Guided Image Generation
Spatial control is a core capability in controllable image generation. A...
research
∙
04/12/2023
Adaptive Human Matting for Dynamic Videos
The most recent efforts in video matting have focused on eliminating tri...
research
∙
03/25/2023
Equivariant Similarity for Vision-Language Foundation Models
This study explores the concept of equivariance in vision-language found...
research
∙
03/20/2023
MM-REACT: Prompting ChatGPT for Multimodal Reasoning and Action
We propose MM-REACT, a system paradigm that integrates ChatGPT with a po...
research
∙
02/21/2023
Learning 3D Photography Videos via Self-supervised Diffusion on Single Images
3D photography renders a static image into a video with appealing 3D vis...
research
∙
12/21/2022
Generalized Decoding for Pixel, Image, and Language
We present X-Decoder, a generalized decoding model that can predict pixe...
research
∙
12/01/2022
GRiT: A Generative Region-to-text Transformer for Object Understanding
This paper presents a Generative RegIon-to-Text transformer, GRiT, for o...
research
∙
11/24/2022
MPT: Mesh Pre-Training with Transformers for Human Pose and Mesh Reconstruction
We present Mesh Pre-Training (MPT), a new pre-training framework that le...
research
∙
11/21/2022
Exploring Discrete Diffusion Models for Image Captioning
The image captioning task is typically realized by an auto-regressive me...
research
∙
10/17/2022
Non-Contrastive Learning Meets Language-Image Pre-Training
Contrastive language-image pre-training (CLIP) serves as a de-facto stan...
research
∙
10/17/2022
Vision-Language Pre-training: Basics, Recent Advances, and Future Trends
This paper surveys vision-language pre-training (VLP) methods for multim...
research
∙
10/17/2022
Prompting GPT-3 To Be Reliable
Large language models (LLMs) show impressive abilities via few-shot prom...
research
∙
09/04/2022
An Empirical Study of End-to-End Video-Language Transformers with Masked Visual Modeling
Masked visual modeling (MVM) has been recently proven effective for visu...
research
∙
07/20/2022
NUWA-Infinity: Autoregressive over Autoregressive Generation for Infinite Visual Synthesis
In this paper, we present NUWA-Infinity, a generative model for infinite...
research
∙
06/15/2022
Coarse-to-Fine Vision-Language Pre-training with Fusion in the Backbone
Vision-language (VL) pre-training has recently received considerable att...
research
∙
06/14/2022
LAVENDER: Unifying Video-Language Understanding as Masked Language Modeling
Unified vision-language frameworks have greatly advanced in recent years...
research
∙
06/12/2022
GLIPv2: Unifying Localization and Vision-Language Understanding
We present GLIPv2, a grounded VL understanding model, that serves both l...
research
∙
05/27/2022
GIT: A Generative Image-to-text Transformer for Vision and Language
In this paper, we design and train a Generative Image-to-text Transforme...
research
∙
05/03/2022
Cross-modal Representation Learning for Zero-shot Action Recognition
We present a cross-modal Transformer-based framework, which jointly enco...
research
∙
04/20/2022
K-LITE: Learning Transferable Visual Models with External Knowledge
Recent state-of-the-art computer vision systems are trained from natural...
research
∙
03/10/2022
The Overlooked Classifier in Human-Object Interaction Recognition
Human-Object Interaction (HOI) recognition is challenging due to two fac...
research
∙
12/13/2021
Decoupling Object Detection from Human-Object Interaction Recognition
We propose DEFR, a DEtection-FRee method to recognize Human-Object Inter...
research
∙
12/09/2021
Injecting Semantic Concepts into End-to-End Image Captioning
Tremendous progress has been made in recent years in developing better i...
research
∙
12/08/2021
MLP Architectures for Vision-and-Language Modeling: An Empirical Study
We initiate the first empirical study on the use of MLP architectures fo...
research
∙
12/07/2021
Grounded Language-Image Pre-training
This paper presents a grounded language-image pre-training (GLIP) model ...
research
∙
11/25/2021
SwinBERT: End-to-End Transformers with Sparse Attention for Video Captioning
The canonical approach to video captioning dictates a caption generation...
research
∙
11/24/2021
VIOLET : End-to-End Video-Language Transformers with Masked Visual-token Modeling
A great challenge in video-language (VidL) modeling lies in the disconne...
research
∙
11/24/2021
Scaling Up Vision-Language Pre-training for Image Captioning
In recent years, we have witnessed significant performance boost in the ...
research
∙
11/23/2021
Crossing the Format Boundary of Text and Boxes: Towards Unified Vision-Language Modeling
In this paper, we propose UNICORN, a vision-language (VL) model that uni...
research
∙
11/22/2021
Florence: A New Foundation Model for Computer Vision
Automated visual understanding of our diverse and open world demands com...
research
∙
11/19/2021
UFO: A UniFied TransfOrmer for Vision-Language Representation Learning
In this paper, we propose a single UniFied transfOrmer (UFO), which is c...
research
∙
11/03/2021
An Empirical Study of Training End-to-End Vision-and-Language Transformers
Vision-and-language (VL) pre-training has proven to be highly effective ...
research
∙
09/10/2021
An Empirical Study of GPT-3 for Few-Shot Knowledge-Based VQA
Knowledge-based visual question answering (VQA) involves answering quest...
research
∙
08/08/2021
OVIS: Open-Vocabulary Visual Instance Search via Visual-Semantic Aligned Representation Learning
We introduce the task of open-vocabulary visual instance search (OVIS). ...
research
∙
07/27/2021
Is Object Detection Necessary for Human-Object Interaction Recognition?
This paper revisits human-object interaction (HOI) recognition at image ...
research
∙
06/16/2021
End-to-End Semi-Supervised Object Detection with Soft Teacher
This paper presents an end-to-end semi-supervised object detection appro...
research
∙
06/08/2021
VALUE: A Multi-Task Benchmark for Video-and-Language Understanding Evaluation
Most existing video-and-language (VidL) research focuses on a single dat...
research
∙
04/05/2021
Compressing Visual-linguistic Model via Knowledge Distillation
Despite exciting progress in pre-training for visual-linguistic (VL) rep...
research
∙
04/01/2021
Mesh Graphormer
We present a graph-convolution-reinforced transformer, named Mesh Grapho...
research
∙
03/30/2021
DAP: Detection-Aware Pre-training with Weak Supervision
This paper presents a detection-aware pre-training (DAP) approach, which...
research
∙
03/22/2021