research
∙
07/05/2023
Improving Automatic Parallel Training via Balanced Memory Workload Optimization
Transformer models have emerged as the leading approach for achieving st...
share
research
∙
05/27/2023
FISEdit: Accelerating Text-to-image Editing via Cache-enabled Sparse Diffusion Inference
Due to the recent success of diffusion models, text-to-image generation ...
research
∙
05/16/2023
SpecInfer: Accelerating Generative LLM Serving with Speculative Inference and Token Tree Verification
The high computational and memory requirements of generative large langu...
research
∙
04/08/2023
FlexMoE: Scaling Large-scale Sparse Pre-trained Model Training via Dynamic Device Placement
With the increasing data volume, there is a trend of using large-scale p...
research
∙
03/06/2023
Angel-PTM: A Scalable and Economical Large-scale Pre-training System in Tencent
Recent years have witnessed the unprecedented achievements of large-scal...
research
∙
11/25/2022
Galvatron: Efficient Transformer Training over Multiple GPUs Using Automatic Parallelism
Transformer models have achieved state-of-the-art performance on various...
research
∙
11/01/2022
Distributed Graph Neural Network Training: A Survey
Graph neural networks (GNNs) are a type of deep learning models that lea...
research
∙
09/28/2022
CALIP: Zero-Shot Enhancement of CLIP with Parameter-free Attention
Contrastive Language-Image Pre-training (CLIP) has been shown to learn v...
research
∙
09/27/2022
OSDP: Optimal Sharded Data Parallel for Distributed Deep Learning
Large-scale deep learning models contribute to significant performance i...
research
∙
07/29/2022
Towards Communication-efficient Vertical Federated Learning Training via Cache-enabled Local Updates
Vertical federated learning (VFL) is an emerging paradigm that allows di...
research
∙
03/28/2022
HetuMoE: An Efficient Trillion-scale Mixture-of-Expert Distributed Training System
As giant dense models advance quality but require large-scale expensive ...
research
∙
12/29/2021
Dense-to-Sparse Gate for Mixture-of-Experts
Mixture-of-experts (MoE) is becoming popular due to its success in impro...
research
∙
12/14/2021
HET: Scaling out Huge Embedding Model Training via Cache-enabled Distributed Framework
Embedding models have been an effective learning paradigm for high-dimen...
research
∙
12/04/2021