research
∙
06/17/2022
The State of Sparse Training in Deep Reinforcement Learning
The use of sparse neural networks has seen rapid growth in recent years,...
share
research
∙
03/29/2022
Training Compute-Optimal Large Language Models
We investigate the optimal model size and number of tokens for training ...
research
∙
02/02/2022
Unified Scaling Laws for Routed Language Models
The performance of a language model has been shown to be effectively mod...
research
∙
12/13/2021
Step-unrolled Denoising Autoencoders for Text Generation
In this paper we propose a new generative model of text, Step-unrolled D...
research
∙
12/08/2021
Improving language models by retrieving from trillions of tokens
We enhance auto-regressive language models by conditioning on document c...
research
∙
06/07/2021
Top-KAST: Top-K Always Sparse Training
Sparse neural networks are becoming increasingly important as the field ...
research
∙
06/26/2020
On the Generalization Benefit of Noise in Stochastic Gradient Descent
It has long been argued that minibatch stochastic gradient descent can g...
research
∙
06/18/2020
Sparse GPU Kernels for Deep Learning
Scientific workloads have traditionally exploited high levels of sparsit...
research
∙
06/12/2020
AlgebraNets
Neural networks have historically been built layerwise from the set of f...
research
∙
06/12/2020
A Practical Sparse Approximation for Real Time Recurrent Learning
Current methods for training recurrent neural networks are based on back...
research
∙
06/05/2020
End-to-End Adversarial Text-to-Speech
Modern text-to-speech synthesis pipelines typically involve multiple pro...
research
∙
11/25/2019
Rigging the Lottery: Making All Tickets Winners
Sparse neural networks have been shown to be more parameter and compute ...
research
∙
11/21/2019
Fast Sparse ConvNets
Historically, the pursuit of efficient inference has been one of the dri...
research
∙
09/25/2019
High Fidelity Speech Synthesis with Adversarial Networks
Generative adversarial networks have seen rapid development in recent ye...
research
∙
06/25/2019
The Difficulty of Training Sparse Neural Networks
We investigate the difficulties of training sparse neural networks and m...
research
∙
06/07/2019
Non-Differentiable Supervised Learning with Evolution Strategies and Hybrid Methods
In this work we show that Evolution Strategies (ES) are a viable method ...
research
∙
02/25/2019
The State of Sparsity in Deep Neural Networks
We rigorously evaluate three state-of-the-art techniques for inducing sp...
research
∙
10/29/2018
Enabling Factorized Piano Music Modeling and Generation with the MAESTRO Dataset
Generating musical audio directly with neural networks is notoriously di...
research
∙
02/23/2018
Efficient Neural Audio Synthesis
Sequential models achieve state-of-the-art results in audio, visual and ...
research
∙
11/28/2017
Parallel WaveNet: Fast High-Fidelity Speech Synthesis
The recently-developed WaveNet architecture is the current state of the ...
research
∙
10/30/2017
Onsets and Frames: Dual-Objective Piano Transcription
We consider the problem of transcribing polyphonic piano music with an e...
research
∙
10/10/2017
Mixed Precision Training
Deep neural networks have enabled progress in a wide variety of applicat...
research
∙
04/17/2017
Exploring Sparsity in Recurrent Neural Networks
Recurrent Neural Networks (RNN) are widely used to solve a variety of pr...
research
∙
07/15/2016
DSD: Dense-Sparse-Dense Training for Deep Neural Networks
Modern deep neural networks have a large number of parameters, making th...
research
∙
12/08/2015
Deep Speech 2: End-to-End Speech Recognition in English and Mandarin
We show that an end-to-end deep learning approach can be used to recogni...
research
∙
12/17/2014