research
∙
08/03/2023
Tensor Programs IVb: Adaptive Optimization in the Infinite-Width Limit
Going beyond stochastic gradient descent (SGD), what new phenomena emerg...
share
research
∙
02/01/2023
Width and Depth Limits Commute in Residual Networks
We show that taking the width and depth to infinity in a deep neural net...
research
∙
03/07/2022
Tensor Programs V: Tuning Large Neural Networks via Zero-Shot Hyperparameter Transfer
Hyperparameter (HP) tuning in deep learning is an expensive process, pro...
research
∙
11/04/2021
CLUES: Few-Shot Learning Evaluation in Natural Language Understanding
Most recent progress in natural language understanding (NLU) has been dr...
research
∙
07/01/2021
Implicit Acceleration and Feature Learning in Infinitely Wide Neural Networks with Bottlenecks
We analyze the learning dynamics of infinitely wide neural networks with...
research
∙
05/08/2021
Tensor Programs IIb: Architectural Universality of Neural Tangent Kernel Training Dynamics
Yang (2020a) recently showed that the Neural Tangent Kernel (NTK) at ini...
research
∙
11/30/2020
Feature Learning in Infinite-Width Neural Networks
As its width tends to infinity, a deep neural network's behavior under g...
research
∙
09/22/2020
Tensor Programs III: Neural Matrix Laws
In a neural network (NN), weight matrices linearly transform inputs into...
research
∙
06/25/2020
Tensor Programs II: Neural Tangent Kernel for Any Architecture
We prove that a randomly initialized neural network of *any architecture...
research
∙
04/26/2020
Improved Image Wasserstein Attacks and Defenses
Robustness against image perturbations bounded by a ℓ_p ball have been w...
research
∙
03/04/2020
Black-box Smoothing: A Provable Defense for Pretrained Classifiers
We present a method for provably defending any pretrained image classifi...
research
∙
02/19/2020
Randomized Smoothing of All Shapes and Sizes
Randomized smoothing is a recently proposed defense against adversarial ...
research
∙
10/28/2019
Tensor Programs I: Wide Feedforward or Recurrent Neural Networks of Any Architecture are Gaussian Processes
Wide neural networks with random weights and biases are Gaussian process...
research
∙
09/05/2019
Free resolutions of function classes via order complexes
Function classes are collections of Boolean functions on a finite set, w...
research
∙
07/24/2019
A Fine-Grained Spectral Perspective on Neural Networks
Are neural networks biased toward simple functions? Does depth always he...
research
∙
06/09/2019
Provably Robust Deep Learning via Adversarially Trained Smoothed Classifiers
Recent works have shown the effectiveness of randomized smoothing as a s...
research
∙
02/23/2019
A Convex Relaxation Barrier to Tight Robustness Verification of Neural Networks
Verification of neural networks enables us to gauge their robustness aga...
research
∙
02/23/2019
A Convex Relaxation Barrier to Tight Robust Verification of Neural Networks
Verification of neural networks enables us to gauge their robustness aga...
research
∙
02/21/2019
A Mean Field Theory of Batch Normalization
We develop a mean field theory for batch normalization in fully-connecte...
research
∙
02/13/2019
Scaling Limits of Wide Neural Networks with Weight Sharing: Gaussian Process Behavior, Gradient Independence, and Neural Tangent Kernel Derivation
Several recent trends in machine learning theory and practice, from the ...
research
∙
02/12/2019
NAIL: A General Interactive Fiction Agent
Interactive Fiction (IF) games are complex textual decision making probl...
research
∙
01/25/2019
Dynamical Isometry and a Mean Field Theory of LSTMs and GRUs
Training recurrent neural networks (RNNs) on long sequence tasks is plag...
research
∙
12/24/2017
Mean Field Residual Networks: On the Edge of Chaos
We study randomly initialized residual networks using mean field theory ...
research
∙
11/09/2016
Lie-Access Neural Turing Machines
External neural memory structures have recently become a popular tool fo...
research
∙
02/28/2016