Squareplus: A Softplus-Like Algebraic Rectifier

12/22/2021

∙

by   Jonathan T. Barron, et al.

∙

0

∙

share

We present squareplus, an activation function that resembles softplus, but which can be computed using only algebraic operations: addition, multiplication, and square-root. Because squareplus is  6x faster to evaluate than softplus on a CPU and does not require access to transcendental functions, it may have practical value in resource-limited deep learning applications.