Orthogonal-Padé Activation Functions: Trainable Activation functions for smooth and faster convergence in deep networks

06/17/2021

∙

by   Koushik Biswas, et al.

∙

0

∙

share

We have proposed orthogonal-Padé activation functions, which are trainable activation functions and show that they have faster learning capability and improves the accuracy in standard deep learning datasets and models. Based on our experiments, we have found two best candidates out of six orthogonal-Padé activations, which we call safe Hermite-Pade (HP) activation functions, namely HP-1 and HP-2. When compared to ReLU, HP-1 and HP-2 has an increment in top-1 accuracy by 5.06 respectively in MobileNet V2 model on CIFAR100 dataset while on CIFAR10 dataset top-1 accuracy increases by 2.02 2.24 Efficientnet B0.