Deep Network Approximation: Beyond ReLU to Diverse Activation Functions

07/13/2023
βˆ™
by   Shijun Zhang, et al.
βˆ™
0
βˆ™

This paper explores the expressive power of deep neural networks for a diverse range of activation functions. An activation function set π’œ is defined to encompass the majority of commonly used activation functions, such as πšπšŽπ™»πš„, π™»πšŽπšŠπš”πš’πšπšŽπ™»πš„, πšπšŽπ™»πš„^2, π™΄π™»πš„, πš‚π™΄π™»πš„, πš‚πš˜πšπšπš™πš•πšžπšœ, π™Άπ™΄π™»πš„, πš‚πš’π™»πš„, πš‚πš πš’πšœπš‘, π™Όπš’πšœπš‘, πš‚πš’πšπš–πš˜πš’πš, πšƒπšŠπš—πš‘, π™°πš›πšŒπšπšŠπš—, πš‚πš˜πšπšπšœπš’πšπš—, πšπš‚πš’π™»πš„, and πš‚πšπš‚. We demonstrate that for any activation function Ο±βˆˆπ’œ, a πšπšŽπ™»πš„ network of width N and depth L can be approximated to arbitrary precision by a Ο±-activated network of width 6N and depth 2L on any bounded set. This finding enables the extension of most approximation results achieved with πšπšŽπ™»πš„ networks to a wide variety of other activation functions, at the cost of slightly larger constants.

READ FULL TEXT

Please sign up or login with your details

Forgot password? Click here to reset