Abstract
In deep learning, there are various parameters that helps to drive optimal results. One of those parameters is to use the correct activation function. The activation function must have ideal statistical characteristics. In this paper, a novel deep learning activation function has been proposed. Sigmoid activation function generally used in the output layer for bi-classification problem. Recently, swish activation used sigmoid function with hidden layers. Motivated by this, a new activation function is being proposed as (relu (x) + x * sigmoid (x)) to get the significance benefits of relu and sigmoid in a swish like flavour. The proposed function represents the desired statistical characteristics as unboundedness, monotonicity, zero centred and non-vanishing gradient. The experimental outcomes are also quite significant.
Keywords
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Glorot, X., Bengio, Y.: Understanding the difficulty of training deep feedforward neural networks. In: Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics, pp. 249–256 (2010)
Hahnloser, R.H.R., Sarpeshkar, R., Mahowald, M.A., Douglas, R.J., Seung, H.S.: Digital selection and analogue amplification coexist in a cortex-inspired silicon circuit. Nature 405(6789), 947 (2000)
Maas, A.L., Hannun, A.Y., Ng, A.Y.: Rectifier nonlinearities improve neural network acoustic models. In: International Conference on Machine Learning, vol. 30 (2013)
Clevert, D.-A., Unterthiner, T., Hochreiter, S.: Fast and accurate deep network learning by exponential linear units (elus). arXiv preprint arXiv:1511.07289 (2015)
Klambauer, G., Unterthiner, T., Mayr, A., Hochreiter, S.: Self-normalizing neural networks. arXiv preprint arXiv:1706.02515 (2017)
Schmidhuber, J.: Deep learning in neural networks: an overview. Neural Netw. 61, 85–117 (2015)
Nair, V., Hinton, G.E.: Rectified linear units improve restricted boltzmann machines. In: International Conference on Machine Learning (2010)
Sill, J.: Monotonic networks. In: Advances in Neural Information Processing Systems, pp. 661–667, Monotonic (1998)
Glorot, X., Bengio, Y.: Understanding the difficulty of training deep feedforward neural networks. In: Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics, pp. 249–256 (2010)
Hochreiter, S., Bengio, Y., Frasconi, P., Schmidhuber, J.: Gradient flow in recurrent nets: the difficulty of learning long-term dependencies. In: Kremer, S.C., Kolen, J.F. (eds.) A Field Guide to Dynamical Recurrent Neural Networks. IEEE Press (2001)
Ramachandran, P., Zoph, B., Le, Q.V.: Swish: a self-gated activation function. arXiv preprint arXiv:1710.05941, 7 (2017)
McCabe, T.J.: A complexity measures. IEEE Trans. softw. Eng. (4), 308–320 (1976)
Mhaskar, H.N., Micchelli, C.A.: How to choose an activation function. In: Advances in Neural Information Processing Systems, pp. 319–326 (1994)
Cybenko, G.: Approximation by superpositions of a sigmoidal function. Math. Control, Signals, and Systems (MCSS) 2(4) 303–314 (1989)
https://archive.ics.uci.edu/ml/datasets/reuters-21578+text+categorization+collection
Sharma, O.: A new activation function for deep neural network. In: 2019 International Conference on Machine Learning, Big Data, Cloud and Parallel Computing (COMITCon). IEEE, February 2019
Sharma, O.: Deep challenges associated with deep learning. In: 2019 International Conference on Machine Learning, Big Data, Cloud and Parallel Computing (COMITCon). IEEE, February 2019
https://ml-cheatsheet.readthedocs.io/en/latest/activation_functions.html
https://kasperfred.com/posts/computational-complexity-of-neural-networks
https://towardsdatascience.com/activation-functions-and-its-types-which-is-better-a9a5310cc8f
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2020 Springer Nature Singapore Pte Ltd.
About this paper
Cite this paper
Sharma, O. (2020). A Novel Activation Function in Convolutional Neural Network for Image Classification in Deep Learning. In: Batra, U., Roy, N., Panda, B. (eds) Data Science and Analytics. REDSET 2019. Communications in Computer and Information Science, vol 1229. Springer, Singapore. https://doi.org/10.1007/978-981-15-5827-6_10
Download citation
DOI: https://doi.org/10.1007/978-981-15-5827-6_10
Published:
Publisher Name: Springer, Singapore
Print ISBN: 978-981-15-5826-9
Online ISBN: 978-981-15-5827-6
eBook Packages: Computer ScienceComputer Science (R0)