Different Types of Activation Functions using Animation

In this post, you will be seeing different types of activation functions used in neural networks in form of an animation. If you are starting on deep learning and wanted to know about different types of activation functions, you may want to bookmark this page for quicker access in future.

Without further ado, let’s take a look at the animation which represents different types of activation functions:

Fig 1. Animation of Activation Functions used in Neural Networks (Courtesy: theffork.com)

Here is the list of different types of activation functions shown in above animation:

  • Identity function (Used in Adaline – Adaptive Linear Neuron)
  • Sigmoid function
  • Tanh functon
  • ArcTan function (inverse tangent function)
  • ReLU (Rectified Linear Unit)
  • Leaky ReLU (Improved version of ReLU)
  • Randomized ReLU
  • Parametric ReLU
  • Binary (Perceptron)
  • Exponential linear unit
  • Soft Sign
  • Inverse Square Root Unit (ISRU)
  • Inverse Square Root Linear
  • Square Non-linearity
  • Bipolar ReLU
  • Soft Plus

The following represents different variants of ReLU:

  • Leaky ReLU
  • Randomized ReLU
  • Parametric ReLU
  • Exponential linear unit
  • Bipolar ReLU

Out of the above activation functions, the most commonly / popularly used are the following:

  • Sigmoid
  • Tanh
  • ReLU and its different variants
Ajitesh Kumar
Follow me

Ajitesh Kumar

I have been recently working in the area of Data Science and Machine Learning / Deep Learning. In addition, I am also passionate about various different technologies including programming languages such as Java/JEE, Javascript, Python, R, Julia etc and technologies such as Blockchain, mobile computing, cloud-native technologies, application security, cloud computing platforms, big data etc. I would love to connect with you on Linkedin.
Posted in Deep Learning. Tagged with , , .

Leave a Reply

Your email address will not be published. Required fields are marked *

Time limit is exhausted. Please reload the CAPTCHA.