In this post, you will be seeing different types of **activation functions** used in **neural networks** in form of an animation. If you are starting on deep learning and wanted to know about different types of activation functions, you may want to bookmark this page for quicker access in future.

Without further ado, let’s take a look at the animation which represents different types of activation functions:

Here is the list of different types of activation functions shown in above animation:

- Identity function (Used in Adaline – Adaptive Linear Neuron)
- Sigmoid function
- Tanh functon
- ArcTan function (inverse tangent function)
- ReLU (Rectified Linear Unit)
- Leaky ReLU (Improved version of ReLU)
- Randomized ReLU
- Parametric ReLU
- Binary (Perceptron)
- Exponential linear unit
- Soft Sign
- Inverse Square Root Unit (ISRU)
- Inverse Square Root Linear
- Square Non-linearity
- Bipolar ReLU
- Soft Plus

The following represents different variants of ReLU:

- Leaky ReLU
- Randomized ReLU
- Parametric ReLU
- Exponential linear unit
- Bipolar ReLU

Out of the above activation functions, the most commonly / popularly used are the following:

- Sigmoid
- Tanh
- ReLU and its different variants

- First Principles Understanding based on Physics - April 13, 2021
- Precision & Recall Explained using Covid-19 Example - April 11, 2021
- Moving Average Method for Time-series forecasting - April 4, 2021

## Leave a Reply