Difference between Adaline and Logistic Regression

Logistic Sigmoid Activation Function Representation

In this post, you will understand the key differences between Adaline (Adaptive Linear Neuron) and Logistic Regression

  1. Activation function
  2. Cost function

Difference in Activation Function

The primary difference is the activation function. In Adaline, the activation function is called as linear activation function while in logistic regression, the activation function is called as sigmoid activation function. The diagram below represents the activation functions for Adaline. The  activation function for Adaline, also called as linear activation function, is the identity function which can be represented as the following:

$$\phi(W^TX) = W^TX$$

Adaline Linear Activation Function Representation

Fig 1. Adaline Linear Activation Function Representation

The diagram below represents the activation functions for Logistic Regression. The  activation function for Logistic Regression, also called as sigmoid activation function, is the identity function can be represented as the following:

$$\phi(W^TX) = \frac{1}{(1 + e^-Z)}$$ 

Logistic Sigmoid Activation Function Representation

Fig 2: Logistic Sigmoid Activation Function Representation


Difference in Cost Function

For Adaline, the cost function or loss function looks like below:

$$J(w) = \sum\limits_{i} \frac{1}{2} (\phi(z^{(i)}) – y^{(i)})^2$$

For Adaline, the goal is to minimize the above sum of squared error function.

For Logistic regression, the cost function is created based on likelihood function that looks like below:

$$L(w) = P(y \vert x; w) = \prod\limits_{i=1}^n P(y^{(i)} \vert x^{(i)}; w) = \prod\limits_{i=1}^n (\phi(z^{(i)}))^{y^{(i)}} (1 – \phi(z^{(i)}))^{(1-y^{(i)})} $$

For logistic regression, the idea is to maximize the above likelihood function. For ease of calculation and numerical stability the above equation is converted into log-likelihood function which is then maximized. The below represents the log-likelihood function:

$$\log(L(w)) = \sum\limits_{i=1}^n [y^{(i)}\log(\phi(z^{(i)})) + (i – y^{(i)})\log(1 – \phi(z^{(i)}))]$$

The above log-likelihood function could be written in the following manner as the cost functionJ(w) that can be minimized using the gradient descent

$$J(w) = \sum\limits_{i=1}^n [-y^{(i)}\log(\phi(z^{(i)})) – (i – y^{(i)})\log(1 – \phi(z^{(i)}))]$$

Note that the superscript represents the ith row. Underscript represents the specific feature in that row. 

Ajitesh Kumar

Ajitesh Kumar

I have been recently working in the area of Data analytics including Data Science and Machine Learning / Deep Learning. I am also passionate about different technologies including programming languages such as Java/JEE, Javascript, Python, R, Julia, etc, and technologies such as Blockchain, mobile computing, cloud-native technologies, application security, cloud computing platforms, big data, etc. I would love to connect with you on Linkedin. Check out my latest book titled as First Principles Thinking: Building winning products using first principles thinking.
Posted in AI, Data Science, Machine Learning. Tagged with , .