Linear regression and logistic regression are both methods for modeling relationships between variables. They are both used to build statistical models but perform different tasks. Linear regression is used to model linear relationships, while logistic regression is used to model binary outcomes (i.e. whether or not an event happened). In this blog post, we will discuss the differences between linear and logistic regression, as well as when to use each one. We will also provide examples so that you can understand how they work.
What is linear regression?
Linear regression is used to solve the regression problems. It is used to model linear relationships. This means that it can be used to predict a continuous outcome (i.e. numerical outcome) based on one or more independent variables. Linear regression is the most basic type of regression and is very common in statistical modeling. Linear regression models are of different forms such as simple linear regression and multiple linear regression.
A simple linear regression model is used to model linear relationships between a dependent variable and one independent variable. Multiple linear regression is used to model linear relationships between a dependent variable and two or more independent variables. The formula for a simple linear regression model is:
y = β0 + βx
In the formula, y is the dependent variable, x is the independent variable, β0 is the intercept and β is the slope. The picture below represents a simple linear regression model where salary is modeled using experience.
The formula for a multilinear regression model is:
y = β0 + β1×1+ β2×2+…+βnxn
In the formula, y is the dependent variable, x is the independent vector of independent variables, β0 is the intercept, and β1, β1, …, βn are coefficients for respective features such as x1, x2, …, xn. The features in the above regression model can also be a function of one or more features combined together.
The coefficients of best-fit linear regression models are learned using the least-squares method. The least-squares method is a mathematical procedure for finding the line of best fit for a set of data points. The cost function for linear regression is the sum of the squared residuals. The residual is the difference between the actual value and the predicted value. The gradient descent algorithm is used to find the line of best fit by minimizing the cost function.
Linear regression models are evaluated using R-squared and adjusted R-squared. R-squared represents the variation of the value of dependent variables which is explained by the linear regression model. The greater the value of R-squared, the better is the linear regression model. Adjusted R-squared is used when there are multiple independent variables in the linear regression model. It adjusts for the addition of variables and only increases if the new variable actually improves the model. Read further details in some of the following blogs:
- Linear regression model explained with real-world examples
- Linear regression hypothesis testing examples
Some of the real-world examples where linear regression models can be used are as follows:
- Predict the price of a house based on its size, number of bedrooms, number of bathrooms, etc.
- Predict the demand for a product based on advertising expenditure, price of the product, etc.
- Predict students’ grades based on hours spent studying, the difficulty level of the course, etc.
- Predict the stock price of a company based on its earnings per share, dividend per share, etc.
- Predict the number of taxi rides taken in a city based on weather conditions, time of the day, etc.
What is logistic regression?
Logistic regression is used to solve the classification problems. It is used to model binary outcomes (i.e. whether or not an event happened). This means that it can be used to predict whether or not an event will happen (i.e. yes/no, true/false) based on one or more independent variables. In addition, logistic regression can also be used to model multiclass classification problems. The logistic regression used for multi-class classification is also called multinomial logistic regression. Read further details on this blog – Logistic regression explained with Python example
The logistic regression algorithm represents a sigmoid or logistic function. This function takes a linear combination of input features and maps it to output between 0 and 1. The formula for the sigmoid function is:
g(z) = (e^z)/(e^z+1)
In the formula, z is the linear combination of input features. The output of the sigmoid function g(z) represents the probability that an event will happen. If the probability is greater than 0.50, then the event is classified as “yes” or “true”. If the probability is less than 0.50, then the event is classified as “no” or “false”. The picture below represents a logistic regression model based on the sigmoid function.
Logistic regression is a type of linear regression but with a different cost function and different coefficients. The coefficients of logistic regression are learned using the maximum likelihood method. The maximum likelihood method is a mathematical procedure for finding the set of parameters that maximizes the probability of observing the data points that were actually observed. The cost function for logistic regression is the negative log-likelihood. The log-likelihood is the log of the probability of observing the data points that were actually observed given the model. The gradient descent algorithm is used to find the line of best fit by minimizing the cost function. The coefficients of best-fit logistic regression models are learned using a gradient descent algorithm similar to linear regression models. But, the cost function for the logistic regression model is different from the linear regression cost function. The cost function for logistic regression penalizes incorrect predictions more heavily than linear regression.
Logistic regression models are evaluated using accuracy and the AUC-ROC curve. Accuracy represents the percentage of correctly predicted values (i.e. true positives + true negatives) out of all predicted values. In addition, other evaluation metrics such as precision, recall, and F-measure can also be used to evaluate the logistic regression model. The AUC-ROC curve is a graphical representation of how well the logistic regression model can discriminate between positive and negative outcomes. The greater the area under the curve, the better is the logistic regression model.
Some of the real-world examples where logistic regression models can be used are:
- Predict whether or not a customer will default on a loan
- Predict whether or not a patient will have a heart attack
- Predict whether or not an email is a spam
- Predict whether or not a student will pass/fail an exam
Linear and logistic regression are linear models that use different approaches to solving regression and classification problems respectively. Linear regression uses a linear approach, while logistic regression utilizes a sigmoid function in order to classify events as binary or multiclass outcomes (i.e., yes/no or true/false). The cost functions of linear and logistic regressions are different. The linear regression cost function is the sum of the squared errors, while the logistic regression cost function is the negative log-likelihood. In addition, linear regression coefficients are learned using the least-squares method, while logistic regression coefficients are learned using the maximum likelihood method. Linear and logistic regressions are evaluated using different evaluation metrics. Linear regressions are evaluated using RMSE and R^squared, while logistic regressions are evaluated using accuracy and AUC-ROC curve. Some examples of where linear and logistic regression can be used respectively are predicting continuous-valued outcomes like the sales price of a house or binary classification problems like spam detection in emails.
Check out my latest book titled as First Principles Thinking: Building winning products using first principles thinking
- Generative Modeling in Machine Learning: Examples - March 19, 2023
- Data Analytics Training Program (Beginners) - March 18, 2023
- Histogram Plots using Matplotlib & Pandas: Python - March 18, 2023
Leave a Reply