KNN vs Logistic Regression: Differences, Examples

In this blog, we will learn about the differences between K-Nearest Neighbors (KNN) and Logistic Regression, two pivotal algorithms in machine learning, with the help of examples. The goal is to understand the intricacies of KNN’s instance-based learning and Logistic Regression‘s probability modeling for binary and multinomial outcomes, offering clarity on their core principles.

We will also navigate through the practical applications of K-NN and logistic regression algorithms, showcasing real-world examples in various business domains like healthcare and finance. Accompanying this, we’ll provide concise Python code samples, guiding you through implementing these algorithms with datasets. This dual focus on theory and practicality aims to equip you with both the understanding and tools necessary for applying KNN and Logistic Regression in your data science endeavors.

What’s KNN & Logistic Regression? How do they work?

K-Nearest Neighbors (K-NN) is a simple, versatile, and non-parametric machine learning algorithm used for both classification and regression tasks. It’s based on the principle of feature similarity. On the other hand, Logistic Regression is a statistical method used for binary and multinomial classification. It predicts the probability of occurrence of an event by fitting data to a logistic curve.

How KNN works?

The K-Nearest Neighbors algorithm operates on a simple concept of feature similarity. When a new data point is introduced, K-NN looks at the ‘k’ closest data points in the training set, known as ‘neighbors‘. The algorithm calculates the distance between data points using metrics like Euclidean or Manhattan distance.

In classification tasks, it assigns the new point to the most common class among these neighbors.

In regression tasks, K-NN predicts a value based on the average of the values of its nearest neighbors. The choice of ‘k’ is a critical factor in K-NN’s performance – too small a value for ‘k’ makes the algorithm sensitive to noise, while too large a value can lead to computational inefficiency and potentially lower accuracy.

How Logistic Regression works?

Logistic Regression uses the logistic function, also known as the sigmoid function, to transform linear combinations of input features into a probability format ranging between 0 and 1. This function takes any real-valued number and outputs a value between these two extremes, ideal for binary classification. The coefficients in Logistic Regression, akin to those in linear regression, represent the log odds of the outcome and are used to calculate the odds ratios for easier interpretation. Although primarily known for binary classification, Logistic Regression can be adapted for multiclass problems using techniques such as the one-vs-rest method. This extension allows the model to handle scenarios where more than two classes are present, enhancing its versatility.

Problems Solutions Examples for Logistic Regression & KNN

K-Nearest Neighbors (KNN) and Logistic Regression, while distinct in their methodologies, can both be applied to a variety of problem classes in machine learning. Some of the most common ones are the following:

1. Classification Problems:
   • KNN Example: Classifying images in a facial recognition system where each image is labeled with the person’s identity.
   • Logistic Regression Example: Diagnosing whether a patient has a certain disease (yes or no) based on their medical test results.
2. Pattern Recognition:
   • KNN Example: Identifying the genre of a song based on its acoustic features like tempo, pitch, and rhythm.
   • Logistic Regression Example: Detecting the presence of an object (like a stop sign) in digital images for autonomous vehicle navigation.
3. Recommendation Systems:
   • KNN Example: Suggesting similar products to online shoppers based on the shopping history of customers with similar profiles.
   • Logistic Regression Example: Recommending movies to users by predicting the likelihood of a user liking a movie based on their past ratings.
4. Anomaly Detection:
   • KNN Example: Detecting fraudulent credit card transactions by comparing a transaction to typical user behavior.
   • Logistic Regression Example: Identifying abnormal machine behavior in a manufacturing plant to preemptively address potential failures.
5. Regression problems:
   • K-NN Example: In a regression scenario like predicting housing prices, K-Nearest Neighbors (KNN) can be an effective tool. For instance, when estimating a house’s price, KNN would consider features such as size, number of bedrooms and bathrooms, age, and location. The algorithm calculates the distance between houses using these features, typically through Euclidean distance. It then identifies the ‘k’ closest houses to the one in question. The final price prediction is made by averaging the prices of these ‘k’ nearest houses. This approach is beneficial for capturing non-linear relationships between features and prices, though it requires careful selection of ‘k’ and relevant features, and can be computationally demanding for large datasets.
   • Logistic Regression Example: Well, logistic regression is not used for making predictions for continuous response variable.

In each of these classes, KNN and Logistic Regression can be leveraged effectively, although their suitability depends on the specific characteristics and requirements of the problem at hand. KNN is often chosen for its simplicity and effectiveness in capturing non-linear relationships, while Logistic Regression is preferred for its efficiency and interpretability, especially when the relationship between the predictors and the response is linear or logistic in nature.

When to use K-NN vs Logistic Regression?

Choosing between K-Nearest Neighbors (KNN) and Logistic Regression depends on various factors related to the nature of the data and the specific requirements of the problem. Here are some considerations to help decide when to use each algorithm:

When to Use K-NN?

1. Non-Linear Relationships: KNN can be a better choice when the relationship between features and the target variable is complex and non-linear.
2. Small to Medium Datasets: KNN works well with smaller datasets but can become computationally expensive as the size of the data grows.
3. No Prior Knowledge Needed: Since KNN is a non-parametric method, it doesn’t make any assumptions about the underlying data distribution, making it suitable when you have little to no prior knowledge about the data patterns.
4. Feature Engineering Importance: KNN can perform better if you have well-engineered features and the dataset doesn’t have many irrelevant features.
5. Real-time Decision Making: KNN can be useful in scenarios requiring real-time decisions since it doesn’t require a training phase. However, the prediction phase can be slow with large datasets.

When to Use Logistic Regression?

1. Binary or Multiclass Classification Problems: Logistic Regression is ideal for binary classification problems (yes/no, true/false) and can be extended to multiclass problems using strategies like one-vs-rest.
2. Large Datasets and High Dimensionality: Logistic Regression can handle larger datasets and high-dimensional data more efficiently than KNN.
3. Interpretability: If you need to understand the influence of each feature on the outcome, Logistic Regression is preferable as it provides coefficients indicating feature importance.
4. Linear Relationships: Use Logistic Regression when the relationship between the independent variables and the log odds of the dependent variable is approximately linear.
5. Probability Estimates: When you need a probability outcome (e.g., the probability that an email is spam), Logistic Regression is a more suitable choice.

KNN vs Logistic Regression: Python Code Example

Below are sample Python code snippets for performing classification using the K-Nearest Neighbors (KNN) algorithm and Logistic Regression, utilizing the Iris dataset from the sklearn library. This dataset is a classic in the field of machine learning, featuring measurements of iris flowers and is commonly used for classification tasks.

K-Nearest Neighbors Classification

# Import necessary libraries
from sklearn.datasets import load_iris
from sklearn.model_selection import train_test_split
from sklearn.neighbors import KNeighborsClassifier
from sklearn.metrics import classification_report, accuracy_score

# Load the iris dataset
iris = load_iris()
X = iris.data
y = iris.target

# Split the dataset into training and testing sets
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=42)

# Initialize the KNN classifier
knn = KNeighborsClassifier(n_neighbors=3)

# Train the model
knn.fit(X_train, y_train)

# Make predictions
y_pred = knn.predict(X_test)

# Evaluate the model
print("KNN Classification Report:")
print(classification_report(y_test, y_pred))
print(f"Accuracy: {accuracy_score(y_test, y_pred):.2f}")

Logistic Regression Classification

# Import necessary libraries
from sklearn.datasets import load_iris
from sklearn.model_selection import train_test_split
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import classification_report, accuracy_score

# Load the iris dataset
iris = load_iris()
X = iris.data
y = iris.target

# Split the dataset into training and testing sets
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=42)

# Initialize the Logistic Regression classifier
log_reg = LogisticRegression(max_iter=200)

# Train the model
log_reg.fit(X_train, y_train)

# Make predictions
y_pred = log_reg.predict(X_test)

# Evaluate the model
print("Logistic Regression Classification Report:")
print(classification_report(y_test, y_pred))
print(f"Accuracy: {accuracy_score(y_test, y_pred):.2f}")

In both scripts, the IRIS dataset is loaded and split into training and testing sets. The KNN model is trained with 3 neighbors, and the Logistic Regression model is trained using default parameters (you may adjust max_iter if needed for convergence). Finally, the models are evaluated using classification reports and accuracy scores.