Machine learning models can be parametric or non-parametric. Parametric models are those that require the specification of some parameters before they can be used to make predictions, while non-parametric models do not rely on any specific parameter settings and therefore often produce more accurate results. This blog post discusses parametric vs non-parametric machine learning models with examples along with the key differences.
What are parametric and non-parametric models?
Training machine learning models is about finding a function approximation built using input or predictor variables, and whose output represents the response variable. The reason why it is called function approximation is because there is always an error in relation to the value of function output vs actual or real-world value. And, an aspect of this error is reducible in the sense that further features / technique can be used to improve upon the accuracy. Another aspect of this error is irreducible which represents the random error which can’t be dealt with. Learn greater details on basic concepts of machine learning in my another post – What is machine learning? Concepts and examples.
When estimating the function (called function approximation), the following two steps are followed:
- Identifying the function: The first step is to identify the function such as linear or non-linear function.
- Identifying the parameters of the function in case this is linear function.
In case, the function identified is linear function (model), the training or fitting of the machine learning models boils down to estimating the parameters. Here is an example of the linear model also called as linear regression model and related parameters (coefficients).
Such models are called as parametric machine learning models. The parametric models are linear models which includes determining the parameters such as that shown above. The most common approach to fitting the above model is referred to as ordinary least squares (OLS) method. However, least squares is one of many possible ways to fit the linear model. Example of parametric models include linear algorithms such as Lasso regression, linear regression and to an extent, generalized additive models (GAMs).
Building non-parametric models do not make explicit assumptions about the functional form such as linear model in case of parametric models. Instead non-parametric models can be seen as the function approximation that gets as close to the data points as possible. The advantage over parametric approaches is that by avoiding the assumption of a particular functional form such as linear model, non-parametric models have the potential to accurately fit a wider range of possible shapes for the actual or true function. Any parametric approach brings with it the possibility that the functional form (linear model) which is very different from the true function, in which case the resulting model will not fit the data well. Example of non-parametric models include fully non-linear algorithms such as bagging, boosting, support vector machines bagging boosting with non-linear kernels, and neural networks (deep learning).
What’s the difference between parametric and non-parametric models?
The following is the list of differences between parametric and non-parametric machine learning models.
- In case of parametric models, the assumption related to the functional form is made and linear model is considered. In case of non-parametric models, the assumption about the functional form is not made.
- Parametric models are much easier to fit than non-parametric models because parametric machine learning models only require the estimation of a set of parameters as the model is identified prior as linear model. In case of non-parametric model, one needs to estimate some arbitrary function which is a much difficult task.
- Parametric models often do not match the unknown function we are trying to estimate. The model performance is comparatively lower than the non-parametric models. The estimates done by the parametric models will be farther from being true.
- Parametric models are interpretable unlike the non-parametric models. This essentially means that one can go for parametric models when the goal is to find inference. Instead, one can choose to go for non-parametric models when the goal is to make prediction with higher accuracy and interpretability or inference is not the key ask.
When to use parametric vs non-parametric algorithms / methods for building machine learning models?
When the goal is to achieve models with high performance prediction accuracy, one can go for non-linear methods such as bagging, boosting, support vector machines bagging boosting with non-linear kernels, and neural networks (deep learning). When the goal is to achieve modeling for making inferences, one can go for parametric methods such as lasso regression, linear regression etc which have high interpretability. You may want to check a related post on difference between prediction and inference – Machine learning: Prediction & Inference Difference
The parametric vs. non-parametric machine learning models debate is a longstanding one and it’s not easy to find an answer that satisfies everyone. What we can say for sure is this: parametric models are easier to work with but they don’t always produce the most accurate results, whereas non-parametric models require more time and effort upfront but will give you better accuracy in your predictions if done correctly. Whether you should use parametric or non-parametric depends on what your goals are as well as how comfortable your team members feel about working with them – hopefully now that we’ve given some insight into both sides of the argument, you’ll be able to make up your own mind!
Check out my latest book titled as First Principles Thinking: Building winning products using first principles thinking
- Generative Modeling in Machine Learning: Examples - March 19, 2023
- Data Analytics Training Program (Beginners) - March 18, 2023
- Histogram Plots using Matplotlib & Pandas: Python - March 18, 2023
Nice piece. Thanks a lot. I enjoyed every bit of this blog.