Machine Learning – Validation Techniques (Interview Questions)

Validation techniques in machine learning are used to get the error rate of the ML model which can be considered as close to the true error rate of the population. In case the data volume is large enough to be representative of the population, you may not need the validation techniques. However, in real world scenario, we work with the sample of data which may not be the true representative of the population. This is where validation techniques come into the picture.

In this post, you will briefly learn about different validation techniques such as following and also presented with practice test having questions and answers which could be used for interviews.

• Resubstitution
• Hold-out
• K-fold cross-validation
• LOOCV
• Random subsampling
• Bootstrapping

Revision Notes – Machine Learning Validation Techniques

• Resubstitution: In case, the whole data is used for training the model and the error rate is evaluated based on outcome vs actual value from the same training data set, this error is called as the resubstitution error. This technique is called as resubstitution validation technique.
• Hold-out: In order to avoid the resubstitution error, the data is split into two different data set labeled as training and test data set. This can be a 60-40 or 70-30 or 80-20 split. This technique is called as hold-out validation technique. In this case, there is a likelihood that uneven distribution of different classes of data is found in training and test data set. To fix this, the training and test data set is created with equal distribution of different classes of data. This process is called stratification.
• K-fold cross-validation: In this technique, k-1 folds are used for training and the remaining one is used for testing as shown in the picture given below.
  

  Figure 1. K-fold cross-validation

  The advantage is that entire data is used for training and testing. The error rate of the model is average of the error rate of each iteration. This technique can also be called as a form of Repeated Hold-out Method. The error rate could be improved by using stratification technique.

• Leave-one-out cross-validation (LOOCV): In this technique, all of the data except one record is used for training and one record is used for testing. This process is repeated for N times if there are N records. The advantage is that entire data is used for training and testing. The error rate of the model is average of the error rate of each iteration. The following diagram represents the LOOCV validation technique.
  

  Figure 2. LOOCV validation technique

• Random subsampling: In this technique, multiple sets of data is randomly chosen from the dataset and combined to form a test data set. The remaining data form the training dataset. The following diagram represents the random subsampling validation technique. The error rate of the model is average of the error rate of each iteration.
  

  Figure 3. Random Subsampling validation technique

• Bootstrapping: In this technique, the training data set is randomly selected with replacement. The remaining examples that were not selected for training are used for testing. Unlike K-fold cross-validation, the value is likely to change from fold-to-fold. The error rate of the model is average of the error rate of each iteration. The following diagram represents the same.
  

  Figure 4. Bootstrapping validation technique

Practice Test – ML Model Validation Techniques

Further Reading / References

• Validation techniques – TAMU
• Cross-validation
• Cross-validation

Summary

In this post, you learned about different validation techniques used for finding error rates of machine learning models.

Did you find this article useful? Do you have any questions or suggestions about this article in relation to machine learning model validation techniques? Leave a comment and ask your questions and I shall do my best to address your queries.