We use the terms bias and variance or bias-variance trade-off to describe the performance of a machine learning model. In this article, I will introduce you to the concept of bias and variance in machine learning.
Bias and Variance in Machine Learning
In machine learning, you must have heard that the model has a high variance or high bias. So what does this mean? In general, one could say that a high variance is proportional to the overfitting and a high bias is proportional to the underfitting.
To understand what bias and variance are, suppose we have a point estimator of a parameter or function. Then, the bias is usually defined as the difference between the expected value of the estimator and the parameter we want to estimate.
If the bias is greater than zero, we also say that the estimator is positively biased, if the bias is less than zero, the estimator is negatively biased, and if the bias is exactly zero, the estimator is unbiased. Likewise, we define variance as the difference between the expected value of the estimator squared minus the expectation squared of the estimator.
What is High Bias?
Suppose there is an unknown target function or a true function that we want to get closer to. Suppose now that we have different training sets drawn from an unknown distribution defined as true function + noise. The graph shows different linear regression models, each fitted to a different training set.
None of these hypotheses comes close to the true function, except at two points. Here we can say that the bias is large because the difference between the actual value and the average predicted value is large.
What is High Variance?
Suppose there is an unknown target function or true function that we want to approximate. Suppose now that we have different training sets drawn from an unknown distribution defined as true function + noise. The graphic above shows different models of unadjusted decision trees, each corresponding to a different training set.
Note that these assumptions correspond very closely to the training data. However, if we were to consider the overtraining sets of expectations, the mean hypothesis would match the true function perfectly. However, as can be seen, the variance is very high, because on average a prediction differs a lot from the expected value of the prediction.
I hope you liked this article on the concept of Bias and Variance in Machine Learning. Feel free to ask your valuable questions in the comments section below.
