Bias-Variance Decomposition
1 Introduction
Supervised learning models typically do not achieve perfect generalization accuracy. Still, an understanding of the sources of error helps model designers better understand the behaviour of their models. Thus, much of statistical learning theory is an exploration of error and where error comes from. This unit provides a very basic understanding of the sources of error. It is an especially important unit for students interested in either ML theory or practical application. Understanding the sources of error is key to being able to make good modelling decisions.
We have already briefly explored how both underfitting and overfitting can increase a model’s generalization error. Many models have hyperparameters that control the capacity of the model: for example, \(k\) (the number of neighbours to consider) in \(k\)-Nearest Neighbours, and \(M\) (the maximum degree of the polynomial) in polynomial regression. Setting the capacity to be too high (e.g., \(k\) too low, \(M\) too high) would cause the trained model to overfit, and setting the capacity to be too low (e.g., \(k\) too high, \(M\) too low) would cause the trained model to underfit. Underfitting and overfitting both cause a model’s generalization error to be large. But intuitively, the model’s error comes from two different places: error due to underfitting has a consistency to it and is related to the simplicity of the model. In contrast, error due to overfitting is due to the model learning patterns in the training data that are accidental and do not generalize.
2 Setup
2.1 Regression Problem with One Feature
For this unit, we will focus on a regression problem with one input feature, using the following notation:
- \(x \in \mathbb{R}\) denotes the input feature.
- \(t \in \mathbb{R}\) denotes the target.
While the concepts we develop generalize to higher dimensions, working in one dimension allows us to visualize the key ideas of this section clearly and build intuition before tackling more complex scenarios.
2.2 The Data-Generating Distribution
Before we can analyze how a model’s error behaves, we need to ask: where does our data come from? To answer this question, we need to understand the data-generating distribution.
In supervised learning, we assume there is a true but unknown process that produces both the features and the targets. For a given input, this process does not always produce the same deterministic output due to the presence of noise. Consider predicting house prices: even two houses with the same square footage can sell for different prices, because factors we did not measure — location, condition, timing — introduce noise. Every dataset we collect is a finite sample from this underlying process.
More concretely, each data point \((x, t)\) is drawn from a joint distribution \(p(x, t)\), where \(x\) denotes the feature and \(t\) denotes the target. To interpret \(p(x, t)\), we decompose it into two parts using the chain rule:
\[p(x, t) = p(x) \cdot p(t \mid x)\]
\(p(x)\) is the marginal distribution of the inputs, describing how likely we are to encounter different feature values. For example, handwritten digits are much more likely to have stroke-like shapes than random noise, so \(p(x)\) is highly structured.
\(p(t \mid x)\) is the conditional distribution of targets given an input, capturing how much the target varies even when the input is fixed. This is the source of noise in the relationship between \(x\) and \(t\), and it is the mapping we ultimately aim to learn by fitting a regression model.
Definition: The data-generating distribution \(p(x, t)\) is the joint distribution over features \(x\) and targets \(t\). By the chain rule of probability:
\[p(x, t) = p(x) \cdot p(t \mid x)\]
where \(p(x)\) is the marginal distribution of the features and \(p(t \mid x)\) is the conditional distribution of the targets given the features.
Each dataset is a finite sample drawn from this distribution. We denote a dataset as:
\[\mathcal{D} = \{ (x^{(i)}, t^{(i)})\}_{i=1}^N\]
We write \((x^{(i)}, t^{(i)}) \sim p(x,t)\), meaning each data point is drawn independently and identically (i.i.d.) from \(p(x,t)\). Equivalently, we can think of sampling a dataset as a two-step process for each data point:
- Draw an input \(x^{(i)} \sim p(x)\).
- Draw a target \(t^{(i)} \sim p(t \mid x^{(i)})\).
Here is a key consequence of this setup: the training set \(\mathcal{D}\) is itself a random variable. Two datasets drawn from the same distribution are typically not identical. Different draws from \(p(x)\) give different input values, and \(p(t \mid x)\) adds noise. In particular, two points with the same input can still have different targets. A particular observed dataset is just one realization of this underlying random process.
To give a concrete example, suppose our data-generating distribution is:
- \(x \sim \text{Uniform}(0, 8)\)
- \(t \mid x \sim \mathcal{N}(\sin(x), 0.25^2)\)
In step 1, each input \(x\) is drawn uniformly from \([0, 8]\). In step 2, given that \(x\), the target \(t\) is drawn from a normal distribution centred at \(\sin(x)\) with standard deviation \(0.25\). The true underlying function is \(\sin(x)\), but each dataset we collect will follow the sine curve with some random noise on top of it.
We can see this directly. Pressing “Resample Dataset” below draws a fresh training set of 20 points from this distribution:
Samples from the data-generating distribution
Question: Briefly explain why the training set \(\mathcal{D}\) is a random variable.
2.3 The Learning Algorithm
Now that we have a model of the data, we need a model of the fitting process itself. We have already discussed several algorithms for learning a regressor \(f\) from a training set \(\mathcal{D}\): linear regression, \(k\)-Nearest Neighbours, and decision trees are some examples. The content of this unit can be applied to any learning algorithm, so we will not specify one in particular. Instead, we denote a learning algorithm by \(\mathcal{A}\). A learning algorithm can be thought of as a function that takes a training set \(\mathcal{D}\) as input and produces a predictor \(f\) as output. (So, for example, if our learning algorithm were linear regression, then a predictor would be the line learned from one specific training set.)
For simplicity, we will assume the algorithm is deterministic — meaning that given the same training set, it always produces the same predictor. Most standard algorithms, including linear regression and \(k\)-Nearest Neighbours, satisfy this property.
Definition: A learning algorithm \(\mathcal{A}\) takes a training set \(\mathcal{D}\) as input and produces a predictor \(f\) as output:
\[\mathcal{A}(\mathcal{D}) = f\]
That is, feeding a training set \(\mathcal{D}\) into the algorithm \(\mathcal{A}\) yields a fitted predictor \(f\).
Here is a key consequence: the predictor \(f_\mathcal{D}\) is itself a random variable. Because \(\mathcal{D}\) is a random variable — it changes with each draw from the data-generating distribution — and \(f_\mathcal{D}\) is a deterministic function of \(\mathcal{D}\), the predictor changes too. We write \(f_\mathcal{D}\) instead of just \(f\) to make this dependence on the training set explicit.
Question: Briefly explain why the predictor \(f_{\mathcal{D}}\) is a random variable.
We will work with two concrete algorithms throughout this unit: a degree-15 polynomial regressor and a linear regressor. We choose these two because they behave very differently — one turns out to have high variance and the other high bias (terms you will shortly become acquainted with). Comparing them side by side will build intuition about their properties before the formal derivation.
Polynomial Regressor
Let’s consider a polynomial regression model:
\[y = f(x) = w_0 + w_1 x + w_2 x^2 + \dots + w_{15} x^{15}.\]
The following visualization lets us explore how this model behaves across different training sets. Each press of the button performs two steps:
- A new training set of 20 data points is sampled.
- All parameters \(w_j\) are fit to minimize the total squared loss on this training set.
The main plot shows the 20 sampled points and the fitted polynomial curve.
We will pay special attention to one input \(x = 4.5\), shown as a red dot on the main plot. The panel on the right summarizes what happens at this test point:
The green bar indicates the best possible (optimal) prediction for \(x = 4.5\), given the noise in the data. (We will discuss this concept more formally later.)
The red dots show the predictions made by polynomial regressors trained on different sampled datasets. Each press of the button adds a new red dot to the panel on the right.
The arrow indicates the average prediction across different training sets for the input \(x = 4.5\).
Pressing the button a few times, we can observe how the polynomial regressor changes across training sets. As we do, we should pay attention to two things:
- How large is the spread of the model predictions (the red dots)? This spread is related to what we will call variance.
- How do the model predictions (the red dots) compare to the optimal prediction (the green bar)? A systematic gap is related to what we will call bias.
High-variance polynomial predictor
Linear Regressor
Let’s consider a linear regression model:
\[y = f(x) = wx+b.\]
The following visualization works the same way as the previous one, but for a linear regressor. Each press of the button samples a new training set of 20 points and fits the parameters \(w\) and \(b\) to minimize the squared loss. The panel on the right shows the individual predictions, the average prediction, and the optimal prediction for \(x = 4.5\).
We should pay attention to the same two questions as before:
- How large is the spread of the model predictions (the red dots)?
- How do the model predictions (the red dots) compare to the optimal prediction (the green bar)?
High-bias linear predictor
In the following sections, we will compare the polynomial and linear regressors side by side and develop intuitions for their properties through high-level descriptions of variance and bias. These descriptions are not yet mathematically rigorous, but they will prepare us for the formal derivation that follows.
2.4 Variance
In Figure 2, we noticed that the polynomial regressor’s fitted curve shifts noticeably from one training set to the next. In contrast, Figure 3 shows that the linear regressor’s fitted curve stays relatively stable. This sensitivity to the training set is captured by the concept of variance. Variance measures how much a model’s predictions change when trained on different datasets drawn from the same data-generating distribution. A high-variance model produces quite different predictions depending on which training set it happened to see, whereas a low-variance model produces similar predictions across different training sets.
Question: Based on the description of variance above and the visualizations in Figure 2 and Figure 3, which of the two models has larger variance and why?
- Linear Regressor
- Polynomial Regressor
Answer:
The polynomial regressor has larger variance.Returning to Figure 2, we can see why the polynomial regressor has high variance. The degree-15 polynomial is highly flexible — it can fit many different curve shapes. That flexibility makes it sensitive to the particular training set it sees: small differences between training sets produce large differences in the fitted curve, and therefore large differences in predictions across training sets. As shown in Figure 3, the linear regressor, being less flexible, produces more consistent predictions across training sets and therefore has lower variance.
Variance is an inherent property of the learning algorithm — it does not depend on any particular data point, but rather on how much the algorithm’s output changes in response to different training sets. A high-variance algorithm produces predictions that vary widely across training sets, which means its error on any given input is less predictable and harder to control. This makes variance a fundamental source of error.
In the next section, we will see that the linear regressor pays a different price for its simplicity.
2.5 Bias
We noticed in Figure 3 that no matter how many training sets we sample, the linear regressor’s predictions at \(x = 4.5\) are consistently off from the optimal prediction — always in the same direction. What concept captures this systematic deviation? Bias measures how far the average prediction of the model is from the optimal prediction. In other words, a high-bias model is consistently wrong in the same direction, regardless of which training set it saw.
Question: Based on the description of bias above, which of the two models has larger bias?
- Linear Regressor
- Polynomial Regressor
Answer:
The linear regressor has larger bias.Returning to Figure 3, we can see why the linear regressor has high bias. The linear regressor is constrained to fit a straight line, but the true underlying function is \(\sin(x)\) — which is curved. No matter how much data the linear regressor sees, it cannot represent the true shape of the function. This mismatch causes its average prediction to systematically deviate from the optimal prediction. As shown in Figure 2, the polynomial regressor is more flexible and can approximate the sine curve much more closely, so its average prediction aligns more closely with the optimal prediction. As a result, the polynomial regressor has lower bias.
Bias is an inherent property of the learning algorithm — specifically, of the assumptions the algorithm makes about the relationship between input and target. A high-bias algorithm consistently produces predictions that are off in the same direction, regardless of which training set it sees. This systematic error makes bias a fundamental source of prediction error.
In the next section, we will see that there is a third source of error that neither bias nor variance can explain.
2.6 Bayes Error
We have seen that both bias and variance contribute to a model’s prediction error. But even if we could eliminate both, could a learning algorithm ever achieve zero prediction error? The data-generating process itself contains randomness, so the answer is no.
Recall from Figure 1 that given an input \(x\), the target \(t\) is drawn from the conditional distribution \(p(t \mid x)\) — that is, the target \(t\) is not a fixed function of the input \(x\), but a random draw. This means two data points with the same input \(x\) can have different target values. No deterministic model can predict both correctly. The prediction error that results from this inherent noise in the data — error that no model or algorithm can eliminate — is called the irreducible error or Bayes error.
Given that some error is unavoidable, what is the best prediction we can make for a given input \(x\)? It is the conditional mean \(\mathbb{E}[t \mid x]\) — the average value of the target \(t\) over all possible draws from \(p(t \mid x)\). Predicting this value minimizes the expected squared error at the input \(x\): any prediction above or below the mean incurs extra error on average. A model that outputs \(\mathbb{E}[t \mid x]\) for every input is called the optimal predictor (or the Bayes predictor).
In our running example in Figure 1, the target \(t\) is drawn from a normal distribution centred at \(\sin(x)\) with standard deviation \(0.25\) (i.e., \(t \mid x \sim \mathcal{N}(\sin(x), 0.25^2)\)). Therefore, the conditional mean is \(\mathbb{E}[t \mid x] = \sin(x)\). The optimal predictor is simply the sine function. Pressing “Resample Dataset” below draws a new training set — the optimal predictor stays fixed no matter which dataset appears, because it depends only on the data-generating distribution \(p(t \mid x)\), not on any particular training set.
Optimal predictor
For the specific test point \(x = 4.5\) considered above, the optimal prediction is \(\bar{y}(4.5) = \sin(4.5) = -0.978\).
The Bayes error is the expected prediction error of the optimal predictor — in other words, the lowest prediction error any model can possibly achieve on this data-generating distribution. No matter how powerful the model or how large the training set, no algorithm can do better than the Bayes error. It is a property of the data-generating distribution itself, not of any learning algorithm.
This also means the optimal predictor is not a random variable: it is determined entirely by \(p(t \mid x)\), which is fixed, and does not depend on the training set \(\mathcal{D}\).
Question: Briefly explain why the optimal predictor is not a random variable.
3 Decomposing the Expected Test Error
For the remainder of this unit, our goal is to decompose the expected test error made by a regressor into three sources: the variance, the bias, and the Bayes error. We work with the squared error loss and drop the \(1/2\) factor for notational convenience.
For a regression problem using the squared error loss, the expected test error of a learning algorithm \(\mathcal{A}\) is: \[ \mathbb{E} [ (f_\mathcal{D}(x) - t)^2 ] \]
A common challenge is to interpret the expectations (denoted by \(\mathbb{E} [ \cdot ]\)) in the expression above. Normally, expectations are taken with respect to some random variables. What are the relevant random variables in this expression?
The expectation is taken over the training set \(\mathcal{D}\) and the test data point \((x,t)\). The training set \(\mathcal{D}\) is a random variable because it is drawn from the data-generating distribution \(p_{\mathcal{D}}\). The test data point \((x,t)\) is a random variable because it is drawn from the data-generating distribution \(p(t \mid x)\).
In the following sections, we will show that the expected test error decomposes into three quantities:
\[\underbrace{\mathbb{E}_{x,t,\mathcal{D}}\!\left[ \big(f_\mathcal{D}(x) - t\big)^2 \right]}_{\text{Expected Test Error}} = \underbrace{\mathbb{E}_{x,\mathcal{D}}\!\left[ \big(f_\mathcal{D}(x) - \bar{f}(x)\big)^2 \right]}_{\text{Variance}} + \underbrace{\mathbb{E}_{x}\!\left[ \big(\bar{f}(x) - \bar{y}(x)\big)^2 \right]}_{\text{Bias}^2} + \underbrace{\mathbb{E}_{x,t}\!\left[ \big(\bar{y}(x) - t\big)^2 \right]}_{\text{Bayes Error}}\]
where \(\bar{f}(x) = \mathbb{E}_\mathcal{D}[f_\mathcal{D}(x)]\) is the average predictor — the prediction you would get if you averaged over all possible training sets — and \(\bar{y}(x) = \mathbb{E}_t[ t \mid x]\) is the optimal predictor — the best possible prediction given the input \(x\). Both are defined formally in the sections below.
3.1 Isolating the Variance
We will start by isolating the variance from the expected test error. To do this, we need to define a new concept called the average predictor.
Definition: The average predictor \(\bar{f}\) is the average of predictors trained on all possible training sets drawn from the data-generating distribution. \[ \bar{f}(x) = \mathbb{E}_\mathcal{D}[f_\mathcal{D}(x)] \]
The key idea is to add and subtract the average predictor \(\bar{f}(x)\) inside the square. This splits the single error term into two parts.
\[\begin{align} & \mathbb{E}_{x,t,\mathcal{D}}\!\left[ \big(f_\mathcal{D}(x) - t\big)^2 \right] \\ = &\mathbb{E}_{x,t,\mathcal{D}}\!\left[ \left( \left(f_\mathcal{D}(x) - \bar{f}(x) \right) + \left(\bar{f}(x) - t \right) \right)^2 \right] \label{exp_1} \\ = &\mathbb{E}_{x,t,\mathcal{D}}\!\left[ \big(f_\mathcal{D}(x) - \bar{f}(x)\big)^2 + 2 \big(f_\mathcal{D}(x) - \bar{f}(x)\big) \big( \bar{f}(x) - t\big)+ \big( \bar{f}(x) - t\big)^2 \right] \label{exp_2} \\ = &\mathbb{E}_{x,t,\mathcal{D}}\!\left[ \big(f_\mathcal{D}(x) - \bar{f}(x)\big)^2 \right] + 2 \mathbb{E}_{x,t,\mathcal{D}}\!\left[ \big(f_\mathcal{D}(x) - \bar{f}(x)\big) \big( \bar{f}(x) - t\big)\right] + \mathbb{E}_{x,t,\mathcal{D}}\!\left[ \big( \bar{f}(x) - t\big)^2 \right] \label{exp_3} \end{align}\]
In line \(\eqref{exp_1}\), we add and subtract the average predictor \(\bar{f}(x)\) inside the square. In line \(\eqref{exp_2}\), we expand the square using \((a+b)^2 = a^2 + 2ab + b^2\). In line \(\eqref{exp_3}\), we distribute the expectation across the three terms by linearity.
Next, we can show that the middle term \(2 \mathbb{E}_{x,t,\mathcal{D}}\!\left[ \big(f_\mathcal{D}(x) - \bar{f}(x)\big) \big( \bar{f}(x) - t\big)\right]\) is equal to \(0\) and can be removed.
\[\begin{align} \mathbb{E}_{x,t,\mathcal{D}}\!\left[ \big(f_\mathcal{D}(x) - \bar{f}(x)\big) \big( \bar{f}(x) - t\big)\right] &= \mathbb{E}_{x,t}\left[ \mathbb{E}_{\mathcal{D}}\!\left[ \big(f_\mathcal{D}(x) - \bar{f}(x)\big) \big( \bar{f}(x) - t\big)\right] \right] \label{der_1} \\ &= \mathbb{E}_{x,t}\left[ \mathbb{E}_{\mathcal{D}}\!\left[ \big(f_\mathcal{D}(x) - \bar{f}(x)\big) \right]\big( \bar{f}(x) - t\big) \right] \label{der_2} \\ &= \mathbb{E}_{x,t}\left[ \big( \mathbb{E}_{\mathcal{D}}\!\left[ f_\mathcal{D}(x) \right]- \mathbb{E}_{\mathcal{D}}\!\left[ \bar{f}(x) \right] \big)\big( \bar{f}(x) - t\big) \right] \label{der_3} \\ &= \mathbb{E}_{x,t}\left[ \big( \bar{f}(x)- \bar{f}(x) \big)\big( \bar{f}(x) - t\big) \right] \label{der_4} \\ &= 0 \end{align}\]
In line \(\eqref{der_1}\), we separate the expectation over the training set \(\mathcal{D}\) from the expectation over the test point \((x,t)\) by the law of total expectation. In line \(\eqref{der_2}\), we notice that the average predictor \(\bar{f}(x)\) and the target \(t\) are both independent of the training set \(\mathcal{D}\). As a result, we can move the term \(\bar{f}(x) - t\) outside of the expectation over \(\mathcal{D}\). In line \(\eqref{der_3}\), we distribute the expectation over the training set across the two terms using the linearity of expectation. Finally, in line \(\eqref{der_4}\), we substitute the definition of the average predictor, \(\bar{f}(x) = \mathbb{E}_\mathcal{D}[f_\mathcal{D}(x)]\), which simplifies the expression to \(\bar{f}(x) - \bar{f}(x) = 0\).
After removing the middle term, we are left with the expression below.
\[\begin{align} = & \mathbb{E}_{x,t,\mathcal{D}}\!\left[ \big(f_\mathcal{D}(x) - \bar{f}(x)\big)^2 \right] + \mathbb{E}_{x,t,\mathcal{D}}\!\left[ \big( \bar{f}(x) - t\big)^2 \right] \label{var_1} \end{align}\]
We can simplify both expectations further. Notice that the quantity inside the first term, \(f_\mathcal{D}(x) - \bar{f}(x)\), is independent of the target \(t\). Thus, we can simplify the first term by removing the expectation over \(t\). Similarly, the quantity inside the second term, \(\bar{f}(x) - t\), is independent of the training set \(\mathcal{D}\). Thus, we can simplify the second term by removing the expectation over \(\mathcal{D}\). We are left with the expression below.
\[\begin{align*} = & \underbrace{\mathbb{E}_{x,\mathcal{D}}\!\left[ \big(f_\mathcal{D}(x) - \bar{f}(x)\big)^2 \right]}_{\text{Variance}} + \mathbb{E}_{x,t}\!\left[ \big( \bar{f}(x) - t\big)^2 \right] \end{align*}\]
We will explain later that the first term is the variance. For now, let’s continue the derivation by isolating the bias from the second term.
Question: Consider the expression in line \(\eqref{var_1}\). Why is the first term \(f_\mathcal{D}(x) - \bar{f}(x)\) independent of the target \(t\)? Why is the second term \(\bar{f}(x) - t\) independent of the training set \(\mathcal{D}\)?
3.2 Isolating the Bias
To isolate the bias from the second term, we will need to use the concept of the optimal predictor, which we encountered in the previous sections. Let’s start by formally defining the optimal predictor below.
Definition: The optimal predictor \(\bar{y}\) is the expected target value given the input. \[ \bar{y}(x) = \mathbb{E}_t[ t | x] \] This represents the predictor that achieves the minimum expected loss on the test data.
We will use techniques very similar to those in the previous section to isolate the bias.
\[\begin{align} & \mathbb{E}_{x,t}\!\left[ \left( \bar{f}(x) - t \right)^2 \right] \label{der_6}\\ = & \mathbb{E}_{x,t}\!\left[ \left( \left(\bar{f}(x) - \bar{y}(x) \right) + \left( \bar{y}(x) - t \right) \right)^2 \right] \label{der_7}\\ = & \mathbb{E}_{x,t}\!\left[ \big(\bar{f}(x) - \bar{y}(x)\big)^2 + 2 \big( \bar{f}(x) - \bar{y}(x) \big) \big( \bar{y}(x) - t \big) + \big( \bar{y}(x) - t \big)^2 \right] \label{der_8}\\ = & \mathbb{E}_{x,t}\!\left[ \big(\bar{f}(x) - \bar{y}(x)\big)^2\right] + 2 \mathbb{E}_{x,t}\!\left[ \big(\bar{f}(x) - \bar{y}(x)\big)( \bar{y}(x) - t\big) \right] + \mathbb{E}_{x,t}\!\left[ \left( \bar{y}(x) - t \right)^2 \right] \label{der_9} \end{align}\]
Line \(\eqref{der_6}\) is the second term carried over from the variance derivation. In line \(\eqref{der_7}\), we add and subtract the optimal predictor \(\bar{y}(x)\) inside the square. In line \(\eqref{der_8}\), we expand the square using the rule \((a + b)^2 = a^2 + 2ab + b^2\). In line \(\eqref{der_9}\), we use the linearity of expectation to distribute the expectation over the test point \((x,t)\) across the three terms.
Using an argument similar to the one in the previous section, we can show that the middle term \(2 \mathbb{E}_{x,t}\!\left[ \big(\bar{f}(x) - \bar{y}(x)\big)( \bar{y}(x) - t\big) \right]\) is equal to \(0\) and can be removed.
\[\begin{align} \mathbb{E}_{x,t}\!\left[ \big(\bar{f}(x) - \bar{y}(x)\big) \big( \bar{y}(x) - t\big) \right] = & \mathbb{E}_{x}\!\left[ \mathbb{E}_{t}\!\left[ \big(\bar{f}(x) - \bar{y}(x)\big)( \bar{y}(x) - t\big) | x \right] \right] \label{der_10} \\ = & \mathbb{E}_{x}\!\left[ \big(\bar{f}(x) - \bar{y}(x)\big) \mathbb{E}_{t}\!\left[ \bar{y}(x) - t | x \right] \right] \label{der_10b}\\ = & \mathbb{E}_{x}\!\left[ \left(\bar{f}(x) - \bar{y}(x) \right) \left( \bar{y}(x) - \mathbb{E}_{t}\!\left[ t | x \right] \right) \right] \label{der_11}\\ = & \mathbb{E}_{x}\!\left[ \big( \bar{f}(x) - \bar{y}(x) \big) \big( \bar{y}(x) - \bar{y}(x) \big) \right] \label{der_12} \\ = & 0 \end{align}\]
In line \(\eqref{der_10}\), we separate the expectation over the target value \(t\) from the expectation over the input \(x\) by the law of total expectation. In line \(\eqref{der_10b}\), we recognize that the term \(\bar{f}(x) - \bar{y}(x)\) is independent of the target \(t\) and can be moved outside of the expectation over \(t\). In line \(\eqref{der_11}\), we use linearity of expectation to write \(\mathbb{E}_t[\bar{y}(x) - t \mid x] = \bar{y}(x) - \mathbb{E}_t[t \mid x]\). Finally, in line \(\eqref{der_12}\), we substitute the definition of the optimal predictor \(\bar{y}(x) = \mathbb{E}_t[ t | x]\), which simplifies the expression to \(\bar{y}(x) - \bar{y}(x) = 0\).
After removing the middle term, we are left with two terms. Notice that \(\bar{f}(x) - \bar{y}(x)\) is independent of \(t\), so we can drop \(\mathbb{E}_t\) from the first term. We are left with the expression below.
\[\begin{align*} \mathbb{E}_{x,t}\!\left[ \left( \bar{f}(x) - t \right)^2 \right] = & \underbrace{\mathbb{E}_{x}\!\left[ \big(\bar{f}(x) - \bar{y}(x)\big)^2\right]}_{\text{Bias}^2} + \underbrace{\mathbb{E}_{x,t}\!\left[ \big( \bar{y}(x) - t\big)^2 \right]}_{\text{Bayes Error}} \end{align*}\]
3.3 Summarizing the Decomposition
Combining the two steps, we get the following expression for the expected test error.
\[\begin{align*} & \mathbb{E}_{x,t,\mathcal{D}}\!\left[ \big(f_\mathcal{D}(x) - t\big)^2 \right] = & \underbrace{\mathbb{E}_{x,\mathcal{D}}\!\left[ \big(f_\mathcal{D}(x) - \bar{f}(x)\big)^2 \right]}_{\text{Variance}} + \underbrace{\mathbb{E}_{x}\!\left[ \big( \bar{f}(x) - \bar{y}(x)\big)^2 \right]}_{\text{Bias}^2} + \underbrace{\mathbb{E}_{x,t}\!\left[ \big( \bar{y}(x) - t\big)^2 \right]}_{\text{Bayes Error}} \end{align*}\]
Let’s interpret each term in the decomposition and use them to mathematically define the variance, bias, and Bayes error. As you proceed through these definitions, you should verify for yourself that these terms align with the intuition you have built in this unit.
The first term is the variance. It measures the variability of the predictor \(f_\mathcal{D}(x)\) across different training sets \(\mathcal{D}\).
Definition: The variance of a learning algorithm \(\mathcal{A}\) measures the expected variation in the predictors \(f_\mathcal{D}(x)\) across different training sets \(\mathcal{D}\). \[ \mathbb{E}_{x,\mathcal{D}}\!\left[ \big(f_\mathcal{D}(x) - \bar{f}(x)\big)^2 \right] \]
Going back to Figure 3 and Figure 2, the variance can be visualized by the variation in the red dots, which are the predictions for one test point made by different predictors. We can see that the variance is high for the polynomial regressor with degree 15 and low for the linear regressor.
A learning algorithm tends to have large variance when it is flexible. Such a model is sensitive to noise in the training set and can easily overfit to the training data.
The second term is the bias squared. It measures the amount of systematic deviation from the average prediction to the optimal prediction, which is also the expected target value given the input.
Definition: The bias of a learning algorithm \(\mathcal{A}\) measures the expected amount of systematic deviation from the average predictor \(\bar{f}(x)\) to the optimal predictor \(\bar{y}(x)\). \[ \mathbb{E}_{x}\!\left[ \big( \bar{f}(x) - \bar{y}(x)\big)^2 \right] \]
Looking at Figure 3 and Figure 2, the bias can be visualized by the distance between the average of the red dots and the green line. We can see that the bias is low for the polynomial regressor with degree 15 and high for the linear regressor.
A learning algorithm tends to have large bias when it has made erroneous assumptions about the data-generating process. For example, if the data-generating process is not linear, a linear regressor will have a large bias because no straight line can match the true function on average. Such systematic error often manifests as underfitting, but bias is the cause — underfitting is the symptom.
Last but not least, the third term is the Bayes error. Bayes error is a property of the data-generating process, whereas bias and variance are properties of the learning algorithm. Bayes error measures the amount of inherent noise in the problem, due to multiple test data points having the same features \(x\) but different target values \(t\).
Definition: The Bayes error of a problem (shown below) measures the amount of noise inherent in the problem, due to multiple test data points having the same features \(x\) but different target values \(t\). \[ \mathbb{E}_{x,t}\!\left[ \big( \bar{y}(x) - t\big)^2 \right] \]
Bayes error is often referred to as the irreducible error or the inherent noise in the dataset. If Bayes error is non-zero, then it is impossible to achieve zero loss. One important implication is that it is impossible to achieve a test error lower than the Bayes error. This is because the Bayes error is the minimum amount of error that can be achieved by any predictor given a data-generating process.
However, it is possible to reduce the Bayes error if we can change the data-generating process by collecting new features. These features might reveal previously hidden information, and such information might allow us to distinguish between different target values even when they used to have the same features.
3.4 Interpreting the Decomposition for Practical ML
The decomposition tells us that error in machine learning models comes from three places, and helps us determine what changes to make in order to build models that generalize better.
We have already seen, intuitively, that bias and variance tend to move in opposite directions as we change the algorithm’s capacity. That is, there is a bias-variance tradeoff. The toy example illustrates this directly. The linear regression algorithm is high bias, low variance: its predictions are consistently wrong in the same way, but stable across training sets. The degree-15 polynomial is low bias, high variance: on average it approximates the optimal predictor \(\bar{y}\) well, but its predictions swing widely depending on which training set it saw.
In practice, we cannot directly compute bias or variance for a real problem. Instead, we need to gather indirect evidence from the gap between training error and validation error. If both training error and validation error are high, the model is likely suffering from high bias. The fix is to increase capacity, for example by using a more expressive model or reducing regularization. If training error is low but validation error is substantially higher, the model is likely suffering from high variance. Fixes include collecting more training data, adding regularization, or using ensemble methods, to be discussed in the next section.
In theory, using a simpler model would also be an option to manage variance. However, high variance is easier to diagnose than high bias, since both high bias and high Bayes error manifest in high training and validation error, and are difficult to distinguish. We therefore almost always start by using a model that will overfit, and then use methods like regularization to control that overfitting.
While bias and variance are properties of the learning algorithm \(\mathcal{A}\), the Bayes error is fixed by the problem and no choice of learning algorithm can reduce it. For example, if the learning problem is to predict a person’s hair length given their shoe size, you would not expect any algorithm (or even a human!) to be able to make very accurate predictions. In real problems, the Bayes error is unknown, but human-level performance is often used as a rough proxy. If your validation error is already close to that proxy, further tuning of capacity or regularization will yield diminishing returns.
But that doesn’t mean that we should ignore the Bayes error. In practical applications, knowing that the Bayes error is high may be a cue to change the learning problem. It might tell you that additional features are required, or that the problem space should be restricted (e.g., it may be possible to predict, with reasonable accuracy, the hair length given the shoe size of young infants). In short, besides model capacity, a whole host of settings affect model accuracy: the size of the training set, the features used, some optimization parameters, and more. An understanding of the bias-variance decomposition can help diagnose the source of error and guide decisions about which of these levers to adjust.
3.5 Caveat: Double Descent
The discussion above predicts a U-shaped test error curve as model capacity increases: error is high for very simple models (high bias), decreases as capacity grows, then rises again as overfitting sets in (high variance). However, in practice, deep neural networks with billions of parameters often perform well, despite having a very large capacity, even compared to the amount of training data they are trained on. Why is that?
It turns out that as capacity increases past the point where the model can exactly interpolate the training data, test error decreases again, sometimes well below the classical minimum. This phenomenon is called double descent.
An understanding of double descent is beyond the scope of the course, so we will only cover the intuition here. The idea is that in the heavily overparameterized regime, there are many models that perfectly fit the training data. When using optimization via gradient descent or related techniques, the optimization process tends to find smooth, well-behaved ones—especially when regularization is used or when weights are initialized close to 0. The classical analysis implicitly assumes that interpolating models are necessarily high-variance, but this is not always true.
Double descent does not invalidate the bias-variance decomposition. The decomposition is an exact identity. However, double descent tells us that the relationship between model capacity and variance is more subtle than the classical picture suggests. It remains an active area of research.
4 Summary
The bias-variance decomposition formalizes an important aspect of Fundamental Idea #2: error is not monolithic, but decomposes into bias, variance, and irreducible Bayes error. These three distinct sources call for different remedies. Because none of these quantities can be measured directly in practice, a practitioner must rely on indirect evidence from training and validation error to diagnose which source dominates, and then try different settings to observe the effect. This is the essence of Fundamental Idea #3: model evaluation is empirical.
The derivation also illustrates Fundamental Idea #5: ML demands a probabilistic lens. The key conceptual shift in this chapter is recognizing that the expected test error is not just an expectation over test points, but over the entire system, including the training data that produced the model. Treating the training set as a random variable, and the learned predictor as a function of that random variable, is what makes the decomposition possible. This probabilistic framing is essential; it is the foundation for thinking rigorously about generalization.