Regularization

Learning Objectives

After reading this page, you should be able to:

  1. Explain why large weights are a sign of overfitting.
  2. Explain how adding an L2 regularizer to the cost function reduces overfitting.
  3. Describe how the regularization strength \(\lambda\) affects underfitting and overfitting.

1 Introduction

In the previous section on feature mapping, we discussed how we can engineer additional features by applying non-linear transformations to an existing set of features. In particular, we examined the case of expanding a single feature, \(x\), into a higher-order polynomial of the form:

\[y = b + w_1 x + w_2 x^2 + \cdots + w_M x^M = b + \sum_{j=1}^M w_j x^j = \mathbf{w}^\top \psi(x) + b\]

using the mapping function:

\[\psi(x) = \begin{bmatrix}x\\ x^2 \\ \vdots \\ x^M\end{bmatrix}, \qquad \mathbf{w} = \begin{bmatrix}w_1\\ w_2 \\ \vdots \\ w_M\end{bmatrix}\]

Here, we keep the bias \(b\) separate from the weight vector \(\mathbf{w}\).

From our sine wave figure, recall that as \(M\), the degree of our polynomial, was increased, we eventually observed overfitting in our model. In other words, the training error continued to decrease with greater \(M\), but our model no longer appeared to generalize very well. It no longer looked very much like a sine wave and would not be expected to make very good predictions on unseen data.

Below, we have reproduced the same sine wave figure, with a few changes. First, we generate only ten points instead of twenty by adding noise to our sine wave. Second, we display the weights, stored in vector \(\mathbf{w}\), learned by linear regression for each value of \(M\). Try playing with the slider for \(M\) in the figure below and pay attention to the weights learned. What do you notice?

Polynomial regression with learned weights

Learned weights: b = …


Figure 1: The gray curve is the underlying function \(\sin(x)\); blue dots are training points sampled from it with added noise. The pink curve is the degree-\(M\) polynomial fit to those training points. The weights learned, \(\mathbf{w}\), are displayed to the right of the plot. Drag the slider to change \(M\): at low \(M\) the model underfits because it is too rigid to follow the data; at high \(M\) it overfits by chasing noise and oscillating wildly between data points.

As before, as \(M\) gets larger, we observe overfitting and a function that swings more wildly back and forth. Notice that the more squiggly and complex the function, the larger the magnitude of the weights learned.

In general, large weights are undesirable because they mean that one feature contributes a lot to the prediction. For example, in an image classification problem, a large weight would mean that one pixel significantly changes the prediction, which probably doesn’t make sense for most problems. Usually, a good prediction takes into account information from many different features. Therefore, large weights are often a sign of overfitting.

At the same time, the training error, which is the distance between the function and each of the data points, decreases as \(M\) gets larger. Since we have only ten data points, at \(M=9\) it is possible to learn a function with zero training error that passes cleanly through each point.

In the last section, we saw that we can mitigate this behaviour by tuning \(M\) as a hyperparameter. So, does this mean that the complexity of our models must always mirror the complexity of the underlying data distribution?

As it turns out, the answer is “no”. We can change the objective function, also called the cost function, in our optimization problem so that, instead of training only to minimize the error, we also perform regularization to prefer simpler models.

Definition: Regularization is a modification of the training procedure that prefers some models over others based on criteria other than the training error. Commonly, regularization is used to improve generalization by preferring simpler models that reduce overfitting.

In general, regularization can take many forms. You may have heard of ideas like dropout for neural networks, or data augmentation, which adds noise to the training data. Both of these are examples of regularization that you may learn more about in future courses. Here, we will introduce a simple regularization term that penalizes overfitting.

2 The Regularization Term

One possible modification to the training procedure is to add a penalty term \(\mathcal{R}\) to the cost function, where \(\mathcal{R}\) is some measure of model complexity. Thus, a combination of the training cost \(\mathcal{E}\) and model complexity \(\mathcal{R}\) is minimized. The new cost function \(\mathcal{E}_{reg}(\mathbf{w})\) is:

\[\begin{align} \mathcal{E}_{reg}(\mathbf{w}) = \mathcal{E}(\mathbf{w}) + \lambda \mathcal{R}(\mathbf{w}) \end{align}\]

where:

  • \(\mathbf{w}\) is the vector of weights
  • \(\mathcal{E}(\mathbf{w})\) is the original MSE
  • \(\mathcal{R}(\mathbf{w})\) is a regularization function, also called a regularizer
  • \(\lambda\) is a hyperparameter that sets the relative importance of \(\mathcal{E}(\mathbf{w})\) and \(\mathcal{R}(\mathbf{w})\)

Definition: The regularization function or regularizer \(\mathcal{R}(\mathbf{w})\) is a penalty applied directly to the cost function to encode a preference for some models over others. When using a regularizer, the optimization target for regression will still be the model with the overall lowest cost \(\mathcal{E}_{reg}(\mathbf{w})\), but cost no longer refers only to MSE.

As \(M\) gets larger, we know that the training MSE will inevitably get smaller, so we need the regularization term to grow when overfitting occurs. Using the observation that weights tend to take larger values when overfitting, we can write:

\[\begin{align} \mathcal{R}(\mathbf{w}) = \frac{1}{2} \sum_{j=1}^{D} w_j^2 = \frac{1}{2} \mathbf{w}^{\top} \mathbf{w} = \frac{1}{2} \|\mathbf{w}\|^2_2 \end{align}\]

where \(D\) is the number of features, i.e., the number of entries in \(\mathbf{w}\). For our polynomial example, \(D = M\). Note that the bias, \(b\), is excluded from the regularizer here, as is often done in practice. To see why this might be desirable, think about the role of \(b\) in the equation \(y = wx + b\).

This term is called the L2 regularization term or the L2 regularizer. The L2 regularizer is not the only possible regularization term for penalizing large weight values, but it is a popular choice.

With this term, we can use a fixed, large value of \(M\) and instead tune \(\lambda\) as our hyperparameter. When \(\lambda\) is too small, the MSE term \(\mathcal{E}(\mathbf{w})\) dominates, and the regularizer may not do enough to prevent overfitting. When \(\lambda\) is too large, the regularizer dominates, which may result in underfitting the data instead. So, we must tune \(\lambda\) to strike a balance between the two terms.

Trading one hyperparameter, \(M\), for another, \(\lambda\), in this way may not seem like an improvement, and, depending on the problem, sometimes it may not be. However, doing so allows us to decouple the complexity of our model from the complexity of the underlying problem. This is an extremely important idea. Practitioners often favour this approach: opt for a large model that will definitely overfit, and then manage the overfitting with regularization.

The figure below adds an L2 regularization term to the setup of the figure above. Try experimenting with different values of \(M\) and \(\lambda\). Can you avoid overfitting even for large \(M\)?

L2-regularized polynomial regression

Learned weights: b = …



Figure 2: The gray curve is the underlying function \(\sin(x)\); blue dots are training points sampled from it with added noise. The pink curve is the degree-\(M\) polynomial fit to those training points. The weights learned, \(\mathbf{w}\), are displayed to the right of the plot. Drag the top slider to change \(M\): at low \(M\) the model underfits because it is too rigid to follow the data; at high \(M\) it overfits by chasing noise and oscillating wildly between data points. Drag the bottom slider to change \(\lambda\): at low \(\lambda\) the regularizer has little effect; at high \(\lambda\) the regularizer may cause underfitting.

3 Summary

Regularization is a training modification that allows us to optimize models for more than just minimizing the training error. As we saw in this section, an L2 regularizer can be used to prevent overfitting even when we are performing regression with a polynomial that is much more complex than the underlying generating function. But regularization as a concept is more general than this.

Like many other ideas in ML, regularizers are modular and can be applied to many different kinds of cost functions. They can also be used to correct for problems other than overfitting. For example, imagine how a regularizer might be used to enforce fairness. This ties into Fundamental Idea #1: Learning is Optimization. With regularization, we can construct an optimization objective that attempts to prioritize the things we most care about.