Supervised Learning
Learning Objectives
After reading this page, you should be able to:
- Distinguish a regression task from a classification task.
- Formulate a problem as a supervised learning problem in formal notation.
- Distinguish learning from inference.
- Compute a classifier’s accuracy or error rate on a given data set.
- Explain the roles of the training, validation, and test sets in the context of data leakage.
1 Introduction
In supervised learning, our task is to train a model (sometimes called a predictor) to make a prediction given some input data. The following are possible examples of supervised learning tasks:
Given weather data from the most recent 7 days, predict tomorrow’s highest temperature in Celsius.
Given image data from a lung x-ray, predict the likelihood of lung cancer.
Given the width and height of a leaf in centimetres, predict whether that leaf came from an oak or a maple tree. (We will use this as a running example throughout this section.)
The predictor is a function that maps inputs to an output, and training is the process by which a “good” predictor function is selected (where “good” will be defined more precisely shortly). Crucially, the model designers have labelled input-output pairs that we call the training data, which can be used to train a model. This is what makes supervised learning supervised. During training, the precise way in which a model learns from this training data depends on the kind of model. (You will learn about many different kinds of models in this course!)
In our leaf example, our training data consists of the measurements of several leaves, along with ground-truth labels for whether the leaf is that of an oak (
) or maple (
). Here are some of the labeled data points that make up our training data:
| # | Leaf Width | Leaf Height | Leaf Type |
|---|---|---|---|
| 1 | 7.0 cm | 12.0 cm | |
| 2 | 9.0 cm | 6.0 cm | |
| 3 | 10.0 cm | 18.0 cm | |
| 4 | 5.0 cm | 10.0 cm | |
| 5 | 13.0 cm | 11.0 cm | |
| 6 | 11.0 cm | 9.0 cm | |
| 7 | 9.0 cm | 14.0 cm | |
Once we have trained a model using this data, we can use it to predict the label of previously unseen, unlabeled data. (I.e. Given the width and height of a newly found leaf, our model can predict its species.)
Here is an example of an unlabeled data point whose label we wish to predict:
| Leaf Width | Leaf Height | Leaf Type |
|---|---|---|
| 12.0 cm | 16.0 cm | ? |
It is important for the function to be able to generalize and produce good predictions for the output of new data. That is, we desire a model that generalizes to leaves that are not in the training data.
Question: In the table above, there is one unlabelled data point. What would your (human) prediction be of whether the leaf is an Oak or a Maple? What explanation can you provide for your prediction?
In this unit, we will discuss how to conceptualize, visualize, and evaluate supervised learning models. Unlike future chapters, we will not discuss any specific techniques for building supervised learning models; that will wait until the next chapter. Instead, this chapter will serve as a gentle introduction to ideas discussed throughout these notes, and will hopefully build geometric intuition along the way.
We begin by exploring various examples of supervised learning tasks, distinguishing between regression and classification problems. We then introduce the mathematical notation for formulating supervised learning problems, including how to represent inputs, outputs, and training data. Next, we describe the supervised learning workflow: the process of learning (training) a model from data and using it for inference (making predictions). We then turn to model evaluation, discussing how to visualize classifiers using decision boundaries, measure model performance using accuracy and error rates, and properly evaluate models using training, validation, and test sets. We examine important considerations when splitting data for training, validation, and testing, including how to avoid data leakage. Finally, we discuss how the quality and representativeness of data, as well as the choice of features, impact model performance and fairness.
2 Examples of Supervised Learning Tasks
There are many more examples of learning tasks that can be framed as a supervised learning problem. Let’s think carefully about our data for a few of these examples:
Predicting a person’s height based on their shoe size
Input: A person’s shoe size
Output: The person’s height
Training Data: As training data, we would need to know many people’s shoe sizes and heights, perhaps collected through a survey.Predicting the age of a person given a social media message that they posted
Input: Text from a social media message
Output: The person’s age in years
Training Data: We would need a collection of social media messages, along with the ground-truth age of the persons who wrote them. As an example, this data may be collected through a social media company.Image captioning
Input: An image
Output: An appropriate text caption for the image
Training Data: We would need a collection of captioned images, perhaps collected on the internet.Resume Filtering
Input: A job applicant’s resume
Output: Whether this applicant should be brought in for an interview
Training Data: Perhaps a company uses historical data from applicants, along with whether these applicants were successfully hired.
Note: We shall see that building such a model using such data can be problematic, and cause the model to be biased against historically underrepresented groups.Hand-written digit classification
Input: A 28×28 pixel greyscale image of a hand-written digit
Output: The digit drawn in the image (e.g., 0, 1, 2, … 9)
Training Data: There is a popular dataset called MNIST that we could use to train such a model.
From this list of examples, you might notice that there is great variety in supervised learning problems. There are variations in the kind of input types (numbers, images, text, …) and output types (numbers, categories, text, …), as well as the sources of data that bring complexity to the way that a model should be used in practice. Moreover, using machine learning techniques for these tasks can impact people, both individually and at the societal level. We will discuss some of these important issues in later units.
Formulating a learning problem as a supervised learning problem is not always an easy task. For example, in the “age prediction from social media message” example, perhaps the ultimate goal is to determine whether the message sender is a child, in which case the exact age is not required: only an age range. In this same problem, it may be useful to include as input not just the message text, but also other metadata information like the time that a message was posted (e.g., was it during school time?). These concerns are not technical machine learning decisions, but understanding how your model will be used and the data that is available to you can impact the utility of your model.
3 Regression vs Classification
We call attention to differences in output types among the above examples because the type of quantity we require our model to predict defines the kind of supervised learning task:
We call a supervised learning task a regression task if the output to be predicted is a continuous number. For example, predicting a person’s height (in cm) is an example of a regression task, as is predicting the temperature (in Celsius) tomorrow.
We call a supervised learning task a classification task if the output to be predicted is a finite discrete category. For example, predicting the digit in an MNIST image is a classification task, as is predicting whether a movie review is positive or negative.
There are also cases where the output to be predicted is more complex and highly structured. For example, in the image captioning task, the output is a piece of text, which is a sequence that could be long or short. In these cases, the task is neither a regression or classification task.
Definition: A regression task is a supervised learning task where the output is continuous. A classification task is a supervised learning task where the output is a finite discrete category.
Notice that the kind of task is determined solely by the output. Input types can be continuous, discrete, or a mixture for both regression and classification tasks. As you will see in the following chapters, whether a task is a regression or classification determines what kinds of models and training strategies are applicable.
Question: How can the age prediction task from above be framed as a regression task? A classification task?
4 Formulating a Supervised Learning Problem
This section will summarize the notation we will use for all supervised learning tasks in these notes.
4.1 Representing Inputs
There is also variety in the types of inputs that a model designer might desire, including numbers, images, text, sound, and others. Although input formats can differ, we can always represent an input using several numerical features that we combine into an input vector. In this text, we will represent such an input vector using the vector \(\mathbf{x}\).
Definition: The notation \(\mathbf{x} \in \mathbb{R}^D\) denotes an input vector, where each \(x_i \in \mathbb{R}, i = 1, \dots, D\) denotes an input feature (or feature) and \(D\) is the number of features.
\[ \mathbf{x} = \begin{bmatrix}x_1 \\ x_2 \\ \vdots \\ x_D\end{bmatrix} \in \mathbb{R}^D \]
For example, we can represent a leaf with a width of 7cm and a height of 12cm using the vector \(\mathbf{x} = \begin{bmatrix} 7.0 \\ 12.0 \end{bmatrix}\). Here, \(x_1 = 7.0\) represents the leaf width, and \(x_2 = 12.0\) represents the leaf height. The order of the features is arbitrary. We could have easily chosen \(x_1\) to represent the leaf height and \(x_2\) the leaf width. It is only important that we are consistent in our feature ordering across a task.
When working with several input vectors, we may wish to label each vector with an index to identify specific items. In this text, we will place the index in the superscript position, and wrap the index in brackets to differentiate indexes from exponentiations. Thus, the input vectors from the leaf prediction dataset in Table 1 are
\[\mathbf{x}^{(1)} = \begin{bmatrix} 7.0 \\ 12.0 \end{bmatrix}, \mathbf{x}^{(2)} = \begin{bmatrix} 9.0 \\ 6.0 \end{bmatrix}, \mathbf{x}^{(3)} = \begin{bmatrix} 10.0 \\ 18.0 \end{bmatrix}, \dots, \mathbf{x}^{(7)} = \begin{bmatrix} 9.0 \\ 14.0 \end{bmatrix}.\]
Definition: The \(i\)-th input vector in a data set is denoted \(\mathbf{x}^{(i)}\), where the index \(i\) is placed in brackets in the superscript position.
Question: What does the notation \(\mathbf{x}^{(2)}_1\) represent? What is the value of \(\mathbf{x}^{(2)}_1\) in the dataset above?
Answer:
\(\mathbf{x}^{(2)}_1\) represents the 1st feature of the 2nd input vector.
From the dataset,
\[\mathbf{x}^{(2)}=\begin{bmatrix}9.0\\6.0\end{bmatrix} \quad\Rightarrow\quad \mathbf{x}^{(2)}_1=9.0 \text{ (cm, leaf width).} \]
There is a reason why we begin with a data set with \(D=2\) features: it is possible to visualize the data space on a computer or phone screen! The data space is a representation of all possible data \(\mathbf{x} \in \mathbb{R}^D\) that could arise. In theory, \(\mathbb{R}^D\) extends infinitely along each feature dimension, but in reality the data we collect only occupies a small region within this vast space. In the figure below, we choose to visualize the leaf prediction data by plotting the leaf width measure on the horizontal axis, and the leaf height measure on the vertical axis. Each of the 7 circles represent a training data point, and the colour and shape of the circle represent the leaf type (oak or maple). This geometric perspective, representing data as points in \(\mathbb{R}^D\) space, illustrates Fundamental Idea #4: ML Describes Geometric Processes. By thinking of data points geometrically, we can use concepts like distance, regions, and boundaries to understand how models work.
Interactive scatter plot of the leaf prediction dataset in 2D feature space
It is important to note that each point in the data space, even those not marked by a training data point, represents a possible leaf measurement!
In practice, machine learning model designers tend to work with many features, so \(D\) is generally much larger than 2. In such cases, it is infeasible (and unnecessary) to visualize the data space. However, the intuition that you develop by studying learning problems where \(D=2\) can translate to these bigger problems.
But how do we represent images and text?
It is straightforward to represent width and height measurements into a vector \(\mathbf{x}\), but how can we represent more complex inputs into such vectors? Recall the MNIST dataset we mentioned earlier, where each example is a \(28 \times 28\) grayscale image of a handwritten digit. How can we turn such an image into a vector?
It turns out that in order to display images, your computer already represents them using numbers! Your computer already stores, for each pixel in an image, the amount of light intensity at that pixel, ranging from 0-255.
For an MNIST image, since the image is greyscale and there are \(28 \times 28\) pixels, your computer stores 784 pixel intensities to display this image.
MNIST handwritten digit “3” (left) and its corresponding \(28 \times 28\) matrix of pixel intensity values shown as a Python list of lists (right)
Mathematically, we can represent each image using a vector with \(D=784\) features, where feature \(j\) is the intensity at the pixel \(j\). We can also think of this as a \(28 \times 28\) matrix, where each entry corresponds to the pixel intensity at that position in the image.
Interactive MNIST handwritten digit “3” displayed as a \(28 \times 28\) pixel grid with enlarged pixels to show individual values
For coloured images, the idea is similar, but your computer stores three measurements per pixel: the amount of light intensity for red, blue, and green light. These additional measurements just mean that additional features are required. If the MNIST image above was a coloured image, we would have \(784 \times 3 = 2352\) input features.
For text data, the story becomes a bit more interesting. One approach is to engineer features from the text data. For example, we can compute numerically measurable features like the length of the text, the average number of words, whether the word “hello” appears in the text, etc. Depending on the desired output, some of these engineered features can be useful in the prediction task.
In fact, that last example uses what are called binary bag of word features. Here’s how it works: imagine you have a vocabulary of all possible words you care about (like “hello”, “goodbye”, “cat”, etc.). For each word \(j\) in your vocabulary, you create a feature \(x_j \in \{ 0, 1\}\), where \(x_j = 1\) if that word appears in your text, and \(x_j = 0\) if it doesn’t. So if your vocabulary has 1000 words, you end up with 1000 features, each being either 0 or 1. We will discuss this again in later chapters.
4.2 Representing Outputs
In this text, we represent the target output using the symbol \(t\), and the predicted output with the symbol \(y\). They differ in that \(t\) represents a ground-truth, actual output, whereas \(y\) is a prediction provided by a model. (Outside this text, you may see the symbol \(y\) to denote the ground-truth output, and \(\hat{y}\), pronounced “y-hat”, used to represent the predicted output. We have used \(t\) and \(y\) instead for notational clarity.)
Definition: The notation \(t\) denotes a target output and \(y\) denotes a predicted output.
In a regression problem, \(t\) is a real number, \(t \in \mathbb{R}\). In a classification problem, we can think of \(t\) as coming from a discrete set \(\{1, \ldots, C\}\). (We may wish to represent \(t\) differently in the upcoming chapters.)
4.3 Representing the Training Data
A training data set or a training set is a labelled data set used for the purpose of training a supervised learning model. Such a data set consists of \(N\) input-output pairs, which can be written \(\{(\mathbf{x}^{(1)}, t^{(1)}), (\mathbf{x}^{(2)}, t^{(2)}), ... (\mathbf{x}^{(N)}, t^{(N)}) \}\), where \(N\) is the size of the training set.
For example, if we use \(t=1\) to represent Oak and \(t=2\) to represent Maple, then the training data set from Table 1 can be written:
\[ \left\{ \left( \begin{bmatrix} 7.0 \\ 12.0 \end{bmatrix}, 1 \right), \left( \begin{bmatrix} 9.0 \\ 6.0 \end{bmatrix}, 1 \right), \left( \begin{bmatrix} 10.0 \\ 18.0 \end{bmatrix}, 1 \right), \dots, \left( \begin{bmatrix} 9.0 \\ 14.0 \end{bmatrix}, 2 \right) \right\} \]
Question: In the leaf prediction data above, what is the value of \(t^{(5)}\)?
5 Supervised Learning Workflow
We have been using the word model in a somewhat haphazard way. In the broadest sense, a model is a (simplified) representation of reality that helps us predict or explain a phenomenon. For example, a model for the leaf prediction problem could state: “If a leaf is wider than 10.5cm, then it is Maple”. Like all models, this model is a simplified description of the relationship between the input features and target output. Models do not accurately represent the nuances of reality: i.e., in reality, the leaf type depends on much more than these two features. However, models can still be very useful.
Machine learning practitioners sometimes use the term model to refer to a specific mapping between an input and output. Such mappings are also called a hypothesis or a predictor. We may use symbols such as \(f\) to indicate such a mapping, where \(f\) maps an input \(\mathbf{x}\) to an output prediction \(y\).
For example, for the leaf prediction problem, a predictor \(f_1\) might map all leaves less than 10.5cm wide to the prediction “Oak”, and others to “Maple”. This predictor can also be written: \[ f_1 \left(\begin{bmatrix} x_1 \\ x_2 \end{bmatrix} \right) = \begin{cases} \text{ Oak } & \text{if } x_1 < 10.5 \\ \text{ Maple } & \text{otherwise.} \end{cases}\] Since the leaf prediction task is a classification task, we call \(f_1\) a classifier.
Definition: A hypothesis, predictor (or sometimes model) \(f\) is a specific mapping from an input \(\mathbf{x}\) to an output prediction \(y = f(\mathbf{x})\). In a regression problem, a predictor is also called a regressor. In a classification problem, a predictor is also called a classifier.
Other times, the term model really refers to a model family, which are a class of hypotheses with the same structure. For example, for the leaf prediction problem, one family of models we might consider are those that make a binary decision based on the leaf width, i.e. models of the form: \[ f \left(\begin{bmatrix} x_1 \\ x_2 \end{bmatrix} \right) = \begin{cases} \text{ Oak } & \text{if } x_1 < r \\ \text{ Maple } & \text{otherwise} \end{cases} \] for some threshold \(r \in \mathbb{R}\). Other models families we will study include decision trees, linear regression, neural networks, and others.
Definition: A model family is a class/group of hypotheses with the same structure.
Each of the next few units will study a new model family, along with a method for learning a hypothesis. For example, learning a model for above means selecting a value of \(r\) that “works well” for the training data.
Much of this course will be about how to learn a model \(f\) from a labelled training dataset. For now, the important idea to emphasize is that learning involves using the training data. That is, a computer program that learns a model would need to be provided the training data, and should produce a hypothesis \(f\) that maps an input \(\mathbf{x}\) to an output \(t\).
Definition: Learning is the process of choosing a hypothesis \(f\) using the training data \(\mathcal{D}\). Learning is done using a learning algorithm \(\mathcal{A}\). Learning algorithms typically involve an optimization procedure of some kind, as alluded to in Idea #1 in the introduction.
A learning algorithm takes in training data and produces a hypothesis (model)
Once we have a specific hypothesis in mind, e.g., through a learning algorithm, or even through randomly choosing a hypothesis, we can use the hypothesis to perform inference. That is, we can use the hypothesis to make actual predictions. You can think of inference as using the model.
Definition: Inference is the process of using a hypothesis \(f\) to make predictions. We obtain a prediction for an input \({\bf x}\) by using the model \(f\) to compute \(y=f({\bf x})\)
A hypothesis (model) takes an input and produces an output
These two terms, learning and inference represent the phases of (1) developing and (2) using a supervised learning model.
6 Model Evaluation
In many units, including this one, we will first discuss how to perform inference for models from a certain model family. Only once we understand inference will we discuss how learning is done. Typically, there will be an optimization problem of some sort, where many different hypotheses are considered and evaluated.
Thus, it is important to discuss model evaluation: i.e., determining which hypothesis is “better” than another, which hypothesis to choose, and how to choose between different model families.
6.1 Visualizing a Classifier: The Decision Boundary
A classifier can be represented in different ways. For problems like the leaf prediction problem where there are only \(D=2\) features, we can visualize a classifier in the data space by sketching the regions in the data space with predictions “Oak” vs “Maple”. We do so with two classifiers below:
Comparison of two different classifiers for the leaf prediction problem
When observing these classifiers, our eyes are naturally drawn to the boundaries between the two classes. This boundary is called the decision boundary. The decision boundary is important because knowing the decision boundary characterizes the classifier: if we know what the decision boundary is, then we know what prediction \(y\) the classifier would make on any input \({\bf x}\). By thinking of models as geometric transformations that partition space (Fundamental Idea #4), we gain intuition about how they work and when they might fail.
Definition: The decision boundary is a subset of the data space that separates the different classes. In other words, it is the set of points \(\mathbf{x}\) in the input space where perturbing (changing) the point by a tiny bit would change its corresponding predicted output.
Question: What prediction will each of the two classifiers produce for the input \({\bf x} = \begin{bmatrix}12\\16\end{bmatrix}\) from Table 1?
Answer:
Classifier \(f_1\) will predict Maple.
Classifier \(f_2\) will predict Oak.
For \(f_1\), the decision boundary is a line. For \(f_2\), the decision boundary is more complex and consists of a triangle. In some sense (which can eventually be formalized), \(f_2\) is more complex and has higher capacity than \(f_1\): the decision boundary for \(f_2\) is able to store more information about how the two classes differ. But which one is better? In order to answer that question, we need to consider how to measure how well each classifier performs across various data sets.
Question: Why might one think that \(f_1\) is a better classifier for this problem than \(f_2\)? Why might one think that \(f_2\) is a better classifier for this problem than \(f_1\)?
6.2 Accuracy and Error
For a classification task, one way to measure how well a predictor performs on a data set is by computing its accuracy rate: the rate at which the predictor produces the correct prediction. For example, across the 7 data points visualized above, \(f_1\) has an accuracy of \(\frac{6}{7}\). We can alternatively compute the error rate, which is the rate of incorrect predictions.
Definition: The accuracy rate (or accuracy) of a classifier over a data set is the proportion of correct predictions made by that classifier. That is, \[\text{accuracy} = \frac{\text{num correct predictions}}{\text{num total predictions made}} \\ = \frac{\sum_{i=1}^N \mathbb{I}[t^{(i)} = y^{(i)}]}{N},\] where \(\mathbb{I}[e]\) evaluates to \(1\) if the equation \(e\) holds true, and 0 otherwise. Likewise, the error rate of a classifier is the proportion of incorrect responses \[\text{error rate} = \frac{\text{num incorrect predictions}}{\text{num total predictions made}} = 1 - \text{accuracy}.\]
Question: Why is the accuracy rate defined for classifiers only, and not regressors?
Answer:
“Accuracy” relies on exact equality, and every incorrect prediction \(y^{(i)}\neq t^{(i)}\) is treated the same way. In regression, \(t, y \in\mathbb{R}\) are continuous. Equality almost never holds up, and wrong predictions could still be “close” or “far away”, so incorrect predictions should not always be treated the same.Answer:
There are 7 classifications, and all of them are correct, so the accuracy of the classifier \(f_2\) over these data points is \(\frac{7}{7}\).Notice that the accuracy is measured across a specific data set. Thus, it is incorrect to say that the model \(f_1\) has an accuracy of \(\frac{6}{7}\), since that accuracy is computed over a specific data set. Using a different data set would give us a different accuracy value for the same model.
Answer:
Not necessarily. A higher accuracy on these 7 points reflects in-sample fit and does not guarantee better generalization. The more complex boundary of \(f_2\) could overfit the sample.You may notice a problem with computing and comparing the model accuracies over a data set that a model is trained over. For example, \(f_2\) achieves perfect accuracy, but what if the three edges of the triangular decision boundary were chosen based on the three “Maple” data points? In that case, by construction the trained model will achieve 100% accuracy across the data that it is trained on!
However, \(f_2\) might not generalize to new data. In future sections, we will introduce the term overfitting, and worry that \(f_2\) may have overfit to its training set. In other words, the model may have captured rare patterns that just so happen to be in the training set, but do not appear in general in new data generated in the same way. (For example, the very long maple leaf with leaf height 14cm may have been such a fluke.) This intuition will help us understand Idea #2 from the introduction: there are tradeoffs in sources of error.
Thus, for model comparison and evaluation, it is unwise to use accuracy and other performance metrics computed over data that a model is trained on.
6.3 Training, Validation and Test Sets
In order to properly evaluate models, model designers need to compute accuracy (or other evaluation metrics) on a held-out set of labelled data that is not used for training. In situations where there is a large amount of labeled data, a common approach is to split the data set into three portions: a training set consisting of 60-80% of the labelled data, a validation set consisting of 10-20% of the labelled data, and a test set consisting of 10-20% of the labelled data. (Note that the meaning of “large” depends both on the specific problem and the kind of model being trained.)
The training set is used by the learning algorithm to select a hypothesis from a model family, as discussed earlier.
But why are there suddenly two held-out sets, rather than one? It is because there are two reasons why a model designer might wish to evaluate models.
The first reason is to evaluate and choose between different models. A model designer might wish to determine which of two models to use, which may or may not be from the same model family. In this situation, the model chosen will be the one that performs best on the held-out validation set. Since neither of the models used the validation set for training, the comparison would be “fair”.
When the validation set is used to choose between models from the same model family, it is often because they have different hyperparameter values. Hyperparameters are values that impact the model, but are held fixed during training. Finding appropriate hyperparameter values is an important job of the validation set, which is an idea that will be discussed in more detail in later chapters.
The second reason is to estimate how well a model will perform on new data once deployed. Here, even though seeing good performance is desirable, it is important to obtain an estimate that is unbiased: i.e., avoid over-estimating performance to the greatest extent possible. In order to obtain such an unbiased estimate, we cannot re-use the validation set! Once we make model decisions using a data set, then the model chosen will likely be one that performs better on that data set, compared to new data collected in the same manner. Thus, we use a held-out test set that has never been used to make any model decisions. Good model designers will avoid optimizing models using the test set. If they do so anyway, they should collect a new test set.
Definition: The training set is a portion of the labelled data set used to train models (i.e., choose a hypothesis from a model family using a learning algorithm). The validation set is a portion of the labelled data set used to choose between different models (this definition will be updated in the next unit). The test set is a portion of the labelled data set used to evaluate how well the model will generalize to new data collected in the same way.
Question: Would you generally expect a model’s validation accuracy (defined to be the accuracy over the validation set) to be larger or smaller than its training accuracy (defined to be the accuracy over the training set)?
Answer:
Training accuracy is computed on the data the model was fit to, so it can overfit the data, yielding higher accuracy. Validation accuracy is measured on held-out data and therefore reflects generalization; it is usually lower.To summarize, the typical supervised learning workflow involves the following steps:
- Split the labeled data into three parts: a training set, a validation set, and a test set.
- Train multiple models (or vary the hyperparameters) using only the training set.
- Evaluate each model on the validation set to compare their performance and select the best model or set of hyperparameters.
- Once the best model is chosen, estimate its generalization performance by evaluating it on the test set, which has not been used in any way during model selection or training.
The diagram below illustrates this process: the training data is used to produce several candidate models, each of which is evaluated on the validation set. The model with the best validation accuracy is then evaluated on the test set to estimate its performance on new, unseen data.
Supervised learning workflow showing the process of training multiple models and evaluating them
This empirical approach to model evaluation in a supervised learning workflow reflects Fundamental Idea #3: Learning is Empirical. Model designers need to try different approaches and measure their performance on real data to determine what works best for our specific problem.
7 Data Splitting and Data Leakage
The splitting of the data into training, validation, and test sets might sound straightforward. In many cases, a simple random selection of data points is sufficient. For example, if we have 1000 measurements of leaves in Table 1 with a roughly equal number of oak and maple, we can randomly choose 700 for our training set, 150 for our validation set, and 150 for our test set (800/100/100, or 600/200/200 could also be reasonable).
But what if some samples were collected from the same tree, and the tree identifiers are available in the data set? The same issue appears more clearly in the social media age-prediction task. What if the same author, with their many messages posted on the media platform, appears in the data multiple times? If some of their messages are in the training set, the model might learn something particular about that person (e.g., they always sign off with “xoxo”, and are 20 years old). Then, if some of their other messages are in the test set, the model might be able to make good predictions due to its specialization to the information in the training set (e.g., seeing “xoxo” in the message means the author is 20 years old). However, if this model is intended to be used to predict the ages of social media users not in the training set, the computed test accuracy will overestimate how well the model will perform once deployed.
As another example, when data is temporal in nature (e.g., weather measurements, stock prices), if we randomly split the data, our performance metrics would be artificially inflated: the weather pattern measurements today tend to be similar to those from yesterday, so if today’s data was in the training set and yesterday’s data was in the test set, then the model may perform well on the test set simply because it has already seen very similar data in training. This does not reflect how the model would perform on truly unseen future data. Thus, data splits for time series or temporally ordered data should be performed such that all training data precedes all validation and test data in time; for example, use data from time \(t_0\) to \(t_k\) for training, and data from \(t_{k+1}\) onward for validation and testing.
As a further example, consider data that is naturally represented as a graph, such as a social network (e.g., Instagram friendships). In these cases, each data point might correspond to a user, and features could include information about the user as well as their connections to others. If we randomly split users into training and test sets, there is a risk that information about a user in the test set “leaks” into the training set through their friends: for instance, if the model learns patterns about a user’s friends in training, it may perform unrealistically well when predicting for that user in the test set, since their friends’ data was already seen. This is especially problematic for tasks like node classification or link prediction, where the relationships themselves are the subject of prediction. To avoid this kind of leakage, special care must be taken when splitting graph data, e.g., by using more sophisticated graph partitioning strategies.
These examples highlight the importance of understanding the structure of your data and designing your splits to prevent leakage, ensuring that your test set truly reflects the model’s performance on new, unseen data.
More formally, supervised learning models make the underlying assumption that the labeled data points are independent, identically distributed: that each data point \(\mathbf{x}\) can be thought of as being sampled from the same underlying data sampling distribution \(p_{\mathbf{x}}\), and that the data points are not correlated with one another. As we saw, these assumptions are often untrue in practice.
Thus, an empirical approach to model evaluation needs to be multifaceted, and consider more than just the accuracy number that is reported by ML software.
8 Data Matters and Feature Matters
We will also see, in later sections, that the data that we collect matters. Data should be representative of what will be seen in practical application. This is difficult. For example, Buolamwini and Gebru (2018) found that commercial facial recognition systems had significantly higher error rates for darker-skinned individuals, particularly women, with error rates up to 34% for darker-skinned women compared to less than 1% for lighter-skinned men. This occurred because the training datasets were not representative: they contained predominantly lighter-skinned faces. As another example, Zech et al. (2018), showed that a deep learning model trained to detect pneumonia from chest X-rays learned to exploit hospital-specific visual markers (such as differences in imaging equipment or patient positioning protocols) rather than true disease patterns. When tested on data from different hospitals, the model’s performance degraded significantly because the spurious correlations it learned did not generalize. This highlights how non-representative training data can lead models to learn the wrong patterns entirely, and that empirical evaluation needs to consider many factors.
More generally, the features we use matter. In the leaf prediction example, we used the width and height of the leaf. Maybe there is a simpler relationship between the surface area of the leaf (which can be easily estimated) and the leaf type. Maybe collecting other features, like the weight of the leaf or the colour of the leaf, could also make the problem “easier”. Feature Engineering is a term used to describe the process of designing and testing features to use for a machine learning task. The choice of features determines the geometric structure of the data space (Fundamental Idea #4), and different feature representations can make patterns easier or harder to discover. A good feature representation can transform a difficult learning problem into an easier one.
Of course, collecting data can be an expensive, time-consuming process. In certain cases, synthetic data can and have been used to train and improve machine learning models. Simpler Data augmentation strategies are also commonly used in machine learning: e.g., perturbing/changing the training data by a little bit to create new training data; for example in the MNIST example, we can shift an image 1 pixel to the left/right/up/down, rotate the image by a small angle, or add random noise to the images to change the pixel intensities by a little bit. This practice of adding controlled noise to training data relates to Fundamental Idea #5: Machine Learning Describes Probabilistic Processes. By explicitly modeling the uncertainty and variability in real-world data through augmentation, we can build models that are more robust to the natural randomness they will encounter, and think of randomness as a tool rather than an obstacle.
Even with the perfect data and the perfect features, supervised learning tasks are often such that models rarely achieve 100% test accuracy. Thus, it is imperative to evaluate the kind of mistakes that a model makes. Who will be affected, and how? Are there groups that will be disproportionately affected by mistakes? What systems will be in place to ensure that models do good, and prevent harm? These are all important questions for practitioners to consider. Again, this emphasis on understanding model behavior and its impacts reflects Fundamental Idea #3: Learning is Empirical. We cannot know how a model will perform or who it will affect without careful empirical evaluation on realistic data.