Unsupervised Learning
1 Introduction
So far, every method we have studied predicts a target variable from a set of input features. In classification, the target is a categorical class label, such as oak, birch, or maple. In regression, the target is a continuous value, such as the temperature of the UTM pond in Celsius. In both cases, we trained on a labelled dataset, where every point comes with its known target value.
But what if our data has no labels? Perhaps we never recorded the target values, or perhaps there is nothing in particular we want to predict. Can we still do useful machine learning? And what would that even mean?
Yes, we can. This is called unsupervised learning. Instead of learning from labelled data, unsupervised learning works with unlabelled data. Rather than learning to predict a target, unsupervised learning looks for structure in the data itself. In this course, we will look at two kinds of unsupervised learning:
Clustering asks whether the data form groups, where points in the same group are more similar to each other than to points in other groups. We cover clustering in this chapter.
Dimensionality reduction asks whether we can describe the data using fewer dimensions without losing much useful information. This works when the points lie near a lower-dimensional subspace or curved surface inside the original space. We cover one such method, Principal Component Analysis (PCA), in the next chapter.
Either way, we are trying to learn something useful about how the data is structured. Clustering gives us a few typical examples that summarize the different kinds of points in a varied dataset. Dimensionality reduction describes each point using fewer dimensions, chosen so that we keep as much of the useful information in the data as possible.
2 Clustering
In clustering, we want to identify collections of points that are more similar to each other than they are to other points, where similarity can be taken to mean closeness in the data space. Although it will be convenient to work in two-dimensional space in this section for ease of visualization, the clustering methods we discuss may be applied to data in any number of dimensions.
Consider the following collection of points, which represent leaves whose species we do not know:
Unlabelled Leaf Data
Perhaps you can already see some structure in Figure 1. If you grouped these points into clusters based on how close they are to each other, how many clusters would you make? And roughly where would the centre of each cluster be?
In Figure 2 and Figure 3 below, we give two possible clusterings of this data, for two and three clusters. The centroid of each cluster is drawn with a bold black outline. The centroid is not an actual data point in the dataset. In these figures, each centroid is approximately the average of the points in its cluster. We will see exactly how it is computed later in this chapter.
Unlabelled Leaf Data - 2 Clusters
Unlabelled Leaf Data - 3 Clusters
Clustering finds groups in the data, but it does not tell us what those groups mean. We found both clusterings above by fitting a Gaussian Mixture Model, which we discuss later in this chapter. We might hope that the two clusterings show the data at different levels of detail, such as individual tree species versus tree families. In general, though, there is no guarantee that the clusters are meaningful. Perhaps the leaves were simply collected in two or three different seasons, or perhaps the clusters have no clear interpretation at all.
Even when we cannot interpret the clusters, they can still be useful. Points in the same cluster tend to be more similar to each other than to points in other clusters, so what we learn about a few points in a cluster is likely to hold for the rest. For example, we might test a sample of leaves from each cluster for disease and use the results to estimate how common the disease is in that cluster. Then, when we find a new leaf that is not in our dataset, we might check which cluster it belongs to and use that cluster’s estimate to decide whether to test it.
3 Summary
We have introduced the notion of unsupervised learning, which, unlike the supervised learning we have done so far, does not require labelled training data, and also the notion of clustering, a type of unsupervised learning that finds clusters of similar data points.
For the remainder of this chapter, we will examine two related methods for producing clusterings: K-means and Gaussian Mixture Models (GMMs).