Univariate Gaussian Discriminant Analysis
1 Introduction
Thus far, we have discussed generative classifiers. Instead of modeling the class given the input directly (as in discriminative models like logistic regression), we model the input given the class and then apply Bayes’ rule to obtain the probability of the class given the input. In Naive Bayes, we assumed binary features and used a Bernoulli distribution for each feature conditioned on the class, with a conditional independence assumption across features.
But what happens when our features are continuous? We are still in a classification setting. We have labeled data and want to predict the class of a new input. We can again adopt a generative view and model the distribution of the input given the class with a simple, tractable family of distributions.
Definition: If we assume that each class-conditional distribution is a Gaussian or Normal distribution, then we obtain a Gaussian Discriminant Analysis (GDA) model.
Essentially, we fit a Gaussian per class from data (e.g., by maximum likelihood), then use Bayes’ rule to classify new points.
This section begins by building intuition about GDA using univariate Gaussians and a single input feature, so we can see how learning the distribution and performing inference work in one dimension.
In the next section, we will generalize to the multivariate case with many input features, where each class has its own mean vector and covariance matrix. Multivariate GDA also requires considerations about the covariance between the different input features, and is thus slightly more complex. The next section will also consider variants that impose additional structure to the covariance structure, such as Gaussian Naive Bayes, where features are assumed conditionally independent given the class. We will also compare GDA with familiar discriminative models like logistic regression.
2 Review: Univariate Gaussian Distributions
We begin by reviewing the familiar one-dimensional (univariate) Gaussian (or normal) distribution. This is a probability distribution over a real-valued random variable \(x \in \mathbb{R}\), parameterized by a mean \(\mu\) and variance \(\sigma^2\). The probability density function of this distribution is written:
\[ \begin{aligned} & p(x \,;\, \mu, \sigma^2) = \mathcal{N}(x \,;\, \mu, \sigma^2) \\ & = \frac{1}{\sqrt{2\pi}\,\sigma}\exp\left(-\frac{(x-\mu)^2}{2\sigma^2}\right), \end{aligned} \]
where \(\mu\) (the mean) controls where the distribution is centered along the \(x\)-axis, and \(\sigma^2\) (the variance) controls how spread out the distribution is. The standard deviation is often easier to interpret. It is
\[ \begin{aligned} & \sigma = \sqrt{\sigma^2}. \end{aligned} \]
A larger \(\sigma\) means a wider, flatter bell curve, and a smaller \(\sigma\) means a narrower, taller bell curve.
Interactive univariate Gaussian
2.1 Learning a Univariate Gaussian
A probabilistic distribution is often a useful summary of a dataset and its underlying phenomenon. For example, suppose we once again collect the width measurements of many oak leaves. Although the widths of different oak leaves will vary from tree to tree and from leaf to leaf, they tend to cluster around some “typical” width, with fewer leaves that are extremely narrow or extremely wide. If this variability is well-approximated by a Gaussian, then learning the parameters \((\mu, \sigma^2)\) from data gives us a compact summary of the leaf-width distribution and lets us answer questions like “How likely is it to see a leaf wider than 10cm?” or “What range of widths covers most oak leaves?”
To that end, let’s consider how we can learn the parameters of a Gaussian distribution from data. We will imagine observing a collection of samples \(x^{(1)}, x^{(2)}, \dots, x^{(N)}\). For example, take the widths (in cm) of the oak leaves from the leaf dataset we used in the supervised learning chapter, which gives \(N=4\) samples:
\[ \begin{aligned} & x^{(1)}=7.0, \quad x^{(2)}=9.0, \quad x^{(3)}=10.0, \quad x^{(4)}=5.0. \end{aligned} \]
We will estimate \(\mu\) and \(\sigma^2\) using the same maximum likelihood estimation (MLE) criterion we saw earlier. We choose the parameters \(\mu, \sigma\) that make the observed data most likely under the model.
Assume the samples \(x^{(1)}, \dots, x^{(N)}\) are drawn independently from a Gaussian with mean \(\mu\) and variance \(\sigma^2\). We write \(\mathcal{N}(x \,;\, \mu, \sigma^2)\) for the Gaussian density \(p(x \,;\, \mu, \sigma^2)\). The likelihood of the parameters given the data is the probability of the data under the model:
\[ \begin{aligned} & L(\mu, \sigma^2) = \prod_{i=1}^N \mathcal{N}(x^{(i)} \,;\, \mu, \sigma^2) \\ & = \prod_{i=1}^N \frac{1}{\sqrt{2\pi}\,\sigma}\exp\left(-\frac{(x^{(i)}-\mu)^2}{2\sigma^2}\right). \end{aligned} \]
We maximize this over \(\mu\) and \(\sigma^2\). It is easier to maximize a sum rather than product, so we take the log-likelihood
\[ \begin{aligned} & \ell(\mu, \sigma^2) = \log L(\mu, \sigma^2) \\ & = \sum_{i=1}^N \left[ -\frac{1}{2}\log(2\pi) - \log \sigma - \frac{(x^{(i)}-\mu)^2}{2\sigma^2} \right] \\ & = -\frac{N}{2}\log(2\pi) - N\log\sigma - \frac{1}{2\sigma^2}\sum_{i=1}^N (x^{(i)}-\mu)^2. \end{aligned} \]
Question: Differentiate \(\ell\) with respect to \(\mu\), set the derivative to zero, and solve for the maximum-likelihood estimate \(\mu_{\textrm{MLE}}\).
Answer:
To maximize \(\ell\) with respect to \(\mu\), we differentiate \(\ell\) with respect to \(\mu\) and set to zero. The only term that depends on \(\mu\) is
\[ \begin{aligned} & -\frac{1}{2\sigma^2}\sum_i (x^{(i)}-\mu)^2. \end{aligned} \]
Using the derivative
\[ \begin{aligned} & \frac{\partial}{\partial\mu}(x^{(i)}-\mu)^2 = -2(x^{(i)}-\mu), \end{aligned} \]
we get
\[ \begin{aligned} & \frac{\partial \ell}{\partial \mu} = \frac{1}{\sigma^2}\sum_{i=1}^N (x^{(i)}-\mu) = 0. \end{aligned} \]
Rearranging, we have
\[ \begin{aligned} & \sum_{i=1}^N x^{(i)} = N\mu, \end{aligned} \]
which gives us
\[ \begin{aligned} & \mu_{\textrm{MLE}} = \frac{1}{N}\sum_{i=1}^N x^{(i)}. \end{aligned} \]
So the maximum-likelihood estimate of the mean is the sample mean of the data.
To maximize \(\ell\) with respect to \(\sigma\), we differentiate \(\ell\) with respect to \(\sigma\) and set to zero. We again isolate terms involving \(\sigma\):
\[ \begin{aligned} & \frac{\partial \ell}{\partial \sigma} = \frac{\partial}{\partial \sigma} \left(- N\log\sigma - \frac{1}{2\sigma^2}\sum_{i=1}^N (x^{(i)}-\mu)^2 \right) \\ & = -\frac{N}{\sigma} + \frac{1}{\sigma^3}\sum_{i=1}^N (x^{(i)}-\mu)^2 \\ & = -\frac{1}{\sigma}\left(N - \frac{1}{\sigma^2}\sum_{i=1}^N (x^{(i)}-\mu)^2 \right) = 0 \end{aligned} \]
Solving, and plugging in the MLE for \(\mu\) so the estimate is consistent, we obtain
\[ \begin{aligned} & \sigma^2_{\textrm{MLE}} = \frac{1}{N}\sum_{i=1}^N (x^{(i)} - \mu_{\textrm{MLE}})^2. \end{aligned} \]
So the maximum-likelihood estimate of the variance is the sample variance (average squared deviation from the sample mean).
Question: For the oak leaf widths \(7.0, 9.0, 10.0, 5.0\) (with \(N=4\)), what are the maximum-likelihood estimates \(\mu_{\textrm{MLE}}\) and \(\sigma^2_{\textrm{MLE}}\) of the Gaussian distribution that models these data?
Answer:
We plug the data into the MLE formulas derived above. First compute the sample mean:
\[ \begin{aligned} & \mu_{\textrm{MLE}} = \frac{1}{N}\sum_{i=1}^N x^{(i)} \\ & = \frac{7.0 + 9.0 + 10.0 + 5.0}{4} \\ & = \frac{31}{4} = 7.75 \text{ cm.} \end{aligned} \]
Next compute the sample variance:
\[ \begin{aligned} & \sigma^2_{\textrm{MLE}} = \frac{1}{N}\sum_{i=1}^N (x^{(i)} - \mu_{\textrm{MLE}})^2 \\ & = \frac{1}{4}\big[(7.0-7.75)^2 + (9.0-7.75)^2 + (10.0-7.75)^2 + (5.0-7.75)^2\big]. \end{aligned} \]
Evaluating the squared deviations gives
\[ \begin{aligned} & \sigma^2_{\textrm{MLE}} = \frac{1}{4}(0.5625 + 1.5625 + 5.0625 + 7.5625) \\ & = 3.6875 \text{ cm}^2, \end{aligned} \]
so \(\sigma_{\textrm{MLE}} \approx 1.92\) cm. The learned Gaussian is centered at 7.75 cm with a standard deviation of about 1.92 cm, summarizing the spread of oak leaf widths in the data.
3 The Univariate GDA Model
With the univariate Gaussian and its MLE in hand, we can now define a univariate Gaussian Discriminant Analysis model for binary classification. Our model will involve a single continuous feature \(x\). As a running example, we use the same leaf dataset as before, but now we have two classes (oak vs maple) and a single continuous feature, leaf width \(x \in \mathbb{R}\). The class label is \(c \in \{0, 1\}\) (e.g., \(c=0\) for oak, \(c=1\) for maple). Our observations are the training set
\[ \begin{aligned} & \mathcal{D} = \{(x^{(1)}, c^{(1)}), (x^{(2)}, c^{(2)}), \dots, (x^{(N)}, c^{(N)})\}. \end{aligned} \]
A GDA model is a generative model of the joint distribution of the feature and the class. We write
\[ \begin{aligned} & p(x, c) = p(x \mid c)\,p(c). \end{aligned} \]
The “Gaussian” part of GDA means that we assume that given the class \(c\), the feature \(x\) is drawn from a Gaussian distribution. So for each class \(k \in \{0, 1\}\) we have
\[ \begin{aligned} & p(x \mid c = k) = \mathcal{N}(x \,;\, \mu_k, \sigma_k^2) \\ & = \frac{1}{\sqrt{2\pi}\,\sigma_k}\exp\left(-\frac{(x - \mu_k)^2}{2\sigma_k^2}\right), \end{aligned} \]
with class-specific mean \(\mu_k\) and variance \(\sigma_k^2\). The prior over the class \(c\) tells us how likely each class is before we observe \(x\).
Question: In a binary classification problem, what is a natural choice of distribution for the prior \(p(c)\)? What is its parameter, and how many parameters does it have?
Answer:
For binary \(c \in \{0, 1\}\), we model the prior with a Bernoulli distribution. We write
\[ \begin{aligned} & p(c=1) = \theta, \quad p(c=0) = 1 - \theta, \end{aligned} \]
so the prior is fully specified by a single parameter \(\theta \in [0, 1]\). Thus \(p(c)\) is \(\mathrm{Bernoulli}(\theta)\).
With these parameters learned, we will be able to make inference about new observations \(x\) using the Bayes Rule,
\[ \begin{aligned} & p(c \mid x) = \frac{p(x \mid c)\,p(c)}{p(x)} \\ & = \frac{p(x \mid c)\,p(c)}{\sum_{c'}p(x \mid c')\,p(c')} \end{aligned} \]
In the next two subsections, we will consider learning (how to estimate the parameter of both the Gaussian and the Bernoulli), and inference (how to make predictions).
3.1 Learning: Maximum Likelihood
As before, we will use the maximum likelihood estimation criterion. In Section 2.1, we derived the maximum likelihood estimate for the parameters \(\mu\) and \(\sigma^2\) of a single Gaussian. For our binary classification problem, we must fit one set of these Gaussian parameters per class (\(\mu_0\), \(\sigma_0^2\), \(\mu_1\), \(\sigma_1^2\)), as well as \(\theta\) for our Bernoulli prior, \(p(c)\) from the training set
\[ \begin{aligned} & \mathcal{D} = \{(x^{(1)}, c^{(1)}), \ldots, (x^{(N)}, c^{(N)})\}. \end{aligned} \]
To find formulas for these five parameters, we will once again perform maximum likelihood estimation, this time starting from the full likelihood \(L\) of the data under our model. As you will soon see, even though this expression is more complex than the one for a single Gaussian, it can be cleanly broken into parts that can be solved independently of each other.
The likelihood is
\[ \begin{aligned} & L = p(\mathcal{D}) = \prod_{i=1}^N p(x^{(i)}, c^{(i)}) \\ & = \prod_{i=1}^N p(x^{(i)} \mid c^{(i)})\,p(c^{(i)}). \end{aligned} \]
Each term is the joint probability of one labeled example. Taking the logarithm gives the log-likelihood \(\ell = \log L\):
\[ \begin{aligned} & \ell = \log p(\mathcal{D}) = \sum_{i=1}^N \Bigl( \log p(x^{(i)} \mid c^{(i)}) + \log p(c^{(i)}) \Bigr). \end{aligned} \]
A key insight here is that we can split the sum by class, so that each term in the summation has the same distribution. That is, we write
\[ \begin{aligned} & \ell = \sum_{i : c^{(i)}=0} \Bigl( \log p(x^{(i)} \mid c^{(i)}) + \log p(c^{(i)}) \Bigr) \\ & \quad + \sum_{i : c^{(i)}=1} \Bigl( \log p(x^{(i)} \mid c^{(i)}) + \log p(c^{(i)}) \Bigr). \end{aligned} \]
The first sum is over indices \(i\) where \(c^{(i)}=0\), and the second sum is over the remaining indices. With this split, we have that in the first summation, the term in the sum can be written
\[ \begin{aligned} & \log p(x^{(i)} \mid c=0) + \log(1-\theta). \end{aligned} \]
For the second summation, the term can be written
\[ \begin{aligned} & \log p(x^{(i)} \mid c=1) + \log \theta. \end{aligned} \]
Let \(N_0\) and \(N_1\) be the number of training points in class 0 and 1 respectively (\(N_0 + N_1 = N\)). Then, using the assumption that \(p(x^{(i)} \mid c=0)\) and \(p(x^{(i)} \mid c=1)\) are Gaussian distributions,
\[ \begin{aligned} & \ell = \sum_{i : c^{(i)}=0} \log \mathcal{N}(x^{(i)} \,;\, \mu_0, \sigma_0^2) + N_0 \log(1-\theta) \\ & \quad + \sum_{i : c^{(i)}=1} \log \mathcal{N}(x^{(i)} \,;\, \mu_1, \sigma_1^2) + N_1 \log \theta. \end{aligned} \]
Question: Looking at the expression for \(\ell\) above, why can we maximize it separately with respect to \((\mu_0, \sigma_0^2)\), \((\mu_1, \sigma_1^2)\) and \(\theta\)?
Answer:
Thus \(\ell\) is a sum of three groups of terms. One depends only on \((\mu_0, \sigma_0^2)\), one depends only on \((\mu_1, \sigma_1^2)\), and one depends only on \(\theta\). So we can maximize \(\ell\) with respect to each group independently.
The terms in the first summation depend only on \((\mu_0, \sigma_0^2)\) and \(\theta\). In fact, the class-0 terms are exactly the log-likelihood of the \(N_0\) class-0 points under \(\mathcal{N}(x \,;\, \mu_0, \sigma_0^2)\), plus \(N_0 \log(1-\theta)\). Maximizing with respect to \(\mu_0\) and \(\sigma_0^2\) therefore gives the usual Gaussian MLEs using only the class-0 data:
\[ \begin{aligned} & \mu_0 = \frac{1}{N_0}\sum_{i : c^{(i)}=0} x^{(i)}, \\ & \sigma_0^2 = \frac{1}{N_0}\sum_{i : c^{(i)}=0} (x^{(i)} - \mu_0)^2. \end{aligned} \]
Similarly, the terms in the second summation are the log-likelihood of the \(N_1\) class-1 points under \(\mathcal{N}(x \,;\, \mu_1, \sigma_1^2)\) plus \(N_1 \log \theta\). Maximizing with respect to \(\mu_1\) and \(\sigma_1^2\) gives
\[ \begin{aligned} & \mu_1 = \frac{1}{N_1}\sum_{i : c^{(i)}=1} x^{(i)}, \\ & \sigma_1^2 = \frac{1}{N_1}\sum_{i : c^{(i)}=1} (x^{(i)} - \mu_1)^2. \end{aligned} \]
The only part of the log-likelihood that depends on \(\theta\) is
\[ \begin{aligned} & N_0 \log(1-\theta) + N_1 \log \theta. \end{aligned} \]
Question: Maximize the expression above with respect to \(\theta\) to find \(\theta_{\textrm{MLE}}\).
Answer:
Taking the derivative with respect to \(\theta\) and setting it to zero gives
\[ \begin{aligned} & -\frac{N_0}{1-\theta} + \frac{N_1}{\theta} = 0. \end{aligned} \]
Solving yields the fraction of training points in class 1,
\[ \begin{aligned} & \theta_{\textrm{MLE}} = \frac{N_1}{N}. \end{aligned} \]
Thus the parameters can be learned separately. We fit each class-conditional Gaussian using only the data from that class, and set the prior \(\theta\) to the empirical class fraction.
3.2 Learning: Parameter Estimation
Now that we know how to estimate each of our parameters using maximum likelihood estimation, we can calculate real values from our training data \(\mathcal{D}\), just as we did in Naive Bayes.
For example, the leaf dataset (from the supervised learning chapter) has oak leaves (say \(c=0\)) with widths 7.0, 9.0, 10.0, 5.0 cm, and maple leaves (\(c=1\)) with widths 13.0, 11.0, 9.0 cm. So we have \(N=7\) observations, with \(N_0=4\) oak and \(N_1=3\) maple.
As a reminder, learning this model means learning three groups of parameters. The first group is \(\mu_0\) and \(\sigma_0^2\), the mean and variance leaf width of class \(c=0\) (oak). We actually estimated these quantities above using only the oak leaves widths. The second group is \(\mu_1\) and \(\sigma_1^2\), the mean and variance leaf width of class \(c=1\) (maple). Let’s estimate these parameters below. The third is \(\theta\), the probability that a randomly chosen leaf is in class \(c=1\) (maple). That is,
\[ \begin{aligned} & \theta = p(c=1), \quad 1-\theta = p(c=0). \end{aligned} \]
Let’s estimate these parameters below.
Together, these parameters will specify the two class-conditional Gaussians \(p(x \mid c=0)\) and \(p(x \mid c=1)\) and the prior \(p(c)\) in the univariate GDA model.
Question: Consider the maple leaf widths (so \(N_1=3\))
\[ \begin{aligned} & x^{(1)}=13.0, \quad x^{(2)}=11.0, \quad x^{(3)}=9.0 \text{ cm}. \end{aligned} \]
Compute the MLE estimates \(\mu_1\) and \(\sigma_1^2\) for the class-conditional Gaussian
\[ \begin{aligned} & p(x \mid c=1) = \mathcal{N}(x \,;\, \mu_1, \sigma_1^2). \end{aligned} \]
Answer:
\[ \begin{aligned} & \mu_1 = \frac{1}{N_1}\sum_{i: c^{(i)}=1} x^{(i)} = \frac{13.0 + 11.0 + 9.0}{3} = 11.0 \textrm{ cm}\\ & \sigma_1^2 = \frac{1}{N_1}\sum_{i: c^{(i)}=1} (x^{(i)} - \mu_1)^2 \\ & = \frac{1}{3}\big[(13-11)^2 + (11-11)^2 + (9-11)^2\big] \\ & = \frac{1}{3}(4 + 0 + 4) = \frac{8}{3} \approx 2.67 \textrm{ cm}^2 \\ & \sigma_1 \approx 1.63 \textrm{ cm}. \end{aligned} \]
Question: For the leaf dataset with \(N_0=4\) oak and \(N_1=3\) maple leaves, what is the MLE \(\theta_{\textrm{MLE}}\) for the prior \(p(c)\)? Once you have \(\theta_{\textrm{MLE}}\) together with the other fitted parameters (\(\mu_0\), \(\sigma_0^2\), \(\mu_1\), \(\sigma_1^2\)) from above, write down the full set of parameter estimates for the univariate GDA model.
Answer:
The prior is \(\mathrm{Bernoulli}(\theta)\) with \(p(c=1) = \theta\). The MLE is the fraction of training points in class 1:
\[ \begin{aligned} & \theta_{\textrm{MLE}} = \frac{N_1}{N} = \frac{\text{number of maple leaves}}{\text{total number of leaves}} = \frac{3}{7}. \end{aligned} \]
So the full set of fitted parameters is
\[ \begin{aligned} & \hat{\theta} = 3/7, \quad \hat{\mu}_0 = 7.75, \quad \hat{\sigma}_0^2 = 3.6875, \quad \hat{\mu}_1 = 11.0, \quad \hat{\sigma}_1^2 = 8/3. \end{aligned} \]
With these, we have a fully specified univariate GDA model.
To summarize, fitting univariate GDA to this dataset gives the following parameter estimates:
\[ \begin{aligned} & \mu_0 = 7.75, \qquad \sigma_0^2 = 3.69, \qquad \mu_1 = 11.0, \qquad \sigma_1^2 = 2.67 , \qquad \theta = 3/7. \end{aligned} \]
The figure below visualizes this learned univariate GDA model. Along the horizontal axis we show leaf width in centimeters and plot the seven training examples as individual oak and maple leaves. Above them, the two learned Gaussians summarize how widths are distributed within each class. The oak curve is centered around narrower leaves, while the maple curve is centered around wider leaves. You can interpret the height of each curve at a given width \(x\) as how plausible that width is under each class, and (together with the class prior) these curves determine \(p(c \mid x)\) and the resulting decision boundary.
Univariate GDA on leaf width
3.3 Inference
During inference in univariate GDA, we have a new observation \(x\) and wish to predict its class \(c\).
Intuitively, for our example problem, the figure above shows what widths of oak leaves are likely \(p(x \mid c=0)\), and what widths of maples are likely \(p(x \mid c=1)\). The Bayes rule used for inference essentially compares these quantities, along with the overall likelihood of oaks vs maples.
Mathematically, we use Bayes’ rule to compute the posterior \(p(c \mid x)\) from the learned class-conditional densities \(p(x \mid c=0)\) and \(p(x \mid c=1)\) and the prior \(p(c)\). We then classify by choosing the class with the higher posterior. In the binary case, that is equivalent to thresholding \(p(c=1 \mid x)\) at \(1/2\).
\[ \begin{aligned} & p(c \mid x) = \frac{p(x \mid c)\,p(c)}{p(x)} \\ & = \frac{p(x \mid c)\,p(c)}{\sum_{c'}p(x \mid c')\,p(c')} \end{aligned} \]
In our example, for a new leaf with observed width \(x\), the posterior probability of class 1 (maple) given \(x\) is
\[ \begin{aligned} & p(c=1 \mid x) = \frac{p(x \mid c=1)\,p(c=1)}{p(x \mid c=0)\,p(c=0) + p(x \mid c=1)\,p(c=1)} \end{aligned} \]
Plugging in the Gaussian PDF for \(p(x \mid c=0)\) and \(p(x \mid c=1)\), along with the Bernoulli for \(p(c=0)\) and \(p(c=1)\), we have
\[ \begin{aligned} & p(c=1 \mid x) = \frac{\frac{1}{\sqrt{2\pi}\sigma_1}\exp\!\left(-\frac{(x-\mu_1)^2}{2\sigma_1^2}\right)\,\theta}{\frac{1}{\sqrt{2\pi}\sigma_0}\exp\!\left(-\frac{(x-\mu_0)^2}{2\sigma_0^2}\right)\,(1-\theta)\;+\;\frac{1}{\sqrt{2\pi}\sigma_1}\exp\!\left(-\frac{(x-\mu_1)^2}{2\sigma_1^2}\right)\,\theta}. \end{aligned} \]
We predict maple if \(p(c=1 \mid x) \geq 1/2\) and oak otherwise. The decision boundary is the value(s) of \(x\) where the two terms in the denominator are equal, i.e., where the two Gaussians (weighted by the prior) balance.
Question: If we only desire a binary prediction, do we need to compute the denominator \(p(x)\)? Why or why not?
Answer:
No. Both posteriors \(p(c=1 \mid x)\) and \(p(c=0 \mid x)\) share the same denominator, so it does not change which one is larger. In practice, if we only desire a binary prediction, we can compute and compare just
\[ \begin{aligned} & p(x \mid c=1)\,p(c=1) \quad \text{and} \quad p(x \mid c=0)\,p(c=0). \end{aligned} \]
Interactive inference for univariate GDA
4 Summary
The mathematics of learning a univariate GDA model by maximum likelihood works out so that we essentially learn a separate Gaussian distribution of the feature for each class, and (also separately) learn a Bernoulli distribution for the class probabilities.
Inference uses Bayes’ rule to compute the probability of each class given the input, just like in other generative models.
For multivariate GDA, the ideas are the same. We use one multivariate Gaussian per class and a Bernoulli prior. MLE gives per-class sample means and covariances, plus a prior equal to the fraction of training points in class 1, and inference is again Bayes’ rule. The main added nuance is that each class has a full covariance matrix (controlling shape and orientation), so the decision boundary is in general quadratic in the input, or linear if the two classes share the same covariance.