Backpropagation
Learning Objectives
After reading this page, you should be able to:
- Derive the gradient of the cost with respect to one weight/bias in a feedforward neural network using a computation graph and the univariate/multi-variate chain rule.
- Explain how backpropagation uses dynamic programming to improve efficiency of gradient computations.
- State the recursive backpropagation equations for an \(M\)-layer feedforward network.
- Explain why a forward pass must run before the backward pass during training.
1 Introduction
The Multi-Layer Perceptron chapter defined multi-layer feedforward networks and forward propagation in matrix–vector form. To train those networks we still use the same idea as in Gradient Descent: define a cost, compute partial derivatives of the cost with respect to every parameter, and update weights and biases in the direction that decreases the cost. The difficulty is scale. A network with many layers and units can have millions of parameters, and recomputing each partial derivative from scratch at every step is not feasible.
This page develops the backpropagation algorithm, an efficient way to obtain every gradient in roughly the time of one forward pass. We begin by explaining the challenge of training at scale, then work through gradient calculations by hand on a three-layer network. We introduce a visualization called the computation graph, and with it, illustrate where the chain rule paths “fan out”. That groundwork reveals overlapping subproblems that a single backward pass solves by storing intermediate derivatives—the dynamic-programming view of backpropagation. We apply the method to compute all gradients for the same network, state the recursive equations for an \(M\)-layer network, and explain why a forward pass must precede the backward pass to supply stored activations. A complete algorithm statement follows, along with vectorization and practical training considerations.
2 The Challenge of Training a Neural Network
We have already seen how to train a linear model (for example, logistic regression) using Gradient Descent. Consider \(N\) training examples where \(t^{(i)}\) is the target for example \(i\). Given the input \(\mathbf{x}\), the weight vector \(\mathbf{w}\), and the bias \(b\), the model computes the prediction \(y^{(i)}\) for example \(i\) as follows:
\[y^{(i)} = \sigma(\mathbf{w}^\top \mathbf{x}^{(i)} + b)\]
To train the model, we defined a cost function
\[\mathcal{E}(\mathbf{w}, b) = \frac{1}{N} \sum_{i=1}^{N} \mathcal{L}(y^{(i)}, t^{(i)})\]
Gradient descent then minimized the cost \(\mathcal{E}\) by repeatedly updating both \(\mathbf{w}\) and \(b\):
\[\mathbf{w} \leftarrow \mathbf{w} - \eta \, \nabla_{\mathbf{w}} \mathcal{E}, \qquad b \leftarrow b - \eta \, \frac{\partial \mathcal{E}}{\partial b}\]
where \(\eta > 0\) is the learning rate. The same optimization approach applies to a neural network: we initialize the weights and biases, compute the partial derivatives of the cost with respect to every parameter, and step in the direction that reduces the cost. Note that we can compute the partial derivatives of the cost with respect to each weight and bias using the chain rule.
The question is therefore not how to decide what to optimize and how to set up gradient descent, but how to compute the partial derivatives efficiently for every weight in the network. A neural network can easily have millions of weights and biases. Computing each partial derivative independently would require millions of computations per update step. That is computationally infeasible in practice.
The backpropagation algorithm solves this problem. By propagating information backward through the network and reusing intermediate computations, backpropagation computes the partial derivatives for all parameters in a single pass, with roughly the same cost as one forward pass.
3 Gradients for a 3-Layer Neural Network
To understand what backpropagation does and why it is needed, let us first calculate gradients naively by hand on a concrete network. The network we use has two hidden layers and a vector output. It is the smallest architecture that exposes every chain-rule structure that backpropagation has to handle. We write the forward pass in full non-vectorized form, then derive three (scalar) weight gradients step by step, each one harder than the last. Placing the three derivations side by side reveals the source of inefficiency that backpropagation is designed to eliminate.
3.1 The 3-Layer Architecture and Notation
We specialize the general feedforward-network notation from the notation table in the Neural Networks chapter to \(M = 3\). The network has four layers: an input layer (layer 0, \(D^{(0)}\) units), hidden layer 1 (layer 1, \(D^{(1)}\) units), hidden layer 2 (layer 2, \(D^{(2)}\) units), and an output layer (layer 3, \(D^{(3)}\) units). Because the output is a vector, we require \(D^{(3)} > 1\), and the target \(\mathbf{t} \in \mathbb{R}^{D^{(3)}}\) is likewise a vector.
3.2 Forward Pass Computations
To derive any gradient, we first need to know exactly how each quantity contributes to the loss computation. The forward pass through the 3-layer network proceeds layer by layer, computing pre-activations and activations in order from layer 1 to layer 3.
Layer 1. For each unit \(i = 1, \ldots, D^{(1)}\):
\[ z^{(1)}_i = \sum_{j=1}^{D^{(0)}} W^{(1)}_{i,j} \, a^{(0)}_j + b^{(1)}_i, \qquad a^{(1)}_i = \sigma^{(1)}\!\bigl(z^{(1)}_i\bigr). \]
The pre-activation \(z^{(1)}_i\) is a weighted sum of all input units plus a bias; applying \(\sigma^{(1)}\) element-wise produces the first hidden layer’s output activations.
Layer 2. For each unit \(i = 1, \ldots, D^{(2)}\):
\[ z^{(2)}_i = \sum_{j=1}^{D^{(1)}} W^{(2)}_{i,j} \, a^{(1)}_j + b^{(2)}_i, \qquad a^{(2)}_i = \sigma^{(2)}\!\bigl(z^{(2)}_i\bigr). \]
Layer 2 has the same form as layer 1, but now takes the first hidden layer’s activations \(a^{(1)}_1, \ldots, a^{(1)}_{D^{(1)}}\) as inputs rather than the raw inputs.
Output layer (layer 3). For each unit \(i = 1, \ldots, D^{(3)}\):
\[ z^{(3)}_i = \sum_{j=1}^{D^{(2)}} W^{(3)}_{i,j} \, a^{(2)}_j + b^{(3)}_i, \qquad a^{(3)}_i = \sigma^{(3)}\!\bigl(z^{(3)}_i\bigr). \]
The output layer takes the second hidden layer’s activations as inputs and produces \(D^{(3)}\) output activations \(a^{(3)}_1, \ldots, a^{(3)}_{D^{(3)}}\).
Cost. The cost decomposes as a sum of per-unit losses:
\[ C = \sum_{k=1}^{D^{(3)}} L\!\bigl(a^{(3)}_k,\, t_k\bigr). \]
Each term \(L(a^{(3)}_k, t_k)\) depends only on the \(k\)-th output activation and the \(k\)-th target. Since all other terms in the sum are constant with respect to \(a^{(3)}_k\), the per-unit decomposition gives:
\[\frac{\partial C}{\partial a^{(3)}_k} = \frac{\partial L(a^{(3)}_k, t_k)}{\partial a^{(3)}_k}.\]
We will reference these forward-pass computations repeatedly in the derivations below.
3.3 Worked Example 1: Gradient with Respect to an Output-Layer Weight
We begin with the simplest case: the gradient of the cost with respect to a weight in the output layer. Consider \(W^{(3)}_{1,2}\), the weight connecting unit 2 of hidden layer 2 to unit 1 of the output layer. We want to compute the partial derivative of the cost with respect to \(W^{(3)}_{1,2}\).
Before we apply the chain rule, we need a clear picture of how \(W^{(3)}_{1,2}\) affects the cost through the forward pass. We record those forward dependencies in a computation graph. The computation graph is our main tool for tracing the forward pass computations and deriving the corresponding gradients.
Definition: A computation graph is a directed graph that represents how scalar quantities in the forward pass depend on one another. Each node is one scalar: a weight, a bias, a pre-activation \(z\), an activation \(a\), or the total cost \(C\). Each directed edge from an upstream node to a downstream node means the downstream quantity is computed using the upstream quantity as an input.
Output-layer weight path
Figure 2 is an example of a computation graph that traces the forward path for \(W^{(3)}_{1,2}\). Reading left to right, the weight \(W^{(3)}_{1,2}\) is used to compute the pre-activation \(z^{(3)}_1\). Applying the activation function \(\sigma^{(3)}\) to the pre-activation \(z^{(3)}_1\) produces the activation \(a^{(3)}_1\). Then the activation \(a^{(3)}_1\) is used to calculate the total cost \(C\). Every arrow in the figure is a forward dependency from the layer equations in the previous subsection; we are not yet taking any derivatives.
We can express the same path in compact notation as follows.
\[W^{(3)}_{1,2} \;\longrightarrow\; z^{(3)}_1 \;\longrightarrow\; a^{(3)}_1 \;\longrightarrow\; C.\]
Having traced the forward path, we are ready to compute the partial derivative of the total cost with respect to the weight \(W^{(3)}_{1,2}\). At each step, one variable leads to exactly one downstream variable. Therefore, we only need the single-variable chain rule to compute the partial derivative.
Definition (single-variable chain rule): Let \(x\), \(u\), and \(y\) be scalars. Suppose \(u = g(x)\) and \(y = f(u)\) for differentiable scalar functions \(g\) and \(f\). Then the chain rule for derivatives states that \[ \frac{\partial y}{\partial x} = \frac{\partial y}{\partial u}\cdot\frac{\partial u}{\partial x}. \]
Applying the single-variable chain rule to the forward path, we can write the partial derivative of the total cost with respect to the weight as follows:
\[ \frac{\partial C}{\partial W^{(3)}_{1,2}} = \frac{\partial C}{\partial a^{(3)}_1} \cdot \frac{\partial a^{(3)}_1}{\partial z^{(3)}_1} \cdot \frac{\partial z^{(3)}_1}{\partial W^{(3)}_{1,2}}. \]
We evaluate each factor in the chain rule separately.
For the derivative of the total cost with respect to the output activation \(a^{(3)}_1\), the per-unit loss decomposition gives the derivative of the single contributing term:
\[ \frac{\partial C}{\partial a^{(3)}_1} = \frac{\partial L(a^{(3)}_1, t_1)}{\partial a^{(3)}_1}. \]
For the derivative of the output activation with respect to its pre-activation, the activation function contributes:
\[ \frac{\partial a^{(3)}_1}{\partial z^{(3)}_1} = \sigma^{(3)\prime}\!\bigl(z^{(3)}_1\bigr). \]
For the derivative of the pre-activation with respect to the weight, the output-layer forward equation gives the upstream hidden activation as the coefficient of \(W^{(3)}_{1,2}\):
\[ \frac{\partial z^{(3)}_1}{\partial W^{(3)}_{1,2}} = a^{(2)}_2. \]
Substituting these three factors into the chain rule gives:
\[ \frac{\partial C}{\partial W^{(3)}_{1,2}} = \frac{\partial L(a^{(3)}_1, t_1)}{\partial a^{(3)}_1} \cdot \sigma^{(3)\prime}\!\bigl(z^{(3)}_1\bigr) \cdot a^{(2)}_2. \]
The result is a product of three quantities: the derivative of the per-unit loss with respect to the output activation, the derivative of the output activation function at the pre-activation, and the upstream hidden activation that feeds into this weight. This pattern — a chain of three factors — applies to any output-layer weight \(W^{(3)}_{i,j}\); the indices \(i\) and \(j\) change but the structure does not.
The bias gradient follows the same chain with the last factor replaced by 1, since the partial derivative of \(z^{(3)}_1\) with respect to the bias \(b^{(3)}_1\) equals 1:
\[ \frac{\partial C}{\partial b^{(3)}_1} = \frac{\partial L(a^{(3)}_1, t_1)}{\partial a^{(3)}_1} \cdot \sigma^{(3)\prime}\!\bigl(z^{(3)}_1\bigr). \]
3.4 Worked Example 2: Gradient with Respect to a Hidden-Layer-2 Weight
Now consider a weight one layer deeper: \(W^{(2)}_{3,4}\), connecting unit 4 of hidden layer 1 to unit 3 of hidden layer 2. We want to compute the partial derivative of the cost with respect to \(W^{(2)}_{3,4}\).
In Section 3.3, we ended with a single chain from weight to cost, which required the single-variable chain rule only. The early steps for \(W^{(2)}_{3,4}\) are equally local. The weight \(W^{(2)}_{3,4}\) appears only in the calculation of the pre-activation \(z^{(2)}_3\). Similarly, the pre-activation \(z^{(2)}_3\) is only used to calculate the activation \(a^{(2)}_3\). However, the path splits after the computation of the activation \(a^{(2)}_3\). Specifically, the activation \(a^{(2)}_3\) is used to calculate all the pre-activations \(z^{(3)}_1, \ldots, z^{(3)}_{D^{(3)}}\) in layer 3, and all of those pre-activations influence the total cost \(C\). This is the first time we encounter fan-out in a computation graph, a structural property we now define precisely.
Definition: A node in a computation graph fans out if it has more than one outgoing edge — its value is used as an input to more than one downstream node.
When a node fans out and all downstream paths eventually reach the cost \(C\), the derivative of \(C\) with respect to that node is a sum over all outgoing paths, which is exactly what the multivariate chain rule captures. We draw the computation graph in Figure 3 to visualize where the fan-out occurs.
Hidden-layer 2 fan-out graph
Reading Figure 3 from left to right, \(W^{(2)}_{3,4}\) and \(a^{(1)}_4\) appear in the computation of \(z^{(2)}_3\). Then applying the activation function \(\sigma^{(2)}\) produces the activation \(a^{(2)}_3\). From the activation \(a^{(2)}_3\), the graph fans out: each output pre-activation \(z^{(3)}_k\) that uses \(a^{(2)}_3\) sits on its own branch, linked by the weight \(W^{(3)}_{k,3}\), and every branch continues forward to \(C\).
We can express the same paths in compact notation as follows.
\[W^{(2)}_{3,4} \;\longrightarrow\; z^{(2)}_3 \;\longrightarrow\; a^{(2)}_3 \;\longrightarrow\; \bigl\{\, z^{(3)}_1,\, z^{(3)}_2,\, \ldots,\, z^{(3)}_{D^{(3)}} \,\bigr\} \;\longrightarrow\; C.\]
Because of the fan-out, we need to apply the multivariate chain rule to compute the partial derivative of the total cost with respect to the activation \(a^{(2)}_3\).
Definition (multivariate chain rule): Let \(u\), \(v_1, \ldots, v_m\), and \(y\) be scalars. Suppose each \(v_k = g_k(u)\) for differentiable functions \(g_1, \ldots, g_m\), and \(y = f(v_1, \ldots, v_m)\) for a differentiable function \(f\). Then \[ \frac{\partial y}{\partial u} = \sum_{k=1}^{m} \frac{\partial y}{\partial v_k} \cdot \frac{\partial v_k}{\partial u}. \]
We compute the gradient in two steps, treating the non-branching and branching portions of the forward path separately.
Step 1. From \(W^{(2)}_{3,4}\) to \(a^{(2)}_3\) the path does not branch: the weight appears only in \(z^{(2)}_3\), and \(z^{(2)}_3\) drives only \(a^{(2)}_3\). The single-variable chain rule therefore gives:
\[ \frac{\partial C}{\partial W^{(2)}_{3,4}} = \frac{\partial C}{\partial a^{(2)}_3} \cdot \frac{\partial a^{(2)}_3}{\partial z^{(2)}_3} \cdot \frac{\partial z^{(2)}_3}{\partial W^{(2)}_{3,4}} = \frac{\partial C}{\partial a^{(2)}_3} \cdot \sigma^{(2)\prime}\!\bigl(z^{(2)}_3\bigr) \cdot a^{(1)}_4. \]
This leaves us to evaluate the partial derivative of \(C\) with respect to \(a^{(2)}_3\). Computing this quantity requires the multivariate chain rule, because \(a^{(2)}_3\) fans out.
Step 2. At \(a^{(2)}_3\) the graph fans out to every output pre-activation \(z^{(3)}_k\) for \(k = 1, \ldots, D^{(3)}\). Applying the multivariate chain rule with \(u = a^{(2)}_3\), \(v_k = z^{(3)}_k\), and \(y = C\):
\[ \frac{\partial C}{\partial a^{(2)}_3} = \sum_{k=1}^{D^{(3)}} \frac{\partial C}{\partial z^{(3)}_k} \cdot \frac{\partial z^{(3)}_k}{\partial a^{(2)}_3} = \sum_{k=1}^{D^{(3)}} \frac{\partial C}{\partial z^{(3)}_k} \cdot W^{(3)}_{k,3}. \]
We already derived an expression for one instance of the partial derivative of \(C\) with respect to \(z^{(3)}_k\) in Section 3.3. Below is the expression for the \(k\)-th pre-activation in layer 3.
\[ \frac{\partial C}{\partial z^{(3)}_k} = \frac{\partial L(a^{(3)}_k, t_k)}{\partial a^{(3)}_k} \cdot \sigma^{(3)\prime}\!\bigl(z^{(3)}_k\bigr), \quad k = 1, \ldots, D^{(3)}. \]
Therefore, we can update the expression for the partial derivative of \(C\) with respect to \(a^{(2)}_3\) as follows:
\[ \frac{\partial C}{\partial a^{(2)}_3} = \sum_{k=1}^{D^{(3)}} \frac{\partial L(a^{(3)}_k, t_k)}{\partial a^{(3)}_k} \cdot \sigma^{(3)\prime}\!\bigl(z^{(3)}_k\bigr) \cdot W^{(3)}_{k,3}. \]
Let’s substitute the result of Step 2 into Step 1. The result is as follows.
\[ \begin{aligned} \frac{\partial C}{\partial W^{(2)}_{3,4}} &= \left(\, \sum_{k=1}^{D^{(3)}} \frac{\partial C}{\partial z^{(3)}_k} \cdot W^{(3)}_{k,3} \right) \cdot \sigma^{(2)\prime}\!\bigl(z^{(2)}_3\bigr) \cdot a^{(1)}_4 \\ &= \left(\, \sum_{k=1}^{D^{(3)}} \frac{\partial L(a^{(3)}_k, t_k)}{\partial a^{(3)}_k} \cdot \sigma^{(3)\prime}\!\bigl(z^{(3)}_k\bigr) \cdot W^{(3)}_{k,3} \right) \cdot \sigma^{(2)\prime}\!\bigl(z^{(2)}_3\bigr) \cdot a^{(1)}_4. \end{aligned} \]
To summarize, Section 3.4 added one structural feature beyond Section 3.3: the fan-out at \(a^{(2)}_3\). The path before the fan-out is still a single chain handled by the single-variable chain rule; the fan-out itself is handled by the multivariate chain rule, which produces a sum with one term per output unit. The inner factor of each term — the partial derivative of \(C\) with respect to \(z^{(3)}_k\) — is exactly the output-layer derivative from Section 3.3, now needed at every \(k\) rather than just one.
3.5 Worked Example 3: Gradient with Respect to a Hidden-Layer-1 Weight
For the deepest case, consider \(W^{(1)}_{2,3}\), connecting unit 3 of the input layer to unit 2 of hidden layer 1. We want to derive the partial derivative of the cost with respect to \(W^{(1)}_{2,3}\). Rather than presenting the detailed derivations, we leave this task as an exercise for the reader. Below, we present the computation graph and the results of the derivation. We strongly encourage you to attempt the derivation yourself before reading the steps below — use the computation graph and the chain rules from Section 3.3 and Section 3.4 as your guide.
Hidden-layer 1 fan-out graph
The forward path has fan-out at two depths:
\[W^{(1)}_{2,3} \;\longrightarrow\; z^{(1)}_2 \;\longrightarrow\; a^{(1)}_2 \;\longrightarrow\; \bigl\{\, z^{(2)}_i \,\bigr\}_{i=1}^{D^{(2)}} \;\longrightarrow\; \bigl\{\, z^{(3)}_k \,\bigr\}_{k=1}^{D^{(3)}} \;\longrightarrow\; C.\]
We present the derivation in three steps.
Step 1. Partial derivative of \(C\) with respect to \(a^{(1)}_2\) (first fan-out, multivariate chain rule):
\[ \frac{\partial C}{\partial a^{(1)}_2} = \sum_{i=1}^{D^{(2)}} \frac{\partial C}{\partial z^{(2)}_i} \cdot W^{(2)}_{i,2}. \]
Step 2. Partial derivative of \(C\) with respect to \(z^{(2)}_i\) and \(a^{(2)}_i\) (second fan-out, multivariate chain rule):
\[ \begin{aligned} \frac{\partial C}{\partial z^{(2)}_i} &= \frac{\partial C}{\partial a^{(2)}_i} \cdot \sigma^{(2)\prime}\!\bigl(z^{(2)}_i\bigr), \\ \frac{\partial C}{\partial a^{(2)}_i} &= \sum_{k=1}^{D^{(3)}} \frac{\partial C}{\partial z^{(3)}_k} \cdot W^{(3)}_{k,i}, \\ &= \sum_{k=1}^{D^{(3)}} \frac{\partial C}{\partial z^{(3)}_k} \cdot W^{(3)}_{k,i} \cdot \sigma^{(2)\prime}\!\bigl(z^{(2)}_i\bigr). \end{aligned} \]
Step 3. Full gradient with respect to \(W^{(1)}_{2,3}\):
\[ \frac{\partial C}{\partial W^{(1)}_{2,3}} = \left[\, \sum_{i=1}^{D^{(2)}} \left( \sum_{k=1}^{D^{(3)}} \frac{\partial C}{\partial z^{(3)}_k} \cdot W^{(3)}_{k,i} \right) \sigma^{(2)\prime}\!\bigl(z^{(2)}_i\bigr) \cdot W^{(2)}_{i,2} \,\right] \;\cdot\; \sigma^{(1)\prime}\!\bigl(z^{(1)}_2\bigr) \;\cdot\; a^{(0)}_3. \]
To evaluate this expression, the inner sum over \(k\) must be computed once for each \(i\) in layer 2. The partial derivative of \(C\) with respect to \(z^{(3)}_k\) does not depend on \(i\), yet in this direct calculation it is recomputed from scratch \(D^{(2)}\) times. Comparing Section 3.3, Section 3.4, and Section 3.5, the same intermediate quantities — the partial derivative of \(C\) with respect to \(z^{(3)}_k\) for each output unit \(k\), and the partial derivative of \(C\) with respect to \(a^{(2)}_i\) for each hidden-layer-2 unit \(i\) — reappear in multiple places, each time computed from scratch. The next section makes this redundancy precise and shows how reusing these quantities reduces the total work to a single backward pass through the network.
4 Computing Gradients Efficiently
The three worked examples above compute one weight gradient each, but they expose a pattern: the same intermediate derivatives appear again and again. If we repeated that procedure for every weight and bias in the network, we would redo most of the chain-rule algebra many times over. In the analysis below, we will make that redundancy precise and introduce an alternative that computes every gradient in one backward pass by reusing intermediate results.
The direct approach is inefficient because it recomputes the same partial derivatives whenever a new weight gradient is needed. In Section 3.4, the sum over output units requires \(\partial C / \partial z^{(3)}_k\) for every \(k = 1, \ldots, D^{(3)}\). We already know that each term has the same form as in Section 3.3:
\[ \frac{\partial C}{\partial z^{(3)}_k} = \frac{\partial L(a^{(3)}_k, t_k)}{\partial a^{(3)}_k} \cdot \sigma^{(3)\prime}\!\bigl(z^{(3)}_k\bigr), \qquad k = 1, \ldots, D^{(3)}. \]
In Section 3.5, the redundancy is worse. The gradient for \(W^{(1)}_{2,3}\) nests a sum over layer-2 units \(i = 1, \ldots, D^{(2)}\), and inside each term it again needs \(\partial C / \partial z^{(3)}_k\) for every output unit \(k = 1, \ldots, D^{(3)}\). The factor \(\partial C / \partial z^{(3)}_k\) does not depend on \(i\), yet the nested derivation treats it as something to rebuild for each \(i = 1, \ldots, D^{(2)}\). Likewise, \(\partial C / \partial a^{(2)}_i\) is a sum over \(k = 1, \ldots, D^{(3)}\) that reappears whenever we differentiate through a layer-2 activation:
\[ \frac{\partial C}{\partial a^{(2)}_i} = \sum_{k=1}^{D^{(3)}} \frac{\partial C}{\partial z^{(3)}_k} \cdot W^{(3)}_{k,i}. \]
Scaling this pattern to all parameters, computing each \(\partial C / \partial W^{(m)}_{i,j}\) independently would repeat the same backward chains countless times.
The alternative is a single backward pass through the computation graph. Starting at the cost \(C\), we apply the chain rule in the direction opposite to the forward pass: we first compute how \(C\) changes with respect to the output-layer quantities, then propagate those gradients one layer backward, and continue toward the inputs.
Computing the gradients through this backward traversal uses the idea of dynamic programming — a technique that solves a complex problem by breaking it into overlapping subproblems, solving each subproblem once in a specific order, so that the result can be stored and reused. In this case, the subproblem at each node is the partial derivative of \(C\) with respect to that node’s quantity, and each subproblem is solved exactly once. We store every intermediate derivative as we go — for example, all of \(\partial C / \partial z^{(3)}_k\) for \(k = 1, \ldots, D^{(3)}\), then all of \(\partial C / \partial a^{(2)}_i\) for \(i = 1, \ldots, D^{(2)}\) — and look them up whenever a later step needs them. When we reach a weight \(W^{(m)}_{i,j}\), its gradient is a product of two things: the stored derivative \(\partial C / \partial z^{(m)}_i\) (already computed when we processed layer \(m\)) and the activation \(a^{(m-1)}_j\). For a bias \(b^{(m)}_i\), the same stored derivative is all that is needed, since \(\partial z^{(m)}_i / \partial b^{(m)}_i = 1\). One backward pass therefore produces \(\nabla_{\mathbf{W}^{(m)}} C\) and \(\nabla_{\mathbf{b}^{(m)}} C\) for every layer \(m\) simultaneously, with no redundant computation.
Figure 5 below can help visualize this process. Each node is an entire layer quantity — \(\mathbf{a}^{(m)}\), \(\mathbf{W}^{(m)}\), \(\mathbf{b}^{(m)}\), or \(\mathbf{z}^{(m)}\) — and the arrows are the vectorized forward operations from the Neural Networks chapter. A backward pass starts from the cost on the right and traverses this graph in reverse. Every step adds a corresponding derivative to the chain rule computation. When we reach the derivative with respect to a pre-activation \(z^{(m)}_i\), we store the result and use it to compute the gradients with respect to the weights and biases that feed into \(z^{(m)}_i\). To foreshadow what follows, the backpropagation algorithm is a recursive procedure whose recursive calls happen at the pre-activation nodes in each layer.
Vectorized forward graph
5 Backpropagation for a 3-Layer Network
Let’s consider our 3-layer network again and go through the process of computing all the gradients using one backward pass through the computation graph. We will complete the following steps, in order:
- Compute gradients for the output layer weights.
\[ \begin{aligned} \frac{\partial C}{\partial z^{(3)}_i} &= ?, && i = 1, \ldots, D^{(3)}, \\ \frac{\partial C}{\partial W^{(3)}_{i,j}} &= ?, && i = 1, \ldots, D^{(3)},\; j = 1, \ldots, D^{(2)}, \\ \frac{\partial C}{\partial b^{(3)}_i} &= ?, && i = 1, \ldots, D^{(3)}. \end{aligned} \]
- Compute gradients for hidden layer 2 weights.
\[ \begin{aligned} \frac{\partial C}{\partial z^{(2)}_i} &= ?, && i = 1, \ldots, D^{(2)}, \\ \frac{\partial C}{\partial W^{(2)}_{i,j}} &= ?, && i = 1, \ldots, D^{(2)},\; j = 1, \ldots, D^{(1)}, \\ \frac{\partial C}{\partial b^{(2)}_i} &= ?, && i = 1, \ldots, D^{(2)}. \end{aligned} \]
- Compute gradients for hidden layer 1 weights.
\[ \begin{aligned} \frac{\partial C}{\partial z^{(1)}_i} &= ?, && i = 1, \ldots, D^{(1)}, \\ \frac{\partial C}{\partial W^{(1)}_{i,j}} &= ?, && i = 1, \ldots, D^{(1)},\; j = 1, \ldots, D^{(0)}, \\ \frac{\partial C}{\partial b^{(1)}_i} &= ?, && i = 1, \ldots, D^{(1)}. \end{aligned} \]
We fill in each step below, working backward from the cost. At every layer we use the same index convention as the forward pass: \(i\) labels a unit in the current layer and \(j\) labels a unit in the layer below.
5.1 Computing Gradients for the Output Layer
We begin at the output layer, where the cost is defined, and then work backward towards the inputs. The cost is a sum of per-unit losses, so each output activation \(a^{(3)}_i\) only appears in the \(i\)-th loss term, where \(i = 1, \ldots, D^{(3)}\); its gradient \(\partial C / \partial a^{(3)}_i\) is just the derivative of that single term. Next, we compute the derivative with respect to the pre-activation \(z^{(3)}_i\) by multiplying by the derivative of the non-linearity based on the chain rule. We store \(\partial C / \partial z^{(3)}_i\) at each unit \(i\) and reuse this intermediate value in the next step. With the derivative with respect to the pre-activation \(\partial C / \partial z^{(3)}_i\) in hand, the weight and bias gradients are immediate; Figure 6 highlights the output-layer nodes in the vectorized graph.
\[ \begin{aligned} \frac{\partial C}{\partial z^{(3)}_i} &= \frac{\partial C}{\partial a^{(3)}_i} \cdot \sigma^{(3)\prime}\!\bigl(z^{(3)}_i\bigr), && i = 1, \ldots, D^{(3)}, \\ \frac{\partial C}{\partial W^{(3)}_{i,j}} &= \frac{\partial C}{\partial z^{(3)}_i} \cdot a^{(2)}_j, && i = 1, \ldots, D^{(3)},\; j = 1, \ldots, D^{(2)}, \\ \frac{\partial C}{\partial b^{(3)}_i} &= \frac{\partial C}{\partial z^{(3)}_i}, && i = 1, \ldots, D^{(3)}. \end{aligned} \]
Output-layer backpropagation nodes
6 The Backpropagation Algorithm (An Almost Complete Version)
Let’s summarize the backpropagation algorithm for an \(M\)-layer network.
- Compute gradients for the output layer.
\[ \begin{aligned} \frac{\partial C}{\partial z^{(M)}_i} &= \frac{\partial C}{\partial a^{(M)}_i} \cdot \sigma^{(M)\prime}\!\bigl(z^{(M)}_i\bigr), && i = 1, \ldots, D^{(M)} \end{aligned} \]
- Compute gradients for each hidden layer recursively.
For each \(m = M-1, ..., 1\),
\[ \begin{aligned} \frac{\partial C}{\partial z^{(m)}_i} &= \left( \sum_{k=1}^{D^{(m+1)}} \frac{\partial C}{\partial z^{(m+1)}_k} \cdot W^{(m+1)}_{k,i} \right) \cdot \sigma^{(m)\prime}\!\bigl(z^{(m)}_i\bigr), && i = 1, \ldots, D^{(m)}, \end{aligned} \]
- Compute gradients for the weights and biases for each hidden layer.
For each \(m = M-1, ..., 1\),
\[ \begin{aligned} \frac{\partial C}{\partial W^{(m)}_{i,j}} &= \frac{\partial C}{\partial z^{(m)}_i} \cdot a^{(m-1)}_j, && i = 1, \ldots, D^{(m)},\; j = 1, \ldots, D^{(m-1)}, \\ \frac{\partial C}{\partial b^{(m)}_i} &= \frac{\partial C}{\partial z^{(m)}_i}, && i = 1, \ldots, D^{(m)}. \end{aligned} \]
7 Why Do We Need the Forward Pass?
Let’s examine the backpropagation equations for an \(M\)-layer network below and identify the quantities we need to prepare before carrying out the computations.
\[ \begin{aligned} \frac{\partial C}{\partial z^{(M)}_i} &= \frac{\partial C}{\partial a^{(M)}_i} \cdot \sigma^{(M)\prime}\!\bigl(z^{(M)}_i\bigr), && i = 1, \ldots, D^{(M)}, \\ \text{ For each } m &= M-1, ..., 1, \\ \frac{\partial C}{\partial z^{(m)}_i} &= \left( \sum_{k=1}^{D^{(m+1)}} \frac{\partial C}{\partial z^{(m+1)}_k} \cdot W^{(m+1)}_{k,i} \right) \cdot \sigma^{(m)\prime}\!\bigl(z^{(m)}_i\bigr), && i = 1, \ldots, D^{(m)}, \\ \text{ For each } m &= M-1, ..., 1, \\ \frac{\partial C}{\partial W^{(m)}_{i,j}} &= \frac{\partial C}{\partial z^{(m)}_i} \cdot a^{(m-1)}_j, && i = 1, \ldots, D^{(m)},\; j = 1, \ldots, D^{(m-1)}, \\ \frac{\partial C}{\partial b^{(m)}_i} &= \frac{\partial C}{\partial z^{(m)}_i}, && i = 1, \ldots, D^{(m)}. \end{aligned} \]
Three of the required quantities can be prepared in advance. The derivative of the loss with respect to the output activations \(\frac{\partial C}{\partial a^{(M)}_i}\) is determined once we fix the loss function. The derivative of the non-linearity with respect to the pre-activations \(\sigma^{(m)\prime}\!\bigl(z^{(m)}_i\bigr)\) is determined once we fix the activation function for each layer. The weight matrices \(W^{(m)}\) are the quantities backpropagation exists to update, so we always keep track of them.
One quantity cannot be prepared in advance: the activations \(a^{(m-1)}_j\) from the previous layer, which appear in the gradient formula for each weight matrix. These activations depend on the specific input fed to the network — they differ for every training example.
This is why the backward pass must be preceded by a forward pass. The forward pass computes and stores the intermediate activations at each layer, making them available when the backward pass needs them.
7.1 The Backpropagation Algorithm (Complete Version)
Here is the complete backpropagation algorithm for an \(M\)-layer network.
- Perform the forward pass. Compute and store the activations for each layer.
For each layer \(m = 1, \ldots, M\),
\[ \begin{aligned} z^{(m)}_i &= \sum_{j=1}^{D^{(m-1)}} W^{(m)}_{i,j} \, a^{(m-1)}_j + b^{(m)}_i, && i = 1, \ldots, D^{(m)}, \\ a^{(m)}_i &= \sigma^{(m)}\!\bigl(z^{(m)}_i\bigr), && i = 1, \ldots, D^{(m)}. \end{aligned} \]
- Perform the backward pass. Compute the gradients for the weights and biases for each layer.
For each layer \(m = M-1, ..., 1\),
\[ \begin{aligned} \frac{\partial C}{\partial z^{(M)}_i} &= \frac{\partial C}{\partial a^{(M)}_i} \cdot \sigma^{(M)\prime}\!\bigl(z^{(M)}_i\bigr), && i = 1, \ldots, D^{(M)}, \\[5pt] \text{ For each } m &= M-1, ..., 1, \\ \frac{\partial C}{\partial z^{(m)}_i} &= \left( \sum_{k=1}^{D^{(m+1)}} \frac{\partial C}{\partial z^{(m+1)}_k} \cdot W^{(m+1)}_{k,i} \right) \cdot \sigma^{(m)\prime}\!\bigl(z^{(m)}_i\bigr), && i = 1, \ldots, D^{(m)}, \\[5pt] \text{ For each } m &= M-1, ..., 1, \\ \frac{\partial C}{\partial W^{(m)}_{i,j}} &= \frac{\partial C}{\partial z^{(m)}_i} \cdot a^{(m-1)}_j, && i = 1, \ldots, D^{(m)},\; j = 1, \ldots, D^{(m-1)}, \\ \frac{\partial C}{\partial b^{(m)}_i} &= \frac{\partial C}{\partial z^{(m)}_i}, && i = 1, \ldots, D^{(m)}. \end{aligned} \]
- Update the weights and biases through gradient descent.
For each layer \(m = M-1, ..., 1\),
\[ \begin{aligned} W^{(m)}_{i,j} &\leftarrow W^{(m)}_{i,j} - \eta \cdot \frac{\partial C}{\partial W^{(m)}_{i,j}}, && i = 1, \ldots, D^{(m)},\; j = 1, \ldots, D^{(m-1)}, \\ b^{(m)}_i &\leftarrow b^{(m)}_i - \eta \cdot \frac{\partial C}{\partial b^{(m)}_i}, && i = 1, \ldots, D^{(m)}. \end{aligned} \]
where \(\eta > 0\) is the learning rate.
8 Vectorizing the Backpropagation Algorithm
In practice, we will vectorize the backpropagation algorithm to improve efficiency. In this section, we will present a few exercises to guide you through the vectorization process. Vectorization is a skill that requires considerable practice and experience. We encourage you to take advantage of this great opportunity to practice your vectorization skills.
8.1 Vectorizing the Forward Pass
Let’s start with the forward pass. How would you vectorize the following equation to compute the pre-activations \(\mathbf{z}^{(m)}\) for each layer \(m\)?
Question: What is the vectorized expression for the following equation to compute the pre-activations \(\mathbf{z}^{(m)}\) for each layer \(m\)?
\[ z^{(m)}_i = \sum_{j=1}^{D^{(m-1)}} W^{(m)}_{i,j} \, a^{(m-1)}_j + b^{(m)}_i, \qquad i = 1, \ldots, D^{(m)}. \]
- \((\mathbf{W}^{(m)})^\top \mathbf{a}^{(m-1)} + \mathbf{b}^{(m)}\)
- \((\mathbf{a}^{(m-1)})^\top \mathbf{W}^{(m)} + \mathbf{b}^{(m)}\)
- \(\mathbf{W}^{(m)} \mathbf{a}^{(m-1)} + \mathbf{b}^{(m)}\)
- \(\mathbf{a}^{(m-1)} \mathbf{W}^{(m)} + \mathbf{b}^{(m)}\)
Answer:
The correct answer is (C).
The weight matrix has shape \(W^{(m)} \in \mathbb{R}^{D^{(m)} \times D^{(m-1)}}\), and the activation vector has shape \(a^{(m-1)} \in \mathbb{R}^{D^{(m-1)}}\). The weighted sum is expected to have shape \(z^{(m)} \in \mathbb{R}^{D^{(m)}}\). Therefore, the correct way of multiplying them together is \(\mathbf{W}^{(m)} \mathbf{a}^{(m-1)}\). Adding the bias vector \(\mathbf{b}^{(m)} \in \mathbb{R}^{D^{(m)}}\) gives the final vectorized expression \(\mathbf{z}^{(m)} = \mathbf{W}^{(m)} \mathbf{a}^{(m-1)} + \mathbf{b}^{(m)}\).
Therefore, a summary of the vectorized forward pass is:
For each layer \(m = 1, \ldots, M-1\),
\[ \begin{aligned} \mathbf{z}^{(m)} &= \mathbf{W}^{(m)} \mathbf{a}^{(m-1)} + \mathbf{b}^{(m)}, \\ \mathbf{a}^{(m)} &= \sigma^{(m)}(\mathbf{z}^{(m)}). \end{aligned} \]
8.2 Vectorizing the Backward Pass
We are now ready to vectorize the backward pass. Let’s break down the backward pass into two parts, the base case and the recursive case. The base case focuses on computing the gradients for the output layer. The recursive case focuses on computing the gradients for the hidden layers.
We begin with the base case, which computes the gradients for the output layer. How would you vectorize the following equation to compute the gradients of the cost with respect to the pre-activations \(\mathbf{z}^{(M)}\) in the output layer?
Question: What is the vectorized expression for the following equation to compute the gradients of the cost with respect to the pre-activations \(\mathbf{z}^{(M)}\) in the output layer?
\[ \frac{\partial C}{\partial z^{(M)}_i} = \frac{\partial C}{\partial a^{(M)}_i} \cdot \sigma^{(M)\prime}\!\bigl(z^{(M)}_i\bigr), \qquad i = 1, \ldots, D^{(M)}. \]
- \((\nabla_{\mathbf{a}^{(M)}} C)^\top (\sigma^{(M)\prime}(\mathbf{z}^{(M)}))\)
- \((\sigma^{(M)\prime}(\mathbf{z}^{(M)}))^\top (\nabla_{\mathbf{a}^{(M)}} C)\)
- \((\nabla_{\mathbf{a}^{(M)}} C) \odot (\sigma^{(M)\prime}(\mathbf{z}^{(M)}))\)
- \((\sigma^{(M)\prime}(\mathbf{z}^{(M)})) (\nabla_{\mathbf{a}^{(M)}} C)^\top\)
- \((\nabla_{\mathbf{a}^{(M)}} C) (\sigma^{(M)\prime}(\mathbf{z}^{(M)}))^\top\)
Answer:
The correct answer is (C).
The gradient of the cost with respect to the activations \(\nabla_{\mathbf{a}^{(M)}} C\) has shape \(\mathbb{R}^{D^{(M)}}\). The derivative of the activation function \(\sigma^{(M)\prime}(\mathbf{z}^{(M)})\) has shape \(\mathbb{R}^{D^{(M)}}\). The result \(\nabla_{\mathbf{z}^{(M)}} C\) is also in \(\mathbb{R}^{D^{(M)}}\). Because each index \(i\) contributes independently, the correct combination is the element-wise product \(\nabla_{\mathbf{a}^{(M)}} C \odot \sigma^{(M)\prime}(\mathbf{z}^{(M)})\).
Next, let’s consider the recursive case, which computes the gradients for the pre-activations \(\mathbf{z}^{(m)}\) for each layer \(m\) in the hidden layers. How would you vectorize the following equation to compute the gradients of the cost with respect to the pre-activations \(\mathbf{z}^{(m)}\) in the hidden layers?
Question: The hidden-layer pre-activation gradient satisfies
\[ \frac{\partial C}{\partial z^{(m)}_i} = \left( \sum_{k=1}^{D^{(m+1)}} \frac{\partial C}{\partial z^{(m+1)}_k} \cdot W^{(m+1)}_{k,i} \right) \cdot \sigma^{(m)\prime}\!\bigl(z^{(m)}_i\bigr), \qquad i = 1, \ldots, D^{(m)}. \]
What is the vectorized expression for the weighted sum in the brackets?
\[ \sum_{k=1}^{D^{(m+1)}} \frac{\partial C}{\partial z^{(m+1)}_k} \cdot W^{(m+1)}_{k,i}, \qquad i = 1, \ldots, D^{(m)}. \]
- \((\nabla_{\mathbf{z}^{(m+1)}} C)^\top (\mathbf{W}^{(m+1)})^\top\)
- \((\mathbf{W}^{(m+1)})^\top (\nabla_{\mathbf{z}^{(m+1)}} C)\)
- \((\nabla_{\mathbf{z}^{(m+1)}} C) \odot (\mathbf{W}^{(m+1)})^\top\)
- \(\mathbf{W}^{(m+1)} (\nabla_{\mathbf{z}^{(m+1)}} C)\)
- \((\nabla_{\mathbf{z}^{(m+1)}} C) (\mathbf{W}^{(m+1)})^\top\)
Answer:
The correct answer is (B).
The weight matrix \(\mathbf{W}^{(m+1)}\) has shape \(\mathbb{R}^{D^{(m+1)} \times D^{(m)}}\). The gradient of the cost with respect to the pre-activations \(\nabla_{\mathbf{z}^{(m+1)}} C\) has shape \(\mathbb{R}^{D^{(m+1)}}\). The result \(\nabla_{\mathbf{z}^{(m)}} C\) is expected to have shape \(\mathbb{R}^{D^{(m)}}\). Therefore, the correct way of multiplying the matrix and the vector together is \((\mathbf{W}^{(m+1)})^\top (\nabla_{\mathbf{z}^{(m+1)}} C)\).
In a previous exercise, we vectorized the equation to compute the gradients of the cost with respect to the pre-activations \(\mathbf{z}^{(m+1)}\) in each layer. The resulting expression uses the element-wise product.
Putting it together, the vectorized expression for the gradients of the cost with respect to the pre-activations \(\mathbf{z}^{(m)}\) in the hidden layers is:
\[ \nabla_{\mathbf{z}^{(m)}} C = (\mathbf{W}^{(m+1)})^\top (\nabla_{\mathbf{z}^{(m+1)}} C) \odot \sigma^{(m)\prime}(\mathbf{z}^{(m)}). \]
Next, we need to vectorize the equation to compute the gradients of the cost with respect to the weight matrix \(\mathbf{W}^{(m)}\) for each layer \(m\). How would you vectorize the following equation to compute the gradients of the cost with respect to the weight matrix \(\mathbf{W}^{(m)}\) and the bias vector \(\mathbf{b}^{(m)}\) for each layer \(m\)?
Question: What is the vectorized expression for the following equation to compute the gradients of the cost with respect to the weight matrix \(\mathbf{W}^{(m)}\) for each layer \(m\)?
\[ \frac{\partial C}{\partial W^{(m)}_{i,j}} = \frac{\partial C}{\partial z^{(m)}_i} \cdot a^{(m-1)}_j, \qquad i = 1, \ldots, D^{(m)},\; j = 1, \ldots, D^{(m-1)}. \]
- \((\nabla_{\mathbf{z}^{(m)}} C)^\top (\mathbf{a}^{(m-1)})\)
- \((\mathbf{a}^{(m-1)})^\top (\nabla_{\mathbf{z}^{(m)}} C)\)
- \((\nabla_{\mathbf{z}^{(m)}} C) \odot (\mathbf{a}^{(m-1)})\)
- \((\mathbf{a}^{(m-1)}) (\nabla_{\mathbf{z}^{(m)}} C)^\top\)
- \((\nabla_{\mathbf{z}^{(m)}} C) (\mathbf{a}^{(m-1)})^\top\)
Answer:
The correct answer is (E).
The gradient of the cost with respect to the pre-activations \(\nabla_{\mathbf{z}^{(m)}} C\) has shape \(\mathbb{R}^{D^{(m)}}\). The activations \(a^{(m-1)}_j\) have shape \(\mathbb{R}^{D^{(m-1)}}\). The result \(\nabla_{\mathbf{W}^{(m)}} C\) is expected to have shape \(\mathbb{R}^{D^{(m)} \times D^{(m-1)}}\). We want to produce a matrix using two vectors, and a natural way to do this is to use the outer product. Therefore, the correct way of multiplying them together is \((\nabla_{\mathbf{z}^{(m)}} C) (\mathbf{a}^{(m-1)})^\top\).
Therefore, a summary of the vectorized backward pass is:
- Compute the gradients for the output layer.
\[ \nabla_{\mathbf{z}^{(M)}} C = (\nabla_{\mathbf{a}^{(M)}} C) \odot \sigma^{(M)\prime}(\mathbf{z}^{(M)}). \]
- Compute the gradients for each layer recursively.
For each layer \(m = M-1, \ldots, 1\),
\[ \begin{aligned} \nabla_{\mathbf{z}^{(m)}} C &= (\mathbf{W}^{(m+1)})^\top (\nabla_{\mathbf{z}^{(m+1)}} C) \odot \sigma^{(m)\prime}(\mathbf{z}^{(m)}). \end{aligned} \]
- Compute the gradients for the weights and biases for each layer.
For each layer \(m = M-1, \ldots, 1\),
\[ \begin{aligned} \nabla_{\mathbf{W}^{(m)}} C &= (\nabla_{\mathbf{z}^{(m)}} C) (\mathbf{a}^{(m-1)})^\top, \\ \nabla_{\mathbf{b}^{(m)}} C &= \nabla_{\mathbf{z}^{(m)}} C. \end{aligned} \]
It is interesting to note that, although the backpropagation algorithm only has a handful of formulas, the vectorization of these formulas requires us to use all the different operations: matrix-vector product, element-wise product (or Hadamard product), and the outer product. This is therefore an excellent exercise to practice your vectorization skills.
9 Summary
Backpropagation makes large-scale neural network training feasible. The key insight is dynamic programming: instead of recomputing each gradient from scratch, the backward pass stores intermediate derivatives at every pre-activation node and reuses them for all weights feeding into that node. One backward pass — costing roughly the same compute as one forward pass — produces every weight and bias gradient simultaneously.
Three fundamental ideas from the course run through this chapter. Fundamental Idea #1 (learning is optimization) is the reason backpropagation exists at all: training a neural network means minimizing a cost function over millions of parameters, and backpropagation supplies the gradients that make that optimization tractable. Fundamental Idea #2 (balancing tradeoffs in sources of error) governs every practical choice once the algorithm is in hand: a network with more layers and units can fit the training data more closely, but risks overfitting; regularization and careful initialization are the tools we use to navigate that tradeoff. Fundamental Idea #3 (model evaluation is empirical) explains why architecture decisions — number of layers, units per layer, activation functions — have no universal answer: the only reliable guide is empirical evaluation on the specific problem and dataset.