Backpropagation Algorithm: Description with Worked Examples

Verified Sources

Nov 4, 2026

Backpropagation (often abbreviated backprop) is the standard algorithm used to compute gradients of a neural network’s loss with respect to its parameters (weights and biases). It does so efficiently by applying the chain rule of calculus from the output layer backward through the network layers, enabling gradient-based optimization such as gradient descent.

At a high level, backprop repeats this loop for training examples (or batches):

Forward pass: compute predictions from inputs through the network.
Loss computation: evaluate how wrong the predictions are using a loss function.
Backward pass: compute parameter gradients layer-by-layer using the chain rule.
Update step: move parameters in the direction that reduces the loss.

Key concepts you should internalize while reading the rest:

Forward propagation
Loss function
Gradient
Chain rule
Backpropagation

Backpropagation Explained (Neural Networks)

1) Mathematical setup (what backprop computes)

Consider a fully connected feedforward network with $L$ layers. For layer $l$ :

pre-activation (often called “logits” for the last layer): $z^{(l)} = W^{(l)} a^{(l-1)} + b^{(l)}$
activation: $a^{(l)} = f^{(l)}\!\left(z^{(l)}\right)$

Let the network output be $a^{(L)}=\hat{y}$ . For a single training example $(x,y)$ , define a scalar loss: $J(\theta) = \mathcal{L}(\hat{y}, y)$

Backprop’s goal is to compute: $\frac{\partial J}{\partial W^{(l)}} \quad \text{and} \quad \frac{\partial J}{\partial b^{(l)}} \quad \text{for all layers } l.$

Two additional technical objects are central:

Error signal
Activation derivative

In practice, we compute error signals $\delta^{(l)}$ and then use them to get gradients.

2) Core backprop recursion (output layer to input layer)

2.1 Output layer error

For the output layer $L$ : $\delta^{(L)} = \frac{\partial J}{\partial z^{(L)}}$

Using the chain rule: $\delta^{(L)} = \frac{\partial J}{\partial a^{(L)}} \odot f^{(L)\,'}\!\left(z^{(L)}\right)$ where $\odot$ is elementwise (Hadamard) product.

2.2 Hidden layers error

For a hidden layer $l$ : $\delta^{(l)} = \left(W^{(l+1)}\right)^{T}\delta^{(l+1)} \odot f^{(l)\,'}\!\left(z^{(l)}\right)$

This recurrence captures the “backward flow” of how output error affects earlier layers.

3) Gradients for weights and biases

Once you have $\delta^{(l)}$ , the parameter gradients follow directly:

$\frac{\partial J}{\partial W^{(l)}} = \delta^{(l)}\left(a^{(l-1)}\right)^{T}$

$\frac{\partial J}{\partial b^{(l)}} = \delta^{(l)}$

These formulas are why backprop is computationally efficient: you reuse forward-pass activations $a^{(l-1)}$ and propagate $\delta$ backward.

type="tip" title="Pro Tip" content="Think of $\delta^{(l)}$ as “how much the loss cares” about changing the pre-activations $z^{(l)}$ . Once you have $\delta$ , gradients for $W^{(l)}$ are just an outer product with the previous activations."

type="warning" title="Warning: Common implementation pitfall" content="For backprop, the derivative must match the chosen activation and loss. E.g., the form of $\delta^{(L)}$ differs for MSE vs. cross-entropy (especially when paired with softmax). Mixing them leads to wrong gradients."

Backpropagation training pass (single example)

1
Step 1
Compute $z^{(l)}=W^{(l)}a^{(l-1)}+b^{(l)}$ and $a^{(l)}=f^{(l)}(z^{(l)})$ until the output $\hat{y}=a^{(L)}$ .
2
Step 2
Evaluate $J=\mathcal{L}(\hat{y},y)$ .
3
Step 3
Compute $\delta^{(L)}=\frac{\partial J}{\partial z^{(L)}}=\frac{\partial J}{\partial a^{(L)}}\odot f^{(L)\,'}(z^{(L)})$ .
4
Step 4
For $l=L-1$ down to $1$ , compute $\delta^{(l)}=(W^{(l+1)})^{T}\delta^{(l+1)}\odot f^{(l)\,'}(z^{(l)})$ .
5
Step 5
For each layer: $\frac{\partial J}{\partial W^{(l)}}=\delta^{(l)}(a^{(l-1)})^{T}$ and $\frac{\partial J}{\partial b^{(l)}}=\delta^{(l)}$ .
6
Step 6
Apply gradient-based update, e.g., $W^{(l)}\leftarrow W^{(l)}-\eta\frac{\partial J}{\partial W^{(l)}}$ and $b^{(l)}\leftarrow b^{(l)}-\eta\frac{\partial J}{\partial b^{(l)}}$ .

4) Worked numeric example (small 2-layer network)

To make backprop concrete, consider a tiny network:

Inputs: $a^{(0)} = x = \begin{bmatrix}1\\2\end{bmatrix}$
Hidden layer ( $l=1$ ): 2 neurons, activation $f(z)=\sigma(z)=\\frac{1}{1+e^{-z}}$ $W^{(1)}=\begin{bmatrix}0.1 & 0.2\\ 0.4 & 0.3\end{bmatrix},\quad b^{(1)}=\begin{bmatrix}0.0\\0.1\end{bmatrix}$ $z^{(1)}=W^{(1)}a^{(0)}+b^{(1)}$
Output layer ( $l=2$ ): 1 neuron, activation identity $f(z)=z$ (so $a^{(2)}=z^{(2)}$ ) $W^{(2)}=\begin{bmatrix}0.5 & 0.6\end{bmatrix},\quad b^{(2)}=0.0$
Loss: mean squared error for a single target $y=1$ : $J=\frac{1}{2}(\\hat{y}-y)^2,\quad \\hat{y}=a^{(2)}$

4.1 Forward pass

Compute hidden pre-activations:

\begin{bmatrix} 0.1 & 0.2\\ 0.4 & 0.3 \end{bmatrix} \begin{bmatrix}1\\2\end{bmatrix} + \begin{bmatrix}0.0\\0.1\end{bmatrix} = \begin{bmatrix} 0.1(1)+0.2(2)\\ 0.4(1)+0.3(2) \end{bmatrix} + \begin{bmatrix}0.0\\0.1\end{bmatrix} = \begin{bmatrix}0.5\\1.1\end{bmatrix}$$ Hidden activations: $$a^{(1)}=\sigma(z^{(1)})= \begin{bmatrix}\sigma(0.5)\\\sigma(1.1)\end{bmatrix}

Approximate numerically:

$\sigma(0.5)\approx 0.62246$
$\sigma(1.1)\approx 0.75026$

So: $a^{(1)}\approx\begin{bmatrix}0.62246\\0.75026\end{bmatrix}$

Output pre-activation (and output since identity activation):

\begin{bmatrix}0.5 & 0.6\end{bmatrix} \begin{bmatrix}0.62246\\0.75026\end{bmatrix} \approx 0.31123+0.45016=0.76139$$ Thus $\hat{y}=0.76139$. Loss: $$J=\frac{1}{2}(0.76139-1)^2=\frac{1}{2}(-0.23861)^2\approx \frac{1}{2}(0.05694)\approx 0.02847$$ ### 4.2 Backward pass Output layer error: Since $J=\frac{1}{2}(a^{(2)}-y)^2$, we have: $$\frac{\partial J}{\partial a^{(2)}} = (a^{(2)}-y) = (0.76139-1)=-0.23861$$ and with identity activation $f(z)=z$, $f'(z)=1$, so: $$\delta^{(2)}=\frac{\partial J}{\partial z^{(2)}}=-0.23861$$ Gradients for output weights: $$\frac{\partial J}{\partial W^{(2)}}=\delta^{(2)}(a^{(1)})^{T} =-0.23861\begin{bmatrix}0.62246 & 0.75026\end{bmatrix} \approx \begin{bmatrix}-0.14848 & -0.17895\end{bmatrix}$$ Gradient for output bias: $$\frac{\partial J}{\partial b^{(2)}}=\delta^{(2)}=-0.23861$$ Hidden layer error: $$\delta^{(1)}=\left(W^{(2)}\right)^T\delta^{(2)}\odot \sigma'(z^{(1)})$$ First compute $(W^{(2)})^T\delta^{(2)}$: $$\left(W^{(2)}\right)^T\delta^{(2)}= \begin{bmatrix}0.5\\0.6\end{bmatrix}(-0.23861) \approx \begin{bmatrix}-0.11931\\-0.14317\end{bmatrix}$$ For sigmoid, $\sigma'(z)=\sigma(z)(1-\sigma(z))$: - $\sigma'(0.5)\approx 0.62246(1-0.62246)=0.23500$ - $\sigma'(1.1)\approx 0.75026(1-0.75026)=0.18745$ So: $$\delta^{(1)}\approx \begin{bmatrix}-0.11931\\-0.14317\end{bmatrix}\odot \begin{bmatrix}0.23500\\0.18745\end{bmatrix} = \begin{bmatrix}-0.02803\\-0.02682\end{bmatrix}$$ Gradients for hidden weights: $$\frac{\partial J}{\partial W^{(1)}}=\delta^{(1)}(a^{(0)})^T = \begin{bmatrix}-0.02803\\-0.02682\end{bmatrix} \begin{bmatrix}1 & 2\end{bmatrix} = \begin{bmatrix} -0.02803 & -0.05606\\ -0.02682 & -0.05364 \end{bmatrix}$$ Hidden bias gradients: $$\frac{\partial J}{\partial b^{(1)}}=\delta^{(1)}\approx \begin{bmatrix}-0.02803\\-0.02682\end{bmatrix}$$ This completes one full backprop iteration with explicit numbers.

5) Pseudocode and mini-batch training

For mini-batch size $m$ , backprop computes gradients per example and then averages (or sums) across the batch before updating parameters. Conceptually:

Mini-batch gradient descent Stochastic gradient descent (SGD)

Backpropagation in practice (training lifecycle)

Initialize parameters

Step 0

Choose starting weights $W^{(l)}$ and biases $b^{(l)}$ ."

Repeat over batches

Step 1

Run forward pass, compute loss, run backward pass."

Gradient computation

Step 2

Compute all $\delta^{(l)}$ , then $\partial J/\partial W^{(l)}$ and $\partial J/\partial b^{(l)}$ ."

Parameter update

Step 3

Apply optimizer update (SGD, Momentum, Adam, etc.)."

Stop criterion

Step 4

Stop when validation loss plateaus or after a fixed number of epochs."

6) Conceptual example: why “reverse mode” matters

If you attempted to compute all derivatives by perturbing parameters (numerical differentiation), you’d need many evaluations of the loss. Backprop instead computes gradients in one forward + one backward pass by reusing intermediate values from the forward pass (stored activations and pre-activations).

This is why backprop scales efficiently with network depth and number of parameters: the backward pass essentially applies the chain rule in reverse.

Common variants and edge cases

Knowledge Check

Question 1 of 3

Q1Single choice

In backpropagation, the error signal for a hidden layer is computed using which general relationship?

$\delta^{(l)} = (W^{(l+1)})^T\delta^{(l+1)} \odot f^{(l)\,'}(z^{(l)})$

$\delta^{(l)} = W^{(l+1)}\delta^{(l+1)} \odot f^{(l)\,'}(z^{(l)})$

$\delta^{(l)} = (W^{(l)})^T\delta^{(l+1)} \odot f^{(l+1)\,'}(z^{(l+1)})$

$\delta^{(l)} = \delta^{(l+1)} \odot f^{(l)\,'}(z^{(l)})$

Explore Related Topics

Breadth-First Search (BFS) and Depth-First Search (DFS): Concepts and Worked Examples

Deep Learning: Short Note with Core Concepts, Training, and Practice

How a Multilayer Network Learns Using Gradient Descent

Browse all research articles