Backpropagation: How Neural Networks Learn via Gradient-Based Weight Adjustment

Verified Sources

Nov 4, 2026

Backpropagation is the training method that makes a neural network learn by computing how each parameter (weight and bias) should change to reduce a chosen loss function. Concretely, backpropagation uses the chain rule to compute gradients of the loss with respect to each layer’s parameters efficiently, then applies an optimization step such as gradient descent.

A typical training iteration has two phases:

Forward pass: compute predictions and the loss.
Backward pass: propagate “blame” (error signal) from the loss back through layers to get gradients.
Parameter update: update weights so the loss decreases.

How backpropagation connects “error” to “weight updates”

For parameters $\theta$ (e.g., all weights and biases), training tries to reduce a scalar objective $L(\theta)$ . Gradient-based learning updates parameters in the opposite direction of the gradient:

\theta \leftarrow \theta - \eta \nabla_{\theta} L

where $\eta$ is the learning rate. Backpropagation is the mechanism that efficiently computes $\nabla_{\theta}L$ for multi-layer networks by reusing intermediate derivatives from the forward pass.

Mini example setup (2 layers + loss)

We’ll build a minimal network with:

Layer 1: affine transform + nonlinearity
Layer 2: affine transform to produce a scalar prediction
Loss: mean squared error (MSE) for simplicity

This keeps the math small while still showing the core idea: gradients flow backward, determining exactly how weights change.

Key terms introduced here:

Activation function
Backpropagation
Gradient

Backpropagation Intuition (Chain Rule + Gradient Descent)

Two-layer backpropagation with a concrete weight-update example

1
Step 1
Let $x=1$ be the input and the target be $y=1$ . Use a 1-hidden-unit, 2-layer network:

Layer 1: $h = \sigma(z_1)$ where $z_1 = w_1 x + b_1$

Layer 2: $\hat{y} = z_2$ where $z_2 = w_2 h + b_2$

Choose sigmoid $\sigma(t)=\frac{1}{1+e^{-t}}$ .

Initialize: $w_1=0.5,\; b_1=0.0,\; w_2=1.0,\; b_2=0.0$ .
2
Step 2
Use mean squared error (single example):
$L = (\hat{y}-y)^2$
3
Step 3
Compute:

$z_1 = w_1x+b_1 = 0.5\cdot 1 + 0 = 0.5$

$h=\sigma(0.5)\approx 0.622459$

$\hat{y} = w_2h+b_2 = 1.0\cdot 0.622459 + 0 = 0.622459$

$L=(0.622459-1)^2\approx 0.14216$
4
Step 4
First derivative of loss w.r.t. prediction:
$\frac{\partial L}{\partial \hat{y}} = 2(\hat{y}-y)$
Numerically:
$\frac{\partial L}{\partial \hat{y}} = 2(0.622459-1) \approx -0.755082$
5
Step 5
Since $\hat{y}=w_2h+b_2$ :
$\frac{\partial L}{\partial w_2} = \frac{\partial L}{\partial \hat{y}}\cdot \frac{\partial \hat{y}}{\partial w_2} = \frac{\partial L}{\partial \hat{y}}\cdot h$ $\frac{\partial L}{\partial b_2} = \frac{\partial L}{\partial \hat{y}}\cdot 1$
Numerically:
$\frac{\partial L}{\partial w_2} \approx (-0.755082)\cdot 0.622459 \approx -0.470007$ $\frac{\partial L}{\partial b_2} \approx -0.755082$
6
Step 6
Compute derivative of loss w.r.t. hidden activation $h$ :
$\frac{\partial L}{\partial h} = \frac{\partial L}{\partial \hat{y}}\cdot \frac{\partial \hat{y}}{\partial h} = \frac{\partial L}{\partial \hat{y}}\cdot w_2$
So:
$\frac{\partial L}{\partial h} \approx (-0.755082)\cdot 1.0 = -0.755082$
7
Step 7
For sigmoid, $\sigma'(t)=\sigma(t)(1-\sigma(t))$ .
$\frac{\partial h}{\partial z_1} = h(1-h)$
Numerically:
$\frac{\partial h}{\partial z_1} \approx 0.622459(1-0.622459) \approx 0.235004$
8
Step 8
Since $z_1 = w_1x+b_1$ :
$\frac{\partial L}{\partial z_1} = \frac{\partial L}{\partial h}\cdot \frac{\partial h}{\partial z_1}$ $\frac{\partial L}{\partial w_1} = \frac{\partial L}{\partial z_1}\cdot x$ $\frac{\partial L}{\partial b_1} = \frac{\partial L}{\partial z_1}\cdot 1$
Numerically:
$\frac{\partial L}{\partial z_1} \approx (-0.755082)\cdot 0.235004 \approx -0.177996$ $\frac{\partial L}{\partial w_1} \approx -0.177996\cdot 1 = -0.177996$ $\frac{\partial L}{\partial b_1} \approx -0.177996$
9
Step 9
Choose learning rate $\eta=0.1$ .

Gradient descent update:
$\theta \leftarrow \theta - \eta \frac{\partial L}{\partial \theta}$
Updates:

$w_2 \leftarrow 1.0 - 0.1(-0.470007)=1.0470007$

$b_2 \leftarrow 0.0 - 0.1(-0.755082)=0.0755082$

$w_1 \leftarrow 0.5 - 0.1(-0.177996)=0.5177996$

$b_1 \leftarrow 0.0 - 0.1(-0.177996)=0.0177996$

Interpretation: because the prediction is too low ( $\hat{y}<y$ ), the gradients push weights in the direction that increases $\hat{y}$ in the next iteration.

Weight-adjustment interpretation (why signs make sense)

In our example, $\\hat{y}=0.622<1$ , so we want to increase the prediction. The key quantity is

\\frac{\\partial L}{\\partial \\hat{y}} = 2(\\hat{y}-y) < 0

A negative gradient means increasing $\\hat{y}$ would decrease the loss. Because

$\\partial L/\\partial w_2 = (\\partial L/\\partial \\hat{y})\\cdot h$ is also negative, gradient descent increases $w_2$ .
the error then backpropagates through $h=\\sigma(z_1)$ to influence $w_1$ and $b_1$ .

This is the core mechanism: backpropagation turns the mismatch at the output into a layer-by-layer gradient signal, which the optimizer converts into weight adjustments.

loss function and gradient descent together form the learning loop.

Pro Tip: Track gradients through the computational graph

Write the forward computation as a small graph (like Layer 1 → Layer 2 → Loss). Then in the backward pass, multiply derivatives using the chain rule. This prevents sign mistakes and clarifies how each parameter affects the loss.

Beware of vanishing gradients with saturating activations

If you use sigmoid/tanh, their derivatives $\sigma'(t)$ can become small when the units saturate. Backpropagated gradients can shrink across many layers, slowing or stopping learning.

Backpropagation training workflow

Forward pass

Step A

Compute layer outputs and the loss for the current mini-batch."

Compute output gradient

Step B

Differentiate the loss with respect to predictions."

Backpropagate through layers

Step C

Use the chain rule to get gradients for each weight/bias."

Update parameters

Step D

Apply $\theta \leftarrow \theta - \eta \ abla_\theta L$ ."

Repeat until convergence

Step E

Repeat over batches/epochs to reduce loss and improve accuracy."

Numerical gradients in the two-layer example

Negative gradients imply gradient descent increases the corresponding parameter.

Common questions about backpropagation

Backpropagation quick recall

1 / 5

Question · Term

What does backpropagation compute?

Click to reveal

Answer · Definition

Gradients of the loss w.r.t. each parameter, using the chain rule through the network.

Knowledge Check

Question 1 of 4

Q1Single choice

In backpropagation, what is the main mathematical idea used to compute gradients?

Apply the chain rule to propagate derivatives from the loss backward through the network

Approximate gradients using random perturbations only

Compute gradients only for the last layer, ignoring earlier layers

Replace the loss function with a simpler surrogate and never use derivatives

Explore Related Topics

Learning: Meaning, Learning Methods, and Rule-Based Systems (Pros & Cons)

Bayesian Networks for Uncertainty: Concepts, Inference, and Practical Use

Logistic Regression: Model, Training, and Worked Examples

Browse all research articles