How a Multilayer Network Learns Using Gradient Descent

A multilayer network learns by repeatedly adjusting its parameters so that its predictions become closer to the desired targets. The central training cycle combines four ideas:

1. Forward propagation computes a prediction.
2. A loss function measures prediction error.
3. Backpropagation computes how each parameter contributed to that error.
4. Gradient descent updates the parameters to reduce future error.

A multilayer perceptron (MLP) is a feed-forward network in which each layer typically performs an affine transformation followed by an activation function. For layer $l$:

$$\mathbf{z}^{(l)}=\mathbf{W}^{(l)}\mathbf{a}^{(l-1)}+\mathbf{b}^{(l)}$$ $$\mathbf{a}^{(l)}=g^{(l)}(\mathbf{z}^{(l)})$$

Here, $\mathbf{W}^{(l)}$ is the weight matrix, $\mathbf{b}^{(l)}$ is the bias vector, $\mathbf{z}^{(l)}$ is the pre-activation, $\mathbf{a}^{(l)}$ is the activation, and $g^{(l)}$ is the activation function.

The network does not directly “know” the correct weights. It begins with initialized parameters and learns through many small, directed changes determined by the gradient of the loss.

Footnotes

1. Multilayer perceptrons for digit recognition with Core APIs - TensorFlow explanation of dense layers, activations, and cross-entropy. ↩

2. Gradient Descent - Stanford CS231n discussion of loss optimization and gradient descent. ↩

1. The components of a learning problem

A supervised learning problem supplies training examples:

$$\mathcal{D}=\{(\mathbf{x}_i,\mathbf{y}_i)\}_{i=1}^{N}$$

where $\mathbf{x}_i$ is an input and $\mathbf{y}_i$ is its target output.

The network contains a collection of learnable parameters:

$$\boldsymbol{\theta}=\{\mathbf{W}^{(1)},\mathbf{b}^{(1)},\ldots,\mathbf{W}^{(L)},\mathbf{b}^{(L)}\}$$

The loss function evaluates the quality of the prediction. The training objective is to minimize the average loss:

$$J(\boldsymbol{\theta})= \frac{1}{N}\sum_{i=1}^{N} \mathcal{L}\left(f(\mathbf{x}_i;\boldsymbol{\theta}),\mathbf{y}_i\right)$$

For multiclass classification, a common choice is cross-entropy:

$$\mathcal{L}(\mathbf{y},\hat{\mathbf{y}}) = -\sum_{k=1}^{K}y_k\log(\hat{y}_k)$$

where $K$ is the number of classes, $y_k$ is the target indicator, and $\hat{y}_k$ is the predicted probability.

For regression, mean squared error is often used:

$$\mathcal{L}(\mathbf{y},\hat{\mathbf{y}}) = \frac{1}{2}\|\hat{\mathbf{y}}-\mathbf{y}\|^2$$

The network learns when the loss provides a signal that tells each parameter how its value should change.

Footnotes

1. Multilayer perceptrons for digit recognition with Core APIs - TensorFlow explanation of dense layers, activations, and cross-entropy. ↩

2. Forward propagation

During forward propagation, information moves from left to right through the network.

Consider a network with two hidden layers:

$$\mathbf{a}^{(0)}=\mathbf{x}$$ $$\mathbf{z}^{(1)}=\mathbf{W}^{(1)}\mathbf{x}+\mathbf{b}^{(1)}$$ $$\mathbf{a}^{(1)}=g^{(1)}(\mathbf{z}^{(1)})$$ $$\mathbf{z}^{(2)}=\mathbf{W}^{(2)}\mathbf{a}^{(1)}+\mathbf{b}^{(2)}$$ $$\mathbf{a}^{(2)}=g^{(2)}(\mathbf{z}^{(2)})$$ $$\mathbf{z}^{(3)}=\mathbf{W}^{(3)}\mathbf{a}^{(2)}+\mathbf{b}^{(3)}$$

The output layer then produces $\hat{\mathbf{y}}$. For classification, the output may use softmax:

$$\hat{y}_k= \frac{e^{z_k^{(3)}}} {\sum_{j=1}^{K}e^{z_j^{(3)}}}$$

The activations and pre-activations are stored because backpropagation needs them to calculate derivatives efficiently.

The nonlinear activation functions are essential. Without them, composing multiple affine transformations would still produce one affine transformation, so additional layers would not increase the network’s expressive power. Common choices include ReLU:

$$g(z)=\max(0,z)$$

and sigmoid:

$$\sigma(z)=\frac{1}{1+e^{-z}}$$

Footnotes

1. Waybackprop - TensorFlow explanation of forward computation, reverse differentiation, and stored activations. ↩

3. The loss landscape and the gradient

After the forward pass, the prediction is compared with the target. This produces a scalar loss $J$.

The gradient with respect to all parameters is:

$$\nabla_{\boldsymbol{\theta}}J = \left[ \frac{\partial J}{\partial \theta_1}, \frac{\partial J}{\partial \theta_2}, \ldots, \frac{\partial J}{\partial \theta_P} \right]$$

Each component answers a local sensitivity question:

If this parameter increases slightly, will the loss increase or decrease, and by how much?

For a single parameter $\theta_j$:

• $\frac{\partial J}{\partial \theta_j}>0$: increasing $\theta_j$ increases the loss, so it should generally be decreased.
• $\frac{\partial J}{\partial \theta_j}<0$: increasing $\theta_j$ decreases the loss, so it should generally be increased.
• $\frac{\partial J}{\partial \theta_j}\approx 0$: the parameter has little local effect on the current loss.

The negative gradient is the direction of steepest local decrease. Therefore, the basic gradient descent update is:

$$\boldsymbol{\theta}_{t+1} = \boldsymbol{\theta}_t - \eta\nabla_{\boldsymbol{\theta}}J(\boldsymbol{\theta}_t)$$

where $\eta>0$ is the learning rate and $t$ indexes the training step.

Footnotes

1. A Tutorial on Deep Learning Part 1 - Stanford tutorial covering backpropagation, stochastic gradient descent, and gradient behavior. ↩

4. Why backpropagation is needed

A multilayer network is a nested composition of functions:

$$\hat{\mathbf{y}} = f^{(L)} \left( f^{(L-1)} \left( \cdots f^{(2)} \left( f^{(1)}(\mathbf{x}) \right) \cdots \right) \right)$$

The loss depends directly on the output, but it depends indirectly on parameters in earlier layers. To find the effect of an early parameter, the chain rule is applied through every subsequent computation.

For a parameter $\theta$ in layer $l$:

$$\frac{\partial J}{\partial \theta} = \frac{\partial J}{\partial \mathbf{a}^{(L)}} \frac{\partial \mathbf{a}^{(L)}}{\partial \mathbf{z}^{(L)}} \cdots \frac{\partial \mathbf{a}^{(l)}}{\partial \mathbf{z}^{(l)}} \frac{\partial \mathbf{z}^{(l)}}{\partial \theta}$$

Backpropagation evaluates these derivatives from the output layer backward. It reuses intermediate results rather than separately differentiating the complete network for every parameter.2

For a dense layer:

$$\mathbf{z}^{(l)} = \mathbf{W}^{(l)}\mathbf{a}^{(l-1)} + \mathbf{b}^{(l)}$$

define the error signal:

$$\boldsymbol{\delta}^{(l)} = \frac{\partial J}{\partial \mathbf{z}^{(l)}}$$

Then:

$$\frac{\partial J}{\partial \mathbf{W}^{(l)}} = \boldsymbol{\delta}^{(l)} \left(\mathbf{a}^{(l-1)}\right)^T$$ $$\frac{\partial J}{\partial \mathbf{b}^{(l)}} = \boldsymbol{\delta}^{(l)}$$

and the error signal for the previous layer is:

$$\boldsymbol{\delta}^{(l-1)} = \left(\mathbf{W}^{(l)}\right)^T \boldsymbol{\delta}^{(l)} \odot g'^{(l-1)}\left(\mathbf{z}^{(l-1)}\right)$$

where $\odot$ denotes elementwise multiplication.

This recurrence explains how an output error is transmitted backward and converted into gradients for each layer.

Footnotes

1. Waybackprop - TensorFlow explanation of forward computation, reverse differentiation, and stored activations. ↩

2. Backpropagation - Stanford CS231n explanation of the chain rule and efficient reverse-mode gradient computation. ↩

5. Deriving the update for a two-layer network

Consider a two-layer network:

$$\mathbf{z}^{(1)}=\mathbf{W}^{(1)}\mathbf{x}+\mathbf{b}^{(1)}$$ $$\mathbf{a}^{(1)}=g(\mathbf{z}^{(1)})$$ $$\mathbf{z}^{(2)}=\mathbf{W}^{(2)}\mathbf{a}^{(1)}+\mathbf{b}^{(2)}$$ $$\hat{\mathbf{y}}=s(\mathbf{z}^{(2)})$$

where $s$ may be softmax.

For the output layer, calculate:

$$\boldsymbol{\delta}^{(2)} = \frac{\partial J}{\partial \mathbf{z}^{(2)}}$$

For softmax combined with cross-entropy, this simplifies to:

$$\boldsymbol{\delta}^{(2)} = \hat{\mathbf{y}}-\mathbf{y}$$

The output-layer gradients are:

$$\frac{\partial J}{\partial \mathbf{W}^{(2)}} = \boldsymbol{\delta}^{(2)} \left(\mathbf{a}^{(1)}\right)^T$$ $$\frac{\partial J}{\partial \mathbf{b}^{(2)}} = \boldsymbol{\delta}^{(2)}$$

Propagate the error into the hidden layer:

$$\boldsymbol{\delta}^{(1)} = \left(\mathbf{W}^{(2)}\right)^T \boldsymbol{\delta}^{(2)} \odot g'(\mathbf{z}^{(1)})$$

Then calculate:

$$\frac{\partial J}{\partial \mathbf{W}^{(1)}} = \boldsymbol{\delta}^{(1)} \mathbf{x}^T$$ $$\frac{\partial J}{\partial \mathbf{b}^{(1)}} = \boldsymbol{\delta}^{(1)}$$

Finally, update:

$$\mathbf{W}^{(1)} \leftarrow \mathbf{W}^{(1)} - \eta \frac{\partial J}{\partial \mathbf{W}^{(1)}}$$ $$\mathbf{b}^{(1)} \leftarrow \mathbf{b}^{(1)} - \eta \frac{\partial J}{\partial \mathbf{b}^{(1)}}$$

and similarly for $\mathbf{W}^{(2)}$ and $\mathbf{b}^{(2)}$.

The important point is that the hidden layer is not given a direct target. Its learning signal is inferred from how its activations affect later layers and, ultimately, the loss.

6. Batch, stochastic, and mini-batch gradient descent

The exact objective averages loss over all $N$ training examples:

$$J(\boldsymbol{\theta}) = \frac{1}{N} \sum_{i=1}^{N} \mathcal{L}_i(\boldsymbol{\theta})$$

Three common estimation strategies are used:

For a mini-batch $\mathcal{B}$:

$$\widehat{\nabla J} = \frac{1}{|\mathcal{B}|} \sum_{i\in\mathcal{B}} \nabla_{\boldsymbol{\theta}}\mathcal{L}_i$$

The update becomes:

$$\boldsymbol{\theta}_{t+1} = \boldsymbol{\theta}_t - \eta\widehat{\nabla J}$$

Mini-batches enable vectorized computation and produce a gradient estimate that is usually less noisy than a single-example update. The noise in stochastic or mini-batch updates can sometimes help optimization move away from shallow or unfavorable regions, but it also makes the loss curve fluctuate.

Footnotes

1. A Tutorial on Deep Learning Part 1 - Stanford tutorial covering backpropagation, stochastic gradient descent, and gradient behavior. ↩

7. What the network is actually learning

A multilayer network learns a hierarchy of representations.

• Early layers may learn simple combinations of input features.
• Intermediate layers combine these into more useful patterns.
• Later layers transform those representations into task-specific outputs.

This behavior is not programmed as a list of explicit rules. It emerges because gradient descent changes parameters in whatever way reduces the loss on the training data.

The network’s prediction can be viewed as a function:

$$f(\mathbf{x};\boldsymbol{\theta})$$

Training searches for parameter values that produce low empirical risk:

$$\boldsymbol{\theta}^{*} \approx \arg\min_{\boldsymbol{\theta}}J(\boldsymbol{\theta})$$

Because neural-network loss surfaces are generally non-convex, gradient descent is not guaranteed to find the globally best parameter configuration. Nevertheless, the repeated local updates are effective in many practical settings.

Footnotes

1. A Tutorial on Deep Learning Part 1 - Stanford tutorial covering backpropagation, stochastic gradient descent, and gradient behavior. ↩

8. Learning-rate schedules and adaptive optimizers

The basic update uses a fixed $\eta$, but practical training often changes the learning rate over time. A schedule may reduce the step size as optimization approaches a useful region.

A simple decay rule is:

$$\eta_t=\frac{\eta_0}{1+kt}$$

where $\eta_0$ is the initial learning rate and $k$ controls decay.

Momentum adds a running direction to reduce oscillation:

$$\mathbf{v}_t = \beta\mathbf{v}_{t-1} + (1-\beta)\nabla J_t$$ $$\boldsymbol{\theta}_{t+1} = \boldsymbol{\theta}_t-\eta\mathbf{v}_t$$

Adaptive methods such as Adam maintain estimates of the first and second moments of gradients:

$$\mathbf{m}_t = \beta_1\mathbf{m}_{t-1} + (1-\beta_1)\mathbf{g}_t$$ $$\mathbf{v}_t = \beta_2\mathbf{v}_{t-1} + (1-\beta_2)\mathbf{g}_t^2$$

with bias-corrected estimates used to scale parameter updates. These methods remain gradient-based: backpropagation still supplies the derivatives, while the optimizer determines how to use them.

Footnotes

1. Introduction to gradients and automatic differentiation - TensorFlow guide connecting automatic differentiation with neural-network training. ↩

9. A compact pseudocode implementation

The essential training loop can be expressed as follows:

1initialize parameters theta
2
3for epoch in range(number_of_epochs):
4    shuffle(training_data)
5
6    for X_batch, Y_batch in mini_batches:
7        # Forward propagation
8        activations = forward(X_batch, theta)
9
10        # Loss
11        loss = compute_loss(activations.output, Y_batch)
12
13        # Backpropagation
14        gradients = backward(
15            X_batch,
16            Y_batch,
17            activations,
18            theta
19        )
20
21        # Gradient-descent update
22        for parameter in theta:
23            parameter -= learning_rate * gradients[parameter]

Automatic differentiation systems perform the derivative bookkeeping, but the conceptual sequence remains forward computation, loss evaluation, reverse differentiation, and parameter update.

Footnotes

1. Multilayer perceptrons for digit recognition with Core APIs - TensorFlow training-loop example showing batch loss, gradients, and parameter updates. ↩

10. How to evaluate whether learning is working

Training loss should generally decrease, but loss alone is insufficient. A robust evaluation separates:

• Training data: used to calculate gradients.
• Validation data: used to select settings and detect overfitting.
• Test data: held out for final evaluation.

Useful diagnostics include:

1. Plot training and validation loss by epoch.
2. Compare training and validation accuracy where appropriate.
3. Inspect gradient norms.
4. Check whether activations are saturated or inactive.
5. Verify that labels and output dimensions are correct.
6. Compare against a simple baseline.

A typical healthy pattern is decreasing training loss with validation loss that decreases initially and then levels off. If training loss continues to fall while validation loss rises, the model is likely overfitting.

11. Summary model

A multilayer network learns through the following mathematical cycle:

$$\text{Input} \rightarrow \text{Forward propagation} \rightarrow \text{Prediction} \rightarrow \text{Loss} \rightarrow \text{Backpropagation} \rightarrow \text{Gradient} \rightarrow \text{Parameter update}$$

In compact form:

$$\boldsymbol{\theta}_{t+1} = \boldsymbol{\theta}_{t} - \eta \nabla_{\boldsymbol{\theta}} \left[ \frac{1}{|\mathcal{B}|} \sum_{i\in\mathcal{B}} \mathcal{L} \left( f(\mathbf{x}_i;\boldsymbol{\theta}_t), \mathbf{y}_i \right) \right]$$

The network improves because each update uses the current error to modify every parameter in a direction expected to reduce that error. Backpropagation supplies efficient credit assignment across layers, while gradient descent supplies the optimization rule that turns those gradients into learning.3

Footnotes

1. Gradient Descent - Stanford CS231n discussion of loss optimization and gradient descent. ↩

2. Waybackprop - TensorFlow explanation of forward computation, reverse differentiation, and stored activations. ↩

3. Backpropagation - Stanford CS231n explanation of the chain rule and efficient reverse-mode gradient computation. ↩