Backpropagation and the maths behind it

Caution! This article is 4 years old. It may be obsolete or show old techniques. It may also still be relevant, and you may find it useful! So it has been marked as deprecated, just in case.

The process of moving the data forward through the network is called forward propagation. Backpropagation is the tool that gradient descent uses to calculate the gradient of the loss function with respect to each weight.

The value of an output node is the weighted sum of the outputs of the previous layer, after passing through the activation function of its own layer:


node output = activation ( weighted sum of inputs )
            = activation ( a1w1 + a2w2 + ... + aNwN )

Therefore, if we want to update the values of the output nodes, we can update their weights. Another way is by changing the activation output from the previous layer.

We can't directly change the activation output because it's a calculation based on the weights and the previous layer's output. But, we can indirectly change this layer's output by jumping backwards and updating the previous weights.

We continue this process until we reach the input layer.

Mathematical representation

The notation supposes that we have:

L layers with index l in the range l = [0, L - 1]
n nodes in a layer l with index j in the range j = [0, n - 1]
m nodes in the previous layer l - 1 with index k in the range k = [0, m - 1]

Loss function

For any node j in the output layer L, if the activation output is a_j^L, and the actual output is y_j, then the loss C of training sample 0 is:

C_{0 j} = (a_{j}^{L} - y_{j})^{2}

where y_j is constant. The total loss will be the sum of the loss of all nodes:

C_{0} = \sum_{j = 0}^{n - 1} C_{0 j} = \sum_{j = 0}^{n - 1} (a_{j}^{L} - y_{j})

This is what we have seen already when we covered the loss function and learning rate.

Input and output

For any node j in any layer l, the input z_j^l is the weighted sum of the activation outputs of all nodes in layer l - 1, a_k^{l - 1}, where the weights are w_jk^l.

z_{j}^{l} = \sum_{k = 0}^{m - 1} w_{j k}^{l} a_{k}^{l - 1}

The output of node j in layer l will be the activation function of layer l g^l applied to this weighted sum:

a_{j}^{l} = g^{l} (z_{j}^{l})

This is what we have seen already when we covered the introduction to artificial neural networks.

Differenciation

The total loss of the network for a single input 0 is a composition of functions:

C_{0} = C_{0} (a_{j}^{l} (z_{j}^{l} (w_{j}^{l})))

To differentiate a composition of functions, we use the chain rule:

\frac{\partial C_{0}}{\partial w_{j k}^{l}} = \frac{\partial C_{0}}{\partial a_{j}^{l}} \frac{\partial a_{j}^{l}}{\partial z_{j}^{l}} \frac{\partial z_{j}^{l}}{\partial w_{j k}^{l}}

The first term is calculated differently for the output layer and the hidden layers. This is because the loss function is a direct function of the outputs in the last layer, but is an indirect function of the outputs in the hidden layers. So the derivative will be different.

Output layer

Let's check the values of the three terms for l = L, j = 1 and k = 2

$\frac{\partial C_{0}}{\partial a_{j}^{L}} = \frac{\partial}{\partial a_{j}^{L}} \sum_{j = 0}^{n - 1} (a_{j}^{L} - y_{j})^{2} = 2 (a_{1}^{L} - y_{1})$
$\frac{\partial a_{j}^{l}}{\partial z_{j}^{l}} = \frac{\partial g^{L}}{\partial z_{1}^{L}}$
$\frac{\partial z_{j}^{l}}{\partial w_{j k}^{l}} = \frac{\partial}{\partial w_{j k}^{l}} \sum_{k = 0}^{m - 1} w_{j k}^{l} a_{k}^{l - 1} = a_{2}^{L - 1}$

Putting it all together, the partial derivative of the loss with respect to one weight, for a single training sample 0 with j = 1 and k = 2 in the output layer L is:

\frac{\partial C_{0}}{\partial w_{12}^{L}} = 2 (a_{1}^{L} - y_{1}) \frac{\partial g^{L}}{\partial z_{1}^{L}} a_{2}^{L - 1}

If we have a total of N training samples, the derivative of the loss for all of them with respect to one weight in the output layer is the average sum:

\frac{\partial C}{\partial w_{12}^{L}} = \frac{1}{N} \sum_{i = 0}^{N - 1} \frac{\partial C_{i}}{\partial w_{12}^{L}}

Hidden layers

For the hidden layers we use the chain rule as before, but now the first term has to be calculated differently.

For l = L - 1, j = 2 and k = 2:

\frac{\partial C_{0}}{\partial w_{22}^{L - 1}} = \frac{\partial C_{0}}{\partial a_{2}^{L - 1}} \frac{\partial a_{2}^{L - 1}}{\partial z_{2}^{L - 1}} \frac{\partial z_{2}^{L - 1}}{\partial w_{22}^{L - 1}}

In the first term, we need to sum in j because a change in a₂^{L - 1} will affect all nodes in the layer L:

\frac{\partial C_{0}}{\partial w_{22}^{L - 1}} = (\sum_{j = 0}^{n - 1} \frac{\partial C_{0}}{\partial a_{j}^{L}} \frac{\partial a_{j}^{L}}{\partial z_{j}^{L}} \frac{\partial z_{j}^{L}}{\partial a_{2}^{L - 1}}) \frac{\partial a_{2}^{L - 1}}{\partial z_{2}^{L - 1}} \frac{\partial z_{2}^{L - 1}}{\partial w_{22}^{L - 1}}

The first two terms of the first term are calculated as before. The third is new and is calculated like this:

\frac{\partial z_{j}^{L}}{\partial a_{2}^{L - 1}} = \frac{\partial}{\partial a_{2}^{L - 1}} \sum_{k = 0}^{m - 1} w_{j k}^{L} a_{k}^{L - 1} = w_{j 2}^{L}

Putting it all together:

\frac{\partial C_{0}}{\partial w_{22}^{L - 1}} = (\sum_{j = 0}^{n - 1} 2 (a_{j}^{L} - y_{j}) \frac{\partial g^{L}}{\partial z_{j}^{L}} w_{j 2}^{L}) \frac{\partial g^{L - 1}}{\partial z_{2}^{L - 1}} a_{2}^{L - 2}

These gradients will be used to update the weights as we saw when explaining training and learning:

new weight = old weight - (learning rate * gradient)