Backpropagation

The Essence of Machine Learning

Given the structural nature of Neural Networks and the ideas of Gradient Descent in Machine Learning, designing an efficient system to compute the gradient vector and therefore minimize the cost function is paramount.

The heart of Backpropagation lies in the partial derivatives of either $w$ or $b$ in respect to the cost function. Intuitively, this tells us how quickly the cost changes when we change the weights and biases of the neurons. Concretely, we want to compute partial derivatives $\frac{\delta C}{\delta w}$ and $\frac{\delta C}{\delta b}$. We will be using the same cost function (MSE) described in the gradient descent note:

The added notation here $a^L(x)$ is meant to generalize the vector of activations output from the network when $x$ in an input and it has $L$ many layers. Now, for Backprop to work we need two identities to hold:

1. The cost function can be written as an average of cost functions, allowing us to average PDEs across smaller training runs
2. The cost can be written as a function of the outputs from the neural network. Obvious.

The Kernel of Backpropagation

Now we must inevitably deal with the mathematical structure of backpropogation. Conceptually, so far, the idea is simple, we want to minimize costs by changing the weights and biases until we can't minimize further. This, as we have discussed, is done through PDEs. However, in real backprop algorithms its not that simple. We know for sure that we want to compute PDEs $\frac{\delta C}{\delta w^{l}_{jk}}$ and $\frac{\delta C}{\delta b^{l}_{j}}$, but in order to do this we need an intermediate value known as the error. $\delta ^{l}_{j}$ represents the vector of error in This