Backpropagation from scratch on Mini-Batches

Implementation of Backpropagation algo on mini-batches with step by step execution of equations.

You must be thinking, another Backprop from scratch blog? Well kinda yes but I thought this through and came up with something that you can use to tinker around along with easy to understand equations that you usually write down to understand the algorithm.

This blog will focus on implementing the Backpropagation algorithm step-by-step on mini-batches of the dataset. There are plenty of tutorials and blogs to demonstrate the backpropagation algorithm in detail and all the logic behind calculus and algebra happening. So I’ll skip that part and cut to equations in math and implementation using Python (coz why not).

Why from scratch?

This has been a long time community question as to why we should implement an algorithm from scratch even if it’s been readily available to put to use by almost all frameworks. Evidently while using certain high-level frameworks you can’t even notice backpropagation doing its magic. To understand it upside down, in and out completely you should once try to make your hands dirty with this stuff. Backpropagation is something on which experimentation can be done while playing around.

Why Mini-Batches?

The reason behind mini-batches is simple. It saves memory and processing time by dividing data into mini-batches and supply the algorithm a fraction of the dataset on each iteration of the training loop. Feeding a 10000x10000 matrix at once would not only blow up memory but would take a long time to run. Instead, bringing it down to 50 per iteration would not only reduce memory usage but you can track progress.

Note-This is different from the stochastic method where we take a stratified sample from data for each class and train on that assuming the model would generalize.

Implementation time!

This is the head of the data I’ll be using for this implementation.

The target variable here is Occupancy which is a categorical variable (0/1).

This will be the architecture we’ll be coding.

Algorithm:

for i:=1 to i:=m:

1. Perform Forward propagation or Forward pass to calculate Activation values of neurons in each layer.

2. Backpropagation step:

• Calculate the error term (MSE or LogLoss or your wish) using the label in the data:

• Error terms in the hidden layers are calculated using: