Model Compression: needs and importance

Model Compression:needs and importance

Learn about the needs and benefits of different model compression techniques for deep learning

Whether you’re new to computer vision or an expert, you’ve probably heard about AlexNet winning the ImageNet challenge in 2012. That was the turning point in computer vision history because it showed that deep learning models can perform tasks which were considered very difficult for computers, with an unprecedented level of accuracy.

But did you know that AlexNet had 62 million trainable parameters?

Interesting right.

Another popular model VGGNet which came out in 2014 had even more, 138 million trainable parameters.

That’s more than 2 times that of AlexNet.

You might be thinking… I know that the deeper the model is, the better it will perform. So why are you highlighting the number of parameters? Deeper the network, it is obvious that there will be more parameters.

Sure, these deep models have been benchmarks in the computer vision industry. But when you want to create a real-world application, would you choose these models?

I guess the real question we should ask here is: CAN YOU USE THESE MODELS IN YOUR APPLICATION?

Hold that thought for just a minute!

Let me divert here for a bit, before I get to the answer. (But feel free to skip to the end.)

The number of IoT devices is expected to reach 125–500 Billion by 2030 and assuming that 20% of them will have cameras, IoT devices with cameras is a 13–100 billion unit market. [9,10,11]

IoT camera devices include home security cameras (such as Amazon Ring and Google Nest) that open the door when you reach home or notify you if it sees an unknown person, cameras on smart vehicles that assist your driving, or cameras at a parking lot that open the gate when you enter or exit, just to name a few! Some of these IoT devices are already using AI to some extent and others are catching up slowly.

Many real-world applications demand real-time, on device processing capabilities. A self-driving car is a perfect example of this. In order for cars to drive down any road safely, they must observe the road in real-time and stop if a person walks in front of the car. In such a case, processing visual information and making a decision needs to be done in real-time, on device.

So, returning to the earlier question: CAN YOU USE THESE MODELS IN YOUR APPLICATION?

If you’re using Computer Vision, there’s a high chance your application requires an IoT device, and looking at the forecast for the IoT devices, you’re in good company.

The main challenge is that IoT devices are resource constrained; they have limited memory and low compute power. The more trainable parameters in a model, the bigger its size. Inference time of a deep learning model increases along with the increase in number of trainable parameters. Moreover, models with high parameters require more energy and space in comparison to a smaller network with fewer parameters. The end result is that when the size of the model is big, it’s difficult to deploy on resource-constrained devices. While these models have been successful in achieving great results in a lab, they aren’t usable in many real-world applications.

In the lab, you have expensive and high-speed GPUs to get this level of performance [1], but when you deploy in the real-world the cost, power, heat and other issues preclude the “just throw more iron at it” strategy.

Deploying deep learning models on the cloud is an option as it can provide high computational and storage availability. However, it will have poor response times due to network latency, which is unacceptable in many real-time applications (and don’t get me started on the network connectivity’s impact on overall reliability, or privacy!).

In short, AI needs to process close to the data source, preferably on the IoT device itself !

That leaves us with one option: Reducing the size of the model.

Making a smaller model that can run under the constraints of the edge-devices is a key challenge. And that too without compromising on accuracy. It is just not enough to have a small model that can run on resource constrained devices. It should perform well, both in terms of accuracy and inference speed.

So how do you fit these models on limited devices? How do you make them usable in real-world applications?

Here are a few techniques that can be used to reduce the model size so that you can deploy them on your IoT device.

Pruning

Pruning reduces the number of parameters by removing redundant, unimportant connections that are not sensitive to performance. This not only helps reduce the overall model size but also saves on computation time and energy.

Pros:

• Can be applied during or after training
• Can improve the inference time/ model size vs accuracy tradeoff for a given architecture [12]
• Can be applied to both convolutional and fully connected layers

Cons:

• Generally, does not help as much as switching to a better architecture [12]
• Implementations that benefit latency are rare as TensorFlow’s only brings model size benefits

Quantization

In DNN, weights are stored as 32-bit floating-point numbers. Quantization is the idea of representing these weights by reducing the number of bits. The weights can be quantized to 16-bit, 8-bit, 4-bit or even with 1-bit. By reducing the number of bits used, the size of the deep neural network can be significantly reduced.

Pros:

• Quantization can be applied both during and after training
• Can be applied to both convolutional and fully connected layers

Cons:

• Quantized weights make neural networks harder to converge. A smaller learning rate is needed to ensure the network to have good performance. [13]
• Quantized weights make back-propagation infeasible since gradient cannot back-propagate through discrete neurons. Approximation methods are needed to estimate the gradients of the loss function with respect to the input of the discrete neurons [13]
• TensorFlow’s quantize-aware training does not do any quantization during the training itself. Only statistics are gathered during training and those are used to quantize post training. So I am not sure if the above points should be included as cons

Knowledge distillation

In knowledge distillation, a large, complex model is trained on a large dataset. When this large model can generalize and perform well on unseen data, it is transferred to a smaller network. The larger model is also known as the teacher model and the smaller network is also known as the student network.

Pros:

• If you have a pre-trained teacher network, less training data required to train the smaller (student) network.
• If you have a pre-trained teacher network, training of the smaller (student) network is faster.
• Can downsize a network regardless of the structural difference between the teacher and the student network.

Cons:

• If you do not have a pre-trained teacher network, it may require a larger dataset and take more time to train it.

Selective Attention

Selective attention is the idea of focusing on objects or elements of interest, while discarding the others (often background or other task-irrelevant objects). It is inspired by the biology of the human eye. When we look at something, we only focus on one or a few objects at a time, and other regions are blurred out.

This requires adding a selective attention network upstream of your existing AI system or using it by itself if it serves your purpose. It depends on the problem you are trying to solve.

Pros:

• Faster inference
• Smaller model (e.g. a face detector and cropper that’s only 44 KB!)
• Accuracy gain (by focusing downstream AI on only the regions/objects of interest)

Cons:

• Supports only training from scratch

Low-rank factorization

Uses matrix/tensor decomposition to estimate the informative parameters. A weight matrix A with m x n dimension and having a rank r is replaced by smaller dimension matrices. This technique helps by factorizing a large matrix into smaller matrices.

Pros:

• Can be applied during or after training
• Can be applied to both convolutional and fully connected layers
• When applied during training, can reduce training time