5 Awesome Projects to Hone Your Deep Learning Skills

Learn by doing

Deep learning is a very diverse field where things move quickly. In order to stay on top of your game, you need to constantly improve your skills. The most effective way of learning is by doing. When it comes to this, nothing beats hands-on experience. To help you in this process, I have collected five awesome project ideas which you can do right away to take your skills to the next level. Among these, you’ll find that others are more practical, while others are more theoretical. The common thing is however, that you’ll learn a great deal from each of them. Moreover, if you are looking for a job in machine learning, a portfolio of quality side projects is very beneficial.

To maximize the learning, I advise you to try them without looking at fully worked out projects first. Of course if you are stuck, it is useful to consult an existing solution — I’ll even point to some of them — but failing is an important part of learning.

Let’s see the projects themselves!

1. Implementing a convolutional neural network from scratch

In production settings, we typically rely on high level frameworks and libraries to avoid reinventing the wheel and stay productive. This is important to produce results fast, but it hides the low level details from you. My experience is that knowing exactly how neural networks and backpropagation works is like a superpower. Not a lot of people are aware of all the details, yet it gives a very deep insight.

For starting out, I recommend the famous CS231n: Convolutional Neural Networks for Visual Recognition course by Stanford. For our purposes, the most important modules are the following.

• Optimization: Stochastic Gradient Descent
• Backpropagation, Intuitions
• Neural Networks Part 1: Setting up the Architecture
• Neural Networks Part 2: Setting up the Data and the Loss

If you don’t have a low level understanding of neural networks already, I would advise to go through these before starting to code. In case you are stuck, I have written a detailed guide about this, with an accompanying GitHub repository.

In addition, a recent work by Andrej Karpathy is the micrograd library, which is an extremely concise but powerful Autograd engine. This is also definitely worth checking out!

When you built your first neural network from scratch, you’ll appreciate your favorite deep learning framework much more. They do so much work for you!

2. Visual exploration of convolutional networks

One common critique towards neural networks in general is that the resulting models are not interpretable. This is not entirely true, although they are not as simple to interpret as a decision tree. For instance, if we are talking about convolutional networks, the convolutional layers are essentially collections of filters. Each filter represents a single image processing operation. These are rarely tractable like the classical ones, for example the Gaussian filter used for smoothing, but they can still reveal much if you look at them from the right perspective.

One common method to gauge what a filter represents is to construct an image maximizing its response. Let’s quickly consider an example.

What image would maximize the activation of this filter? After a little bit of thinking, it is easy to see that this would be an image consisting exclusively of strong periodic vertical lines. (While an image with horizontal lines wouldn’t activate this filter at all.) In fact, this filter is the well known Sobel operator, used to detect vertical lines. To summarize, the pattern for which the filter is looking for can be found by optimization. Generally, the filters of a convolutional network progressively learn to find more and more complicated patterns as you go deeper. The following is the visualization of the features learned by a GoogLeNet model trained on ImageNet.

If you would like to go deeper and explore the models you train, here are two great starting points.

• Feature Visualization by Chris Olah , Alexander Mordvintsev and Ludwig Schubert
• How convolutional networks see the world by Francois Chollet

3. Building an API for your deep learning model

The life of a model only starts when it is handed off by the data scientist to the developer. To ship it into production and expose for the users, a simple and unified interface should be provided. Without this, the user of the model would have to know the exact framework the model is using and format their input accordingly. This is not only impractical, it might also be a serious technical hurdle if there are differences in the technical stack. Trust me, you don’t want to call a Python script from a JavaScript function.

The way it is usually handled is to put the model behind an API and deploy it as an independent service. The user of the API (i.e. the client who wants to use your service, which can be a mobile app, a smart camera, etc) sends the data and receives the answer in the form of HTTP requests. The API formats the data properly for the model, calls the prediction function and returns the answer as a HTTP response.

In Python, there are several frameworks for building API-s. The most popular ones are

• Flask,
• Pyramid,
• Falcon,
• FastAPI .

For simply packaging your model, I wouldn’t recommend Django, since it is not as lightweight as these. My personal favorite is the very recent FastAPI, which is just perfect for this task.

If you would like to look at a fully worked out example, I have written a detailed guide with example code using FastAPI, Docker and GitHub Actions.

4. Contributing to open source frameworks

An often overlooked aspect of the entire deep learning stack is creating things and making them usable for everyone else. If you think about this for a minute, almost all advances were made possible by the widespread adoption of user-friendly open source machine learning frameworks. In my opinion, familiarity with the contribution process is a must-have soft skill for all machine learning engineer. It requires communicating clearly and working as a part of a (possibly very large) development team, which is an important experience to have. In addition, you’ll get to see how your favorite framework does things behind the curtain, which is a great way to enhance your knowledge.

Before jumping into coding, make sure that you are familiar with the contribution guidelines of the framework. These documents outline the process, which should be followed. For finding problems to work on, the Issues page of the repository is the place to go. If you are just starting out, look for the good first issue label, which are usually easy enough to get yourself familiar with the contribution workflow.

Some projects where you can start:

• scikit-learn ( contribution guide , issues )
• PyTorch ( contribution guide , issues )
• TensorFlow ( contribution guide , issues )
• Keras ( contribution guide , issues )

Contributing is also a great way to get familiar with the ecosystem in general and an excellent opportunity to meet with like-minded people.

5. Paper reproductions

Another way to expand your knowledge is reproducing the results of a research paper. Some are lucky enough to do this for a living (or maybe even produce research papers), but even if this is not your job, there are plenty of benefits. As we have kind of established by now, deep knowledge comes from building things, not from reading about them. Even if you understand a result from a high level, there are plenty of essential details, only revealing themselves when you get your hands dirty. Knowing these is a landmark of mastery in a given subject.

There is a whole spectrum of difficulty when it comes to reproducing research papers. Some are simpler, but some require a lot of effort and hard work. Here is a short and completely subjective list, which can get you started. (My main field is computer vision, so I am biased towards that.)

Beginner level

• U-Net: Convolutional Networks for Biomedical Image Segmentation by Olaf Ronneberger, Philipp Fischer, Thomas Brox
• Deep Residual Learning for Image Recognition by Kaiming He, Xiangyu Zhang, Shaoqing Ren, Jian Sun
• Visualizing the Loss Landscape of Neural Nets by Hao Li, Zheng Xu, Gavin Taylor, Christoph Studer, Tom Goldstein

Intermediate level

• On the Variance of the Adaptive Learning Rate and Beyond by Liyuan Liu et al. (the Rectified Adam paper)
• Unpaired Image-to-Image Translation using Cycle-Consistent Adversarial Networks by Jun-Yan Zhu, Taesung Park, Phillip Isola, Alexei A. Efros
• Generative Adversarial Networks by Ian J. Goodfellow et al.

Advanced level

• Mask R-CNN by Kaiming He, Georgia Gkioxari, Piotr Dollár, Ross Girshick
• Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks by Chelsea Finn, Pieter Abbeel, Sergey Levine

Final thoughts

I hope that you have found these ideas interesting and you are ready to jump into work! :) True knowledge comes from building things on your own and the best things to build are those which serve the good of the community.

Let’s start building things!