Lets Get Artsy! Creating Custom Snapchat Filters With Neural Networks!

In the original paper, he first used the VGG19 model to achieve this result. VGG19 is a popular neural network for image recognition, but we’re mainly going to focus on how it extracts features from a photograph. The feature extraction is done through a number of sequential layers that can detect small patterns . The features it initially picks up are very simple, but gets more complex as more feature layers are added. we can then use these features, to recreate our original image!

Content Loss

In this step, we want to try to recreate, more or less the essence of our content image. To achieve this, we’ll first start off by creating an image full of random noise. We’ll then shape this image to look like our content image, using the extracted features from our network as a guide. The distance between the random image features and the content features is called our “content loss”. To get the best stylized image, we want to minimize this loss function through back-propagation. We’ll finally loop through this whole thing and optimize on each step to get a pretty good recreation of our image.

for i in range(1,epochs+1):target_features = model_activations(target,model)content_loss = torch.mean((content_features['conv4_2']-target_features['conv4_2'])**2)

Style Loss

For this step, we’ll want to recreate the core artistic direction of our style image. Gatsy solves this in an interesting way! Instead of comparing our style features to another image, we’ll compare each feature map to itself. We’ll first transform the style features into a gram matrix, which is basically the inner product of a matrix. The “style loss” was is basically the sum of losses for all feature maps in the model network. We’ll do a similar looping process and optimize on each step.

style_loss = 0for layer in style_wt_meas:style_gram = style_grams[layer]target_gram = target_features[layer]_,d,w,h = target_gram.shapetarget_gram = gram_matrix(target_gram)style_loss += (style_wt_meas[layer]*torch.mean((target_gram-style_gram)**2))/d*w*h

Our final step in this process is to calculate our final loss as the weighted sum of our content loss and style loss. we’ll also optimize this loss at each step of our training loop.

total_loss = content_wt*content_loss + style_wt*style_loss

Problems and Future Work

Gatsy Method for Style Transfer was the first of it’s kind, but it did have some problems. The major problem is that it’s very slow. This is because Since the optimization is done every cycle in the training loop, the algorithm takes some time produce anything. Secondly, sometimes changing the weights will greatly destabilize the photo. There is a fix by adding a total loss variation which aligns the mean and variance of the content and style images. From my research, it looks like Neural Style Transfer is still a hot topic, with applications being applied to video processing, simulations, and designs.

I used Ayush Chaurasia: Artistic Neural Style Transfer From Scratch on Youtube to create the code. Like I mentioned in the beginning of this article, there are plenty of great resources you can find online. If you wanted to get started with computer vision problems, I think that building neural style transfer application can be very fun and exciting! So I hoped this helped and I wish you luck if you try to pursue this!

Thanks for Reading!

Here are some cool results that I’ve taken, I can’t wait to see what you’ll create!