Convolutional Neural Network: Feature Map and Filter Visualization

Convolutional Neural Network: Feature Map and Filter Visualization

Learn how Convolutional Neural Networks understand images.

In this article, we will visualize the intermediate feature representations across different CNN layers to understand what happens inside CNN’s to classify images.

Prerequisites:

Convolutional Neural Network Basics ,

Building Powerful Image Classification Convolutional Neural Network using Keras

Building powerful image classification CNN using Keras

A quick overview of CNN

Supervised Deep Learning and Machine Learning take data and results as an input during training to generate the rules or data patterns. Understanding of data patterns or rules generated by the model helps us understand how the results were derived from the input data.

Training:Convolutional neural network takes a two-dimensional image and the class of the image, like a cat or a dog as an input. As a result of the training, we get trained weights, which are the data patterns or rules extracted from the images.

Inference or Prediction:Image will be the only input passed to the trained model, and the trained model will output the class of the image. The class of the image will be based on the learned data patterns during the training.

CNN Architecture

Apply filtersor feature detectors to the input image to generate the feature maps or the activation maps using the Relu activation function. Feature detectors or filters help identify different features present in an image like edges, vertical lines, horizontal lines, bends, etc.

Pooling is then applied over the feature maps for invariance to translation. Pooling is based on the concept that when we change the input by a small amount, the pooled outputs do not change . We can use min pooling, average pooling, or max pooling. Max pooling provides better performance compared to min or average pooling.

Flatten all the input and pass these flattened inputs to a deep neural network that outputs the class of the object

The class of the image can be binary like a cat or dog, or it can be a multi-class classification like identifying digits or classifying different apparel items.

Neural networks are like a black box, and learned features in a Neural Network are not interpretable. You pass an input image, and the model returns the results.

What if you get an incorrect prediction and would like to figure out why such a decision was made by CNN?

If only you could visualize the intermediate representation applied across different Convolutional layers in CNN to understand how the model learns. Understanding the working of the model will help to know the reason for incorrect predition that will lead to better fine tuning of the model and explain the decisions

The example used here is a deep CNN model for classifying cats and dogs . Before you dive in to learn to visualize both the filters and the feature maps generated by CNN, you will need to understand some of the critical points about Convolutional layers and the filters applied to them.

Key points about Convolution layers and Filters

• The depth of a filter in a CNN must match the depth of the input image . The number of color channels in the filter must remain the same as the input image.
• Different Conv2D filters are created for each of the three channels for a color image.
• Filters for each layer are randomly initialized based on either Normal or Gaussian distribution.
• Initial layers of a convolutional network extract high-level features from the image, so use fewer filters. As we build further deeper layers, we increase the number of filters to twice or thrice the size of the filter of the previous layer.
• Filters of the deeper layers learn more features but are computationally very intensive .

Building a Convolutional Neural Network

We build a CNN for classifying dogs and cats and later visualize the feature maps or activation maps and filters applied to generate them on an input image

Importing required libraries

import tensorflow as tf
from tensorflow.keras.models import Sequential
from tensorflow.keras.layers import Dense, Conv2D, Flatten, Dropout, MaxPooling2D
from tensorflow.keras.preprocessing.image import ImageDataGenerator, img_to_array, load_imgimport os
import numpy as np
import matplotlib.pyplot as plt

Creating data for image processing

Dataset can be downloaded from here

We will unzip the file and create the folders as shown below and split the data into the training dataset with 10,000 cats and 10,000 dogs images and validation dataset containing 2500 cats and 2500 dogs images

Set key parameters

batch_size = 64 
epochs = 50
IMG_HEIGHT = 150
IMG_WIDTH = 150

Rescale and Apply different Augmentation to the training image

train_image_generator = ImageDataGenerator( rescale=1./255, rotation_range=45, width_shift_range=.15, height_shift_range=.15, horizontal_flip=True, zoom_range=0.3)

Rescale Validation data

validation_image_generator = ImageDataGenerator(rescale=1./255)

Generate batches of normalized data for train and validation data set

your data is stored in directories, so use the flow_from_directory() method. flow_from_directory() will take the data from the specified path and generates batches of augmented normalized data.

train_data_gen = train_image_generator.flow_from_directory(batch_size=batch_size, directory=TRAIN_PATH, shuffle=True, target_size=(IMG_HEIGHT, IMG_WIDTH), class_mode='binary')val_data_gen = validation_image_generator.flow_from_directory(batch_size=batch_size, directory=VAL_PATH, target_size=(IMG_HEIGHT, IMG_WIDTH), class_mode='binary')

Create the Deep Convolutional Neural network model

#Build the model
model = Sequential([
 Conv2D(16, 3, padding='same', activation='relu', 
 input_shape=(IMG_HEIGHT, IMG_WIDTH ,3)),
 MaxPooling2D(),
 Dropout(0.2),
 Conv2D(32, 3, padding='same', activation='relu'),
 MaxPooling2D(),
 Conv2D(64, 3, padding='same', activation='relu'),
 MaxPooling2D(),
 Dropout(0.2),
 Flatten(),
 Dense(512, activation='relu'),
 Dense(1)
])# Compile the model
model.compile(optimizer='adam', loss=tf.keras.losses.BinaryCrossentropy(from_logits=True), metrics=['accuracy'])# print the model architecture
model.summary()

Training the Model

We train the model for 50 epochs

history = model.fit_generator(
 train_data_gen,
 steps_per_epoch=1000,
 epochs=epochs,
 validation_data=val_data_gen,
 validation_steps=1000
)

Feature Visualization

Feature Visualization translates the internal features present in an image into visually perceptible or recognizable image patterns. Feature visualization will help us understand the learned features explicitly.

First, you will visualize the different filters or feature detectors that are applied to the input image and, in the next step, visualize the feature maps or activation maps that are generated.

Visualizing Filters or Feature Detectors in a CNN

CNN uses learned filters to convolve the feature maps from the previous layer. Filters are two- dimensional weights and these weights have a spatial relationship with each other.

The steps you will follow to visualize the filters.

1. Iterate through all the layers of the model using model.layers
2. If the layer is a convolutional layer, then extract the weights and bias values using get_weights() for that layer.
3. Normalize the weights for the filters between 0 and 1
4. Plot the filters for each of the convolutional layers and for all the channels. For Color image, you will have 3 channels for RGB. For a grayscale image, the number of channels will be 1

#Iterate thru all the layers of the model
for layer in model.layers:
 if 'conv' in layer.name:
 weights, bias= layer.get_weights()
 print(layer.name, filters.shape)

#normalize filter values between 0 and 1 for visualization
f_min, f_max = weights.min(), weights.max()
 filters = (weights - f_min) / (f_max - f_min) 
 print(filters.shape[3])
 filter_cnt=1

 #plotting all the filters
 for i in range(filters.shape[3]):
#get the filters
 filt=filters[:,:,:, i]
#plotting each of the channel, color image RGB channels
for j in range(filters.shape[0]):
 ax= plt.subplot(filters.shape[3], filters.shape[0], filter_cnt )
 ax.set_xticks([])
 ax.set_yticks([])
 plt.imshow(filt[:,:, j])
 filter_cnt+=1
 plt.show()

Visualizing Feature maps or Activation maps generated in a CNN

Feature maps are generated by applying Filters or Feature detectors to the input image or the feature map output of the prior layers. Feature map visualization will provide insight into the internal representations for specific input for each of the Convolutional layers in the model.

The steps you will follow to visualize the feature maps.

1. Define a new model, visualization_model that will take an image as the input. The output of the model will be feature maps, which are an intermediate representation for all layers after the first layer. This is based on the model we have used for training.
2. Load the input image for which we want to view the Feature map to understand which features were prominent to classify the image.
3. Convert the image to NumPy array
4. Normalize the array by rescaling it
5. Run the input image through the visualization model to obtain all
   intermediate representations for the input image.
6. Create the plot for all of the convolutional layers and the max pool layers but not for the fully connected layer. For plotting the Feature maps, retrieve the layer name for each of the layers in the model.

img_path='\\dogs-vs-cats\\test1\\137.jpg' #dog
# Define a new Model, Input= image 
# Output= intermediate representations for all layers in the 
# previous model after the first.
successive_outputs = [layer.output for layer in model.layers[1:]]#visualization_model = Model(img_input, successive_outputs)
visualization_model = tf.keras.models.Model(inputs = model.input, outputs = successive_outputs)#Load the input image
img = load_img(img_path, target_size=(150, 150))# Convert ht image to Array of dimension (150,150,3)
x = img_to_array(img) 
x = x.reshape((1,) + x.shape)# Rescale by 1/255
x /= 255.0# Let's run input image through our vislauization network
# to obtain all intermediate representations for the image.
successive_feature_maps = visualization_model.predict(x)# Retrieve are the names of the layers, so can have them as part of our plot
layer_names = [layer.name for layer in model.layers]
for layer_name, feature_map in zip(layer_names, successive_feature_maps):
 print(feature_map.shape)
 if len(feature_map.shape) == 4:

# Plot Feature maps for the conv / maxpool layers, not the fully-connected layers

n_features = feature_map.shape[-1] # number of features in the feature map
size = feature_map.shape[ 1] # feature map shape (1, size, size, n_features)

# We will tile our images in this matrix
display_grid = np.zeros((size, size * n_features))

 # Postprocess the feature to be visually palatable
for i in range(n_features):
 x = feature_map[0, :, :, i]
 x -= x.mean()
 x /= x.std ()
 x *= 64
 x += 128
 x = np.clip(x, 0, 255).astype('uint8')
# Tile each filter into a horizontal grid
 display_grid[:, i * size : (i + 1) * size] = x# Display the grid
scale = 20. / n_features
 plt.figure( figsize=(scale * n_features, scale) )
 plt.title ( layer_name )
 plt.grid ( False )
 plt.imshow( display_grid, aspect='auto', cmap='viridis' )

We can see that for the dog image, the snout and the tongue are very prominent features and for the cat image, the ears and tail are prominent in the feature maps.

Code available here

Conclusion:

Visualizing an inside story of how CNN learns to identify different features present in images provides a deeper insight into how the model works. It will also help to understand why the model might be failing to classify some of the images correctly and hence fine-tuning the model for better accuracy and precision.

References:

How Convolutional Neural Networks see the World — A survey of Convolutional Neural Network Visualization Methods

Visualizing and Understanding Convolutional Networks

Feature Visualization — Google AI Blog

https://cs.nyu.edu/~fergus/papers/zeilerECCV2014.pdf