Cluster-based Image Segmentation -Python

Cluster-based Image Segmentation -Python

Understanding Image Segmentation

Recently I was working on an Image classification task where first I wanted to capture the region of interest from the image before feeding it into the model. I tried a technique called cluster-based image segmentation which helped me to improve my model performance by a certain level. Let us see what it is and some sample codes to do cluster segmentation, you can find the Jupyter Notebook at the bottom.

What is Image Segmentation?

Imagine that you are going to cross the road, what you do before you cross the road?

First, you see both sides of the road to determine the approaching vehicles and other environmental objects, then you do some amazing estimation of approaching speed and decide on when and how to cross the road. All these happens within a fraction of time, how amazing isn’t it.

1. Our brain captures the Images of both sides of the road
2. It detects the vehicles and other objects on the road == Object Detection
3. Not only detect before that it determines the shape of every object it detects == Image Segmentation

How amazing our brain is, by determining the shapes of different objects, it able to detect the multiple objects in the same snapshot.

Let me explain furthermore, assume we have our Image Classification model which is able to classify the apple and orange with more than 95% accuracy. When we input an Image that contains both apple and orange the prediction accuracy will go down. As the number of objects in the image increases the classification models’ performances will go down. That is where the object localization comes into play.

Before we detect objects in an image and classify it, the model needs to understand what is in the image, this is where Image Segmentation helps. It creates a pixel-wise mask for the objects in an image which helps models to understand the shape of objects and their position in the image at a more granular level.

What are the types of Segmentation?

Image segmentation is broadly categorized into two main categories.

1. Semantic Segmentation
2. Instance Segmentation

In the first image, we can see that detected objects all are men. In semantic segmentation , we consider all those pixels belong to one class, so we represent them all by one color. On the other hand in instance segmentation , those pixels belong to the same class but we represent different instances of the same class with different colors.

Based on the approach we use segmentation can be divided into many narrower categories.

• Region-Based Segmentation
• Edge Detection based Segmentation
• Cluster-based segmentation
• CNN based Segmentation and etc.

I am only going to give an example of cluster-based segmentation in this article as I promised at the top.

What is Cluster-based Segmentation?

Recall your understanding of clustering algorithms. Clustering algorithms are used to group closer the data points that are more similar to each other, from other group data points.

Now think of an image that holds apple and orange. Most of the pixel points in apple should be red/green, which is different from the pixel values of orange. If we can cluster these points we can distinguish each object from one another right. That’s how the cluster-based segmentation works. Let’s see some code samples now.

from skimage.io import imread
from skimage.color import rgb2gray
import numpy as np
import matplotlib.pyplot as plt
%matplotlib inline
from scipy import ndimage# Scaling the image pixels values within 0-1
img = imread('./apple-orange.jpg') / 255plt.imshow(img)
plt.title('Original')
plt.show()

As it is visible to our naked eye there are five color segments in the Image

1. The green part of Apples
2. The orange part of Oranges
3. Gray Shadows at bottom of the Apples and oranges
4. Bright Yellowish part of Apple’s top and right parts
5. White Background

Let us see whether we can cluster them using our KMeans algorithm from scikit-learn

# For clustering the image using k-means, we first need to convert it into a 2-dimensional array
image_2D = img.reshape(img.shape[0]*img.shape[1], img.shape[2])# Use KMeans clustering algorithm from sklearn.cluster to cluster pixels in image
from sklearn.cluster import KMeans# tweak the cluster size and see what happens to the Output
kmeans = KMeans(n_clusters=5, random_state=0).fit(image_2D)
clustered = kmeans.cluster_centers_[kmeans.labels_]# Reshape back the image from 2D to 3D image
clustered_3D = clustered.reshape(img.shape[0], img.shape[1], img.shape[2])plt.imshow(clustered_3D)
plt.title('Clustered Image')
plt.show()

Wow, that works!!! We are able to cluster all five parts. This is how the cluster segmentation works.

There are many advanced techniques like Mask R-CNN to do more granular level segmentation. Let us see those topics in some other article. I hope you now have some level of Understanding of Image Segmentation.

You can find the notebook for the above example at https://github.com/Mathanraj-Sharma/sample-for-medium-article/blob/master/cluster-based-segmentation-skimage/cluster-based-segmentation.ipynb