3D object classification in 6 steps using kaolin and Colab

A step-by-step hands-on 3D object detection classifier without the pre-requisite for a Linux machine or the GPU in hand

In this article, you will learn to develop 3D object detection classifier without the pre-requisite for a Linux machine or the GPU in hand.

3D object detection is an important aspect to solve real-world applications such as self-driving cars, robots, and AR/VR applications. 3D data provides reliable depth information that can be used to accurately localize objects/vehicles.

3D Representation

Here are some of the types of 3D representation

A point cloud is a collection of points in 3D space, each point specified by an (XYZ) coordinates, optionally along with other attributes (like RGB color). The point cloud is called raw data obtained from the sensors such as LiDAR. These are converted to other representations such as polygon mesh, voxel grid before further processing.

Point cloud (PC) representation is more preferred since the conversion PC to other formats and vice versa is easy. Point representation preserves the original original geometric information in 3D space without any discretization.

Polygon Meshconsists set of polygonal faces with shared vertices that approximate a geometric surface.

Volumetricrepresents a single sample, or data point, on a regularly spaced 3D grid, Voxel grids are 3D grids in which each cell or “voxel” has a fixed size and discrete coordinates.

Projected View RGB(D)projects a 3D object into multiple views and extracts the corresponding view-wise features and then fuses these features for accurate object detection.

Google Colab - Introduction

Google colaboratory a.k.a colab is a jupyter notebook with the GPU access for free. When you train your models on colaboratory, you are allowed to use a GPU-based virtual machine, where you are given a maximum of 12 hours at a time. After which access is lost to that particular virtual machine instance and connected to different virtual machine instances after 12 hours. So please save the data or checkpoints periodically. The colab is completely free.

To choose the GPU for training, select Runtime > Change runtime type choose hardware accelerator (change None to GPU).

colab is pre-installed with major libraries (NumPy, matplotlib) and frameworks (TensorFlow, PyTorch) and for custom installation try ( !pip install ).

Explore Google colab here

Kaolin- 3D deep learning research framework

Kaolin is an open-source PyTorch library developed by a team of NVIDIA, aimed at accelerating 3D deep learning research. Kaolin framework converts the 3D models into deep learning datasets with few lines of code. kaolin provides easy access to loading and pre-processing popular 3D datasets.

Kaolin framework boils down complex 3D algorithms such as the conversion of point cloud data to voxel grid, triangle mesh representation in simple steps.

The kaolin framework benefits researchers in fields such as robotics, self-driving cars, augmented and virtual reality.

Now for 3D object detection classifier we will use Kaolin framework . The git repo can be found here

In summary, here are the Kaolin features we have been looking for.

Dataset

The dataset is available at princeton MODELNET . The ModelNet10.zip ZIP file contains CAD models from the 10 categories used to train the deep network in our 3D object detection classifier . Training and testing split is included in the file. The CAD models are completely cleaned inhouse, and the orientations of the models (not scale) are manually aligned by ourselves.

MODELNET 10 has 10 categories mentioned below:

Bathtub, bed, chair, desk, dresser, monitor, night_stand, sofa, table, toilet

Let’s get started…

The 3D object detection classifier

Step 1: Open google colab here and change runtime to GPU (~ 4–5 mins)

Installing the packages

Step 2: Downloading MODELNET10 dataset

Visualizing the the 3D model with X,Y,Z movement

Step 3: Dataloading

Kaolin provides convenience functions to load popular 3D datasets (ModelNet10). To start, we will define a few important parameters:

model_path variable will hold the path to the ModelNet10 dataset. categories variable to specify which classes we want to learn to classify. num_points is the number of points we will sample from the mesh when transforming it to a pointcloud.

Finally, we will disable multiprocessing and memory pinning if we are using CUDA for our transform operations.

This command defines a transform that first converts a mesh representation to a pointcloud and then normalizes it to be centered at the origin, and have a standard deviation of 1. Much like images, 3D data such as pointclouds need to be normalized for better classification performance.

split='train' argument indicates that we're loading the 'train' split. rep='pointcloud' loads up meshes and converts them into pointclouds. transform=norm applies a normalizing transform to each pointcloud.

Step 4: Setting up our model, optimizer and loss criterion

Step 5: Training the pointcloud classifier (~ 15–20 minutes)

The following line of code will train and validate a PointNet classifier

Hurray !!!, that’s it, you’ve trained your first 3D object detection classifier using Kaolin!!

Step 6: Evaluating the trained 3D object detection model on test data

We will create a new dataloader which will load the same data as our previous val_loader but with shuffling, and take a sample batch.

Next, we setup a plot to visualize the pointcloud, ground truth label and our prediction using a small visualization function.

color-coded the results — green for correct and red for incorrect.

Visualization results

GT- Ground Truth, Pred- Prediction.

Pat your back for completion of 3D object detection classifier.

Thanks for reading…

Jupyter notebook can be found here

References URL’s

1. Krishna Murthy Jatavallabhula , et.al. “Kaolin: A PyTorch Library for Accelerating 3D Deep Learning Research” arXiv:1911.05063v2 paper
2. https://github.com/NVIDIAGameWorks/kaolin
3. Princeton MODELNET

4. Google colab guide