Understanding Latent Space in Machine Learning

Learn a fundamental, yet often ‘hidden,’ concept of deep learning

What is Latent Space?

If I have to describe latent space in one sentence, it simply means a representation of compressed data.

Imagine a large dataset of handwritten digits (0–9) like the one shown above. Handwritten images of the same number (i.e. images that are 3’s) are the most similar to each other compared to other images of different numbers (i.e. 3s vs. 7s). But can we train an algorithm to recognize these similarities? How?

If you have trained a model to classify digits , then you have also trained the model to learn the ‘structural similarities’ between images. In fact, this is how the model is able to classify digits in the first place- by learning the features of each digit.

If it seems that this process is ‘hidden’ from you, it’s because it is. Latent, by definition, means “hidden.”

The concept of “latent space” is important because it’s utility is at the core of ‘deep learning’ — learning the features of data and simplifying data representations for the purpose of finding patterns.

Intrigued? Let’s break latent space down bit by bit:

Why do we compress data in ML?

Data compressionis defined as the process of encoding information using fewer bits than the original representation. This is like taking a 19D data point (need 19 values to define unique point) and squishing all that information into a 9D data point.

More often than not, data is compressed in machine learning to learn important information about data points . Let me explain with an example.

Say we would like to train a model to classify an image using a fully convolutional neural network (FCN). (i.e. output digit number given image of digit). As the model ‘learns’, it is simply learning features at each layer (edges, angles, etc.) and attributing a combination of features to a specific output.

But each time the model learns through a data point, the dimensionality of the image is first reduced before it is ultimately increased. (see Encoder and Bottleneck below). When the dimensionality is reduced, we consider this a form of lossy compression.

Because the model is required to then reconstruct the compressed data (see Decoder), it must learn to store all relevant information and disregard the noise. This is the value of compression- it allows us to get rid of any extraneous information, and only focus on the most important features.

This ‘compressed state’ is the Latent Space Representation of our data.

What do I mean by space?

You may be wondering why we call it a latent space . After all, compressed data, at first glance, may not evoke any sort of “space.”

But here’s the parallel.

In this rather simplistic example, let’s say our original dataset are images with dimensions 5 x 5 x 1. We will set our latent space dimensions to be 3 x 1, meaning our compressed data point is a vector with 3-dimensions.

Now, each compressed data point is uniquely defined by only 3 numbers. That means we can graph this data on a 3D Plane (One number is x, the other y, the other z).

This is the “space” that we are referring to.

Whenever we graph points or think of points in latent space, we can imagine them as coordinates in space in which points that are “similar” are closer together on the graph.

A natural question that arises is how would we imagine space of 4D points or n-dimensional points, or even non-vectors (since the latent space representation is NOT required to be 2 or 3-dimensional vectors, and is oftentimes not since too much information would be lost).

The unsatisfying answer is, we can’t . We are 3-dimensional creatures that cannot fathom n-dimensional space (such that n > 3). However, there are tools such as t-SNE which can transform our higher dimensional latent space representations into representations that we can visualize (2D or 3D). (See Visualizing Latent Space section below.)

But you may be wondering, what are ‘similar’ images, and why does reducing the dimensionality of our data make similar images ‘closer’ together in space?

What do I mean by similar?

If we look at three images, two of a chair and one of a desk, we would easily say that the two chair images are the most similar whereas the desk is the most different from either of the chair images.