Build Your Fully Configurable Interface to Create MNIST-like Dataset with Python in 10 Minutes

Step by step user-friendly python interface to write a dataset from a JSON configuration file (with code)

Motivations

For a personal project and maybe a paper (fingers crossed), I need to create a MNIST-like dataset . I thought it could be useful for others . I will, therefore, explain how I coded this interface with Python . ( github repository )

The purpose of this article is to be able to design its own interface as flexible and fast as possible. This interface will allow us to draw our characters with the mouse and everything else will be done automatically with a large choice of parameters specified in a configuration file.

Fully configurable

I was looking for a fully configurable interface from a JSON file . No more need to touch the code, it is flexible , convenient and scalable . We will come back throughout this article on the different parameters but everything is contained in the file I named config.json . You are free to name it as you wish. It allows you to change:

• The size of the image we draw on.
• The size of the output image to complete the dataset .
• The expected number of image for each class, the preferred drawing rhythm and the presence of the rendering of our output images.
• The folder name containing the dataset with the possibility to create subfolders for each class .
• The size(s) , strength and fading of our pencil..
• The interpolation method(s).
• The names of the classes.

Storage and utilities

First things first. We need to know how we will organize our data. In the file config.json we have the storage part:

• root (str) : path to the dataset’s root folder
• by_class_name (bool) : way to store the dataset

All our images will have the same pattern . The name of the image will always start with the name of the class. Then an underscore and then a 5 character number to identify it. So we have a root/(subfolderclass)/name_id.png format.

You will notice the id’s are set incrementally . In other words you can see how many images for each class is already generated. However as a user you may or may not need to place other files in these folders or sometimes delete images or rename them. We will put at the beginning of our code a small part to reorder our image names automatically .

In a utils.py we set up some functions :

• get_json : return the content of a JSON file as a python dictionary .
• check_folder_path : checks the folder exists , otherwise a new one is created .
• order_filename_by_prefix : reorder the files in the correct name format.

Designer

What we want is to be able to draw our characters by hand (mouse) . From a drawing window we want to quickly create our dataset. Then we need a class Designer which we will be stored in the designer.py file. To understand why we need this class, let’s take a closer look at some parameters in our configuration file:

• input : contains the dimensions of the image to be drawn on and the thickness of the drawing pencil (only for visualization).
• output : contains the dimensions of the output image to make our dataset.
• interpolation → method (str or list of str): the dimensions of our drawing image and our output image are not the same. So we need an interpolation method to resize the image. We then have the choice of all the interpolation methods provided by OpenCV : INTER_NEAREST , INTER_LINEAR , INTER_AREA , INTER_CUBIC , INTER_LANCZOS4 . We can precise RANDOM to change the interpolation method at each image. This parameter can be a list : in this case, there will be as many designers as interpolation methods specified and thus as many output images for an input image.
• line → thickness (float or list of float): pencil thickness on the output image proportional to the diagonal of the image. This parameter can be a list : in this case there will be as many designers as pen thickness. The result is a designer matrix combining the interpolation methods and the specified thickness.
• line → range_value (int or list of int): intensity of the pixel when it is drawn. In the case of a list: the intensity of the drawn pixel will be randomly chosen in the specified range.
• line → fading (float): value of the fade on the edge of the line, the fade is linear. No impact for the value fading=1 .

Note the drawing will be done by line composition . Hence the role of the draw(self, pt_1, pt2) . So we still have to be able to control all this with the mouse.

Controller

First of all we need to create a class Controller and initialize it from the configuration dictionary.

During the initialization several things happen:

• Extraction of all configuration parameters.
• Verification that the folders exist, if not, create them.
• Initialization of the designers .
• Calculation of the number of images by classes which are already done.
• Creation of the OpenCV window for drawing and the OpenCV window for visualizing .

To create all our designers there will be a designer matrix with the vertical for pencil thicknesses and the horizontal for interpolation methods. At the same time we can create a function to catch mouse events on the drawing window. You can only draw with the left mouse button pressed . Finally you can retrieve the number of images already drawn.

Let’s draw

Before we can draw let’s have a look at what our last parameters propose in process :

• volume (int): number of image to reach for each class.
• selection (str): way to select the next character to draw. Can be RANDOM , CLASSBYCLASS to finish a character before go to write the others, or ROTATE to change by rotation over the different classes.
• display_output (bool): visualize or not the output window.

You can now create a new image to draw, draw it and save it in the right format ( root/(subfolderclass)/name_id.png ).

All that remains is to be able to view everything from the OpenCV window and to run the program as long as the image volume per class is not reached.

Main

Finally for more convenience we can leave the choice to the user to specify his configuration file with the command in the shell:

python3 main.py -c path/to/the/config/file.json
or 
python3 main.py -config path/to/the/config/file.json