Using FaceNet For On-Device Face Recognition With Android

1. Convert the Keras model to a TFLite model

The FaceNet Keras model is available on nyoki-mtl/keras-facenet repo. After downloading the .h5 model, we’ll use the tf.lite.TFLiteConverter API to convert our Keras model to a TFLite model.

2. Setting up a Preview and ImageAnalyser using CameraX

To implement a live Camera feed, we use CameraX. I have used the code available in the official docs . Next, we create a FrameAnalyser class which implements ImageAnalysis class, which will help us retrieve camera frames and run inference on them.

All our classification code will come in the analyze method. First, using Firebase MLKit, we’ll get bounding boxes for all faces present in the camera frame ( a Bitmap object ). We’ll create a FirebaseVisionFaceDetector which runs the face detection model on an FirebaseVisionInputImage object.

3. Producing Face Embeddings using FaceNet and Comparing them.

First, we’ll produce face embeddings using our FaceNet model. Before, we’ll create a helper class for handling the FaceNet model. This helper class will,

1. Crop the given camera frame using the bounding box ( as Rect ) which we got from Firebase MLKit.
2. Transform this cropped image from a Bitmap to a ByteBuffer with normalized pixel values.
3. Finally, feed the ByteBuffer to our FaceNet model using the Interpreter class provided by TF Lite Android library.

In the snippet below, see the getFaceEmbedding() method which encapsulates all the above steps.

Now, we have a class that would return us the 128-dimensional embedding for all faces present in the given image. We come back to a FrameAnalyser ‘s analyze() method. Using the helper class which just created, we’ll produce face embeddings and compare each of them with a set of embeddings that we already have.

Before that, we need to get the set of predefined embeddings, right? These embeddings will refer to the people whom we need to recognize. So, the app will read the images folder present in the internal storage of the user’s device. If the user wants to recognize, two users, namely Rahul and Neeta , then he/she needs to create two separate directories within the images folder. Then he/she has to place an image of Rahul and Neeta in their respective sub-directories.

images ->
   rahul ->
      image_rahul.png   neeta ->
      image_neeta.png

Our aim to read these images and produce a HashMap<String,FloatArray> object where the key ( String )will the subject’s name like Rahul or Neeta and the value ( FloatArray ) will the corresponding face embedding. You’ll get an idea of the process like studying the code below.

We’ll compare the embeddings using the cosine similarity metrics which will return a similarity score in the interval [ -1 , 1 ] .

The predictions array is then supplied to the boundingBoxOverlay class which draws the bounding boxes and also displays the label. In the BoundingBoxOverlay.kt class,

The Results

Using the app, I have tried to recognize the faces of Jeff Bezos and Elon Musk,

Also, I had stored in the images in my internal storage as such,

The End

I hope you liked the story. I have included any APK in the GitHub repo so that you can try the app on your device. Thanks for reading!