Solutions to Issues with Edge TPU

Solutions to Issues with Edge TPU

Learn how to solve common issues when working with TPU USB Accelerator

In this article, I will be sharing some common errors that I faced when working on making the Deep Learning model work on Edge TPU USB Accelerator and the solutions that worked for me.

A Coral USB Accelerator is an Application Specific Integrated Chip(ASIC) that adds TPU computation power to your machine. The additional TPU computing makes inferences at the Edge for a deep learning model faster and easier.

Plug the USB Accelerator in USB 3.0 port, copy the TFLite Edge Model, along with the script for making inferences, and you are ready to go. It runs on Mac, Windows, and Linux OS.

To create the Edge TFLite model and make inferences at the Edge using USB Accelerator, you will be using the following steps.

1. Create the model
2. Train the model
3. Save the model
4. Apply post-training quantization
5. Convert the model to TensorFlow Lite version
6. Compile the tflite model using edge TPU compiler for Edge TPU devices like Coral Dev board to TPU USB Accelerator
7. Deploy the model at Edge and make inferences

Issue 1: When comping the TfLite model using edgetpu_compiler get an error “Internal compiler error. Aborting!”

Code that was erroring out

test_dir='dataset' 
def representative_data_gen():
  dataset_list = tf.data.Dataset.list_files(test_dir + '\\*')
  for i in range(100):
    image = next(iter(dataset_list))
    image = tf.io.read_file(image)
    image = tf.io.decode_jpeg(image, channels=3)
  image = tf.image.resize(image, (500,500))
    image = tf.cast(image / 255., tf.float32)
    image = tf.expand_dims(image, 0)# Model has only one input so each data point has one element
yield [image]keras_model='Intel_1.h5'#For loading the saved model and tf.compat.v1 is for compatibility with TF1.15
converter=tf.compat.v1.lite.TFLiteConverter.from_keras_model_file(keras_model)# This enables quantization
converter.optimizations = [tf.lite.Optimize.DEFAULT]# This ensures that if any ops can't be quantized, the converter throws an error
converter.target_spec.supported_ops = [tf.lite.OpsSet.TFLITE_BUILTINS_INT8]# Set the input and output tensors to uint8
converter.inference_input_type = tf.uint8
converter.inference_output_type = tf.uint8# set the representative dataset for the converter so we can quantize the activations
converter.representative_dataset = representative_data_gen
tflite_model = converter.convert()#write the quantized tflite model to a file
with open('Intel_1.tflite', 'wb') as f:
  f.write(tflite_model)with open('Intel_1.tflite', 'wb') as f:
  f.write(tflite_model)
print("TFLite conversion complete")

After converting the model to Tflite, compiling the model for Edge TPU

This error has no description, and you are left in the dark to solve the issue.

The solution that worked for me

I created two custom models with the same architecture but different datasets, different numbers of classes to identify and used the same steps to compile the two tflite models.

One model compiled successfully, while the other model gave the “Internal Compiler Error.”

Now I knew it was not the model or the steps to create the Edge TPU Model but something related to data.

The discussion on this issue: https://github.com/tensorflow/tensorflow/issues/32319 helped me try different options.

My solution was to reduce the image dimensions and voila it worked!!!

Why does this solution work?

Reducing the image dimensions might have helped with the range of activations used for creating the representative dataset.

Representative datasets help with an accurate dynamic range of activations, which is used to quantize the model.

When we quantize a model, we reduce the precision of the numbers used to represent weights, activation, and biases of the TensorFlow model from 32-bit floating points to 8 bit-int, which helps make the model light-weight.

TFLite and EdgeTPU models are light-weight, and hence we have low latency, faster inference time, and reduced power consumption.

The below diagram shows the difference in the size of the saved model in H5 format, TFlite model, and EdgeTPU.

I tried one more thing to solidify my understanding; I tried to compile the same errored model with a higher image dimension(500,500) in a lower version of the Edge TPU runtime version.

It compiled, but the entire model will run on CPU and not TPU, this was the indication that the quantized model still had some float operations.

I retrained the model with image dimension as (100,100) instead of (500, 500)

The code that worked.

keras_model='Intel_epoch50_batch16.h5'#For loading the saved model and tf.compat.v1 is for compatibility with TF1.15
converter=tf.compat.v1.lite.TFLiteConverter.from_keras_model_file(keras_model)# This enables quantization
converter.optimizations = [tf.lite.Optimize.DEFAULT]# This ensures that if any ops can't be quantized, the converter throws an error
converter.target_spec.supported_ops = [tf.lite.OpsSet.TFLITE_BUILTINS_INT8]# Set the input and output tensors to uint8
converter.inference_input_type = tf.uint8
converter.inference_output_type = tf.uint8# set the representative dataset for the converter so we can quantize the activations
converter.representative_dataset = representative_data_gen
tflite_model = converter.convert()#write the quantized tflite model to a file
with open('Intel_class.tflite', 'wb') as f:
  f.write(tflite_model)with open('Intel_epoch50_batch16.tflite', 'wb') as f:
  f.write(tflite_model)

Now the model compiled with the latest version of Edge TPU Compiler.

With the change in dimension, now most of the operation will run on TPU.

Issue 2: ValueError: Found too many dimensions in the input array of operation ‘reshape.’

After the tflite model is successfully complied with the edgetpu_compiler, and we instantiate the interpreter for inference. Error “ValueError: Found too many dimensions in the input array of operation ‘reshape’” is encountered.

I replied to discussion with my solution on this thread: https://github.com/google-coral/edgetpu/issues/74

The solution that worked for me

The issue was with the representative dataset that I used for post-training quantization. The dataset used for creating the representative dataset had more images than what I had provided in the images folder.

My test_dir had 99 images, and I had set the range to 100. When I matched the dataset used for the representative dataset with the no. of images in the folder, then the issue was resolved.

def representative_data_gen():
  dataset_list = tf.data.Dataset.list_files(test_dir + '\\*')
  for i in range(99):
    image = next(iter(dataset_list))
    image = tf.io.read_file(image)
    image = tf.io.decode_jpeg(image, channels=3)
    image = tf.image.resize(image, (100,100))
    image = tf.cast(image / 255., tf.float32)
    image = tf.expand_dims(image, 0)# Model has only one input so each data point has one element
yield [image]

Conclusion:

Step by step, the elimination of variables will lead you to the root cause. I use the 5W, and 1H technique where 5W stands for Who, What, Where, When, Why, and the 1 H stands for How.