TensorFlow Object Detection on VideoStream from ESP32-CAM

Introduction

This Python script is meant to be run on a computer, and attempts to extend the tutorial on Tensor Flow Object Detection API to enable TensorFlow object detection on 'HTTP Multipart MJPEG' video streams.

As noted by the author of the Tensor Flow Object Detection Tutorial, at the time of writing, Object Detection model training and evaluation was not migrated to TensorFlow 2.X, but object detection using pre-trianed models work.

Prerequisites

ESP32-CAM

While any video stream of 'HTTP Multipart MJPEG' should work, this was tested on with video stream from the ESP32-CAM.

The ESP32-CAM consists of a ESP32 module and OV2640 camera sensor, integrated onto a development board. WiFi functionality of this device is enabled by the ESP32 module, and video is achieved by successive calls to the camera.

For the ESP32-CAM, you will need to upload the code for streaming video over WiFi via JPEG HTTP Stream (which I term as 'HTTP Multipart MJPEG' here). A tutorial on how to get the ESP32-CAM (with Video Stream) to connect to your WiFi router can be found here. Note that the tutorial uses UXGA (1600x1200) as default so you will get 6fps if the WiFi conditions are good. I personally set it to SVGA (800x600) to get 12fps.

Python 3

This script was written for Python 3 (preferably 3.7 and above). You will need to have Python 3 installed on your computer.

The script requires the following libraries (and dependencies therein).

TensorFlow

TensorFlow is a Python library for fast numerical computing created and released by Google.

The instructions to install TensorFlow can be found here. The TensorFlow CPU variant should work fine for this script. You can 'upgrade' to the GPU variant in the future if you have compatible hardware.

As I had TensorFlow 2.1 installed, I confirm that TensorFlow 2.1 object detection works with pre-trained models. As noted here, TensorFlow 2.X would work on pre-trained models, but model and training is not yet implemented (as of writing this README), so you might want to go for Tensorflow 1.14 instead if you wish to do more than just object detection.

While using TensorFlow GPU, I found that the code would run for a few frames (i.e. less than a second, since at 800x600 I get a 12fps stream), before freezing. It is unclear if this is because I am using TensorFlow 2.1, but I found that the workaround is to force the use of TensorFlow CPU when running this script by adding os.environ["CUDA_VISIBLE_DEVICES"] = "-1" at the top of the script, right after the imports, as commented in the script.

TensorFlow models

After installing TensorFlow, you will need to install the TensorFlow models. If you are using Anaconda, you can start following the instructions from install-prerequisites to adding necessary environment variables.

If you are not using Anaconda, then you can install the TensorFlow models with the following steps:

1. Install the prequisite packages via pip:

    pip install pillow lxml matplotlib opencv-python
   

2. Download and extract the models (especially the research folder) via this link, and place them in a folder as described by this link (ignore the link in the note):

    TensorFlow
    └─ models
        ├── official
        ├── research
        ├── samples
        └── tutorials
   

   Unfortunately, the TensorFlow Official Release Models does not include the research folder contents (where the pre-trained models are stored).

3. Download and extract Protobuf to a folder of your choice, add it to your PATH, and run the following command via command prompt in the research folder as described here:

    # From within TensorFlow/models/research/
    for /f %i in ('dir /b object_detection\protos\*.proto') do protoc object_detection\protos\%i --python_out=.
   

4. Install the Tensorflow\models\research\object_detection package by executing the following command via command prompt:

    # From within TensorFlow/models/research/
    pip install .
   

5. Finally, add the necessary Environment Variables.

Usage

1. Download and Save the WebStreamObjectDetection.py script from this repository to /TensorFlow/models/research/object_detection.

2. Connect your computer to the same WiFi network as the ESP32-CAM. This can be done by connecting the ESP32-CAM to a router/ mobile hotspot, or letting the ESP32-CAM run in softAP mode.

3. Update the saved WebStreamObjectDetection.py with details of the WiFi connection (i.e. IP Address and Port Number) to the ESP32-CAM 'HTTP Multipart MJPEG' video stream in this line if necessary.

4. Execute the script:

    # From /TensorFlow/models/research/object_detection
    python WebStreamObjectDetection.py
   

5. Press q on the keyboard to terminate the window.