Run ResNet-34* SSD FP32 Training with a TensorFlow* Model Package

Download Command

wget https://storage.googleapis.com/intel-optimized-tensorflow/models/v2_3_0/ssd-resnet34-fp32-training.tar.gz

Description

This document has instructions for running ResNet34* SSD FP32 training using Intel® Optimization for TensorFlow*.

Datasets

ResNet34 SSD training uses the COCO dataset. Use the following instructions to download and preprocess the dataset.

1. Download and extract the 2017 training images and annotations for the COCO dataset:

   export MODEL_WORK_DIR=$(pwd)
   
   # Download and extract train images
   wget http://images.cocodataset.org/zips/train2017.zip
   unzip train2017.zip
   
   # Download and extract annotations
   wget http://images.cocodataset.org/annotations/annotations_trainval2017.zip
   unzip annotations_trainval2017.zip

2. Since we are only using the train and validation dataset in this example, we will create an empty directory and empty annotations JSON file to pass as the test directories in the next step.

   # Create an empty dir to pass for validation and test images
   mkdir empty_dir
   
   # Add an empty .json file to bypass validation/test image preprocessing
   cd annotations
   echo "{ \"images\": {}, \"categories\": {}}" > empty.json
   cd ..

3. Use the TensorFlow models repo scripts to convert the raw images and annotations to the TF records format.

   git clone https://github.com/tensorflow/models.git tf_models
   cd tf_models
   git checkout 7a9934df2afdf95be9405b4e9f1f2480d748dc40
   cd ..

4. Install the prerequisites mentioned in the TensorFlow models object detection installation doc and run protobuf compilation on the code that was cloned in the previous step.

5. After your envionment is set up, run the conversion script:

   cd tf_models/research/object_detection/dataset_tools/
   
   # call script to do conversion
   python create_coco_tf_record.py --logtostderr \
         --train_image_dir="$MODEL_WORK_DIR/train2017" \
         --val_image_dir="$MODEL_WORK_DIR/empty_dir" \
         --test_image_dir="$MODEL_WORK_DIR/empty_dir" \
         --train_annotations_file="$MODEL_WORK_DIR/annotations/instances_train2017.json" \
         --val_annotations_file="$MODEL_WORK_DIR/annotations/empty.json" \
         --testdev_annotations_file="$MODEL_WORK_DIR/annotations/empty.json" \
         --output_dir="$MODEL_WORK_DIR/output"

   The coco_train.record-*-of-* files are what we will use in this training example. Set the output of the preprocessing script (export DATASET_DIR=$MODEL_WORK_DIR/output) when running quick start scripts.

Quick Start Scripts

Bare Metal

To run on bare metal, the following prerequisites must be installed in your environment:

• Python* 3
• Contextlib2
• Cpio
• Cython
• Horovod*
• Intel® Optimization for Tensorflow*
• Jupyter*
• Lxml
• Matplotlib
• NumPy 1.17.4
• OpenCV
• Open MPI
• OpenSSH
• Pillow 7.1.0
• Protoc
• Pycocotools
• TensorFlow add-ins 0.8.1

For more information, see the installation instructions for object detection models at the TensorFlow Model Garden repository.

After installing the prerequisites, download and untar the model package. Clone the TensorFlow Model Garden repo at the commit specified below, and set the TF_MODELS_DIR environment variable to point to that directory. Set the DATASET_DIR to point to the directory with COCO training TF records files and the OUTPUT_DIR to the location where logs will be written. You can optionally set the MPI_NUM_PROCESSES environment variable (defaults to 1). After all the setup is complete, run the quick start script.

export DATASET_DIR=<path to the dataset>
export OUTPUT_DIR=<directory where log files will be written>
export MPI_NUM_PROCESSES=<number of MPI processes (optional, defaults to 1)>

# Clone the tensorflow/models repo at the specified commit
git clone https://github.com/tensorflow/models.git tf_models
cd tf_models
export TF_MODELS_DIR=$(pwd)
git checkout 8110bb64ca63c48d0caee9d565e5b4274db2220a
cd ..

# Download and extract the model package, then run a quickstart script
wget https://storage.googleapis.com/intel-optimized-tensorflow/models/v2_3_0/ssd-resnet34-fp32-training.tar.gz
tar -xzf ssd-resnet34-fp32-training.tar.gz
cd ssd-resnet34-fp32-training

quickstart/fp32_training.sh

Documentation and Sources

Get Started​
Main GitHub*
Readme
Release Notes
Get Started Guide

Code Sources
Report Issue

License Agreement

LEGAL NOTICE: By accessing, downloading or using this software and any required dependent software (the “Software Package”), you agree to the terms and conditions of the software license agreements for the Software Package, which may also include notices, disclaimers, or license terms for third party software included with the Software Package. Please refer to the license file for additional details.

Related Containers and Solutions

ResNet34 SSD FP32 Training TensorFlow* Container

View All Containers and Solutions 🡢