CIFAR-10 Classification using Intel® Optimization for TensorFlow*

Abstract

This work demonstrates the experiments conducted to train and test the deep learning AlexNet* topology with the Intel® optimized TensorFlow* library using CIFAR-10 classification data on Intel® Xeon® scalable processor powered machines. These experiments were conducted with options set at run time. From these runs, the training accuracy, validation accuracy, and testing accuracy numbers were captured for different environment configurations to identify the optimal combination of environment configurations. For the optimal combination identified, the top-1 and top-5 accuracies were plotted.

Document Content

Environment Setup

The following hardware and software environments were used to perform the experiments.

Hardware

Table 1. Intel® Xeon® Gold 6128 configuration

Software

Table 2. On Intel Xeon Gold processor

The Intel Optimized TensorFlow wheel was installed through pip.

pip install https://anaconda.org/intel/tensorflow/1.4.0/download/tensorflow-1.4.0-cp36-cp36m-linux_x86_64.whl

Network Topology and Model Training

This section details the dataset adopted, AlexNet architecture, and training the model in the current work.

Dataset

The CIFAR-10 dataset chosen for these experiments consists of 60,000 32 x 32 color images in 10 classes. Each class has 6,000 images. The 10 classes are: airplane, automobile, bird, cat, deer, dog, frog, horse, ship, and truck.

The dataset was taken from Kaggle*³. The following figure shows a sample set of images for each classification.

For the experiments, out of the 60,000 images, 50,000 images were chosen for training and 10,000 images for testing.

AlexNet* Architecture

The AlexNet network is made of five convolution layers, max-pooling layers, dropout layers, and three fully connected layers. The network was designed to be used for classification with 1,000 possible categories.

Model Training

In these experiments, it was decided to train the model from the beginning using the CIFAR-10 dataset. The dataset is split as 50,000 images for training and validation and 10,000 images for testing.

Experimental Runs

The experiment was conducted in two steps.

Step 1

Training was done with different combinations of environment options. The batch size and number of epochs were fixed in this step. The following table details the options chosen and the results for each run. In addition to the following settings, the Intel® Xeon® processors were made numactl-aware.

It has been observed that the training accuracy for all the runs has minimal difference. Setting the OMP_NUM_THREADS and the intra_op parameters to either 12, 16 or 24 showed that the training performance is 1.4 times faster (refer to configurations) compared to setting these parameters to either 6, 8 or 10.

Step 2

The OMP_NUM_THREADS and the intra_op parameters setting were set to 12 and the runs were performed for different batch sizes. The Top-1 training, validation and inference accuracies were captured in each case. Runs were performed for 25, 100, and 1000 epochs. The results are listed in the following table.

It was observed that while using a larger batch, there is a degradation in the quality of the model as there is a reduction in the stochasticity of the gradient descent. The accuracy fall is steeper when there is an increase in the batch size from 128 to 256. In general, the performance of processors is better if the batch size is a power of 2. Considering this, it was decided to perform runs with a higher epoch count on batch sizes of 64 and 128.

As the epoch count increased, the network showed improvement in accuracy. Also, the validation and testing accuracy was closely matching and the model seems to improve generalization. Continuing the experiments with larger number of epochs, the top-1 and top-5 training and testing accuracies were observed as follows.

The following graph represents the top-1 and top-5 accuracies for training for every 100 epochs:

Comparing the top-1 training and testing accuracy, it can be inferred that the network tends to over fit after 500 epochs. The reason could be that the model is training on the same data again.

Conclusion

Observations from the experiments on training the AlexNet topology on Intel® Xeon® Gold processor powered machines, with Intel optimized TensorFlow using the CIFAR-10 classification dataset showed that by optimally setting environment parameters (especially OMP_NUM_THREADS and intra_op) better throughput and reduced training times can be achieved. With the optimal environment parameter configuration, choosing a smaller batch size helped improve the training accuracy.

Similar runs can be performed on future releases of Intel optimized TensorFlow that supports distributed mode to experience enhanced performance.

About the Author

Rajeswari Ponnuru, Ajit Kumar Pookalangara, Ravi Keron Nidamarty, and Rishabh Kumar Jain are Technical Consulting Engineers working with the Intel® AI Developer Program.

Acronyms and Abbreviations

Configurations

For performance reference under Experimental Runs section: 

Hardware: Refer to Hardware under Environment Setup 
Software: Refer to Software under Environment Setup
While leaving the other settings same, the following 2 settings were changed while repeating the runs:
Runtime settings 1: "OMP_NUM_THREADS" = "6";'intra_op' = 6
Runtime settings 2: "OMP_NUM_THREADS" = "8";'intra_op' = 8
Runtime settings 3: "OMP_NUM_THREADS" = "10";'intra_op' = 10
Runtime settings 4: "OMP_NUM_THREADS" = "12";'intra_op' = 12
Runtime settings 5: "OMP_NUM_THREADS" = "16";'intra_op' = 16
Runtime settings 6: "OMP_NUM_THREADS" = "24";'intra_op' = 24

Test performed: Executed the script ~/Tensorflow_Cifar10_Alexnet_Experiments/ Tensorflow_Cifar10-V1-Alexnet_flags.py 

For more information go to Product Performance.

References

1. TensorFlow* Optimizations on Modern Intel® Architecture
2. Alexnet topology diagram
3. CIFAR-10 dataset taken from

Related Resources

Alexnet details

About CIFAR-10 data