Optimize a BERT-Large FP32 Training Container with TensorFlow*

Pull Command

docker pull intel/language-modeling:tf-latest-bert-large-fp32-training

Description

This document has instructions for running BERT FP32 training using Intel® Optimization for TensorFlow*.

For all fine-tuning, the datasets (Stanford Question Answering Dataset [SQuAD], MultiNLI, Microsoft* Research Paraphrase Corpus [MRPC], and more) and checkpoints should be downloaded as mentioned in the Google* BERT repository.

Refer to Google reference page for checkpoints.

Datasets

Follow instructions in BERT Large datasets to download and preprocess the dataset. You can do either classification training or fine-tuning using SQuAD.

Quick Start Scripts

Docker*

The BERT-Large FP32 training model container includes the scripts and libraries needed to run BERT-Large FP32 fine-tuning. To run one of the quick start scripts using this container, you'll need to provide volume mounts for the pretrained model, dataset, and an output directory where log and checkpoint files will be written. If switching between running squad and classifier training or running classifier training multiple times, use a new empty OUTPUT_DIR to prevent incompatible checkpoints from getting picked up. See the list of quickstart scripts for details on the different options.

The snippet below shows a quick start script running with a single instance:

CHECKPOINT_DIR=<path to the pretrained bert model directory>
DATASET_DIR=<path to the dataset being used>
OUTPUT_DIR=<directory where checkpoints and log files will be saved>

docker run \
  --env CHECKPOINT_DIR=${CHECKPOINT_DIR} \
  --env DATASET_DIR=${DATASET_DIR} \
  --env OUTPUT_DIR=${OUTPUT_DIR} \
  --env http_proxy=${http_proxy} \
  --env https_proxy=${https_proxy} \
  --volume ${CHECKPOINT_DIR}:${CHECKPOINT_DIR} \
  --volume ${DATASET_DIR}:${DATASET_DIR} \
  --volume ${OUTPUT_DIR}:${OUTPUT_DIR} \
  --privileged --init -it \
  intel/language-modeling:tf-latest-bert-large-fp32-training \
  /bin/bash quickstart/<script name>.sh

To run distributed training (one message passing interface [MPI] process per socket) for better throughput, set the MPI_NUM_PROCESSES var to the number of sockets to use. Note that the global batch size is mpi_num_processes * train_batch_size and sometimes the learning rate needs to be adjusted for convergence. By default, the script uses square root learning rate scaling.

For fine-tuning tasks like BERT, state-of-the-art accuracy can be achieved via parallel training without synchronizing gradients between MPI workers. The mpi_workers_sync_gradients=[True/False] var controls whether the MPI workers sync gradients. By default it is set to False meaning the workers are training independently and the best performing training results will be picked in the end. To enable gradients synchronization, set the mpi_workers_sync_gradients to True in BERT options. To modify the bert options, modify the quick start .sh script or call the launch_benchmarks.py script directly with your preferred args.

CHECKPOINT_DIR=<path to the pretrained bert model directory>
DATASET_DIR=<path to the dataset being used>
OUTPUT_DIR=<directory where checkpoints and log files will be saved>
MPI_NUM_PROCESSES=<number of sockets to use>

docker run \
  --env CHECKPOINT_DIR=${CHECKPOINT_DIR} \
  --env DATASET_DIR=${DATASET_DIR} \
  --env OUTPUT_DIR=${OUTPUT_DIR} \
  --env MPI_NUM_PROCESSES=${MPI_NUM_PROCESSES} \
  --env http_proxy=${http_proxy} \
  --env https_proxy=${https_proxy} \
  --volume ${CHECKPOINT_DIR}:${CHECKPOINT_DIR} \
  --volume ${DATASET_DIR}:${DATASET_DIR} \
  --volume ${OUTPUT_DIR}:${OUTPUT_DIR} \
  --privileged --init -it \
  intel/language-modeling:tf-latest-bert-large-fp32-training \
  /bin/bash quickstart/<script name>.sh

Documentation and Sources

Get Started​
Docker* Repository
Main GitHub*
Readme
Release Notes
Get Started Guide

Code Sources
Dockerfile
Report Issue

License Agreement

LEGAL NOTICE: By accessing, downloading or using this software and any required dependent software (the “Software Package”), you agree to the terms and conditions of the software license agreements for the Software Package, which may also include notices, disclaimers, or license terms for third party software included with the Software Package. Please refer to the license file for additional details.

View All Containers and Solutions 🡢