Intel® oneAPI Deep Neural Network Library Developer Guide and Reference

Developer Guide and Reference

  • 2022.1
  • 04/11/2022
  • Public Content
Contents

Local Response Normalization (LRN)

General

The LRN primitive performs a forward or backward local response normalization.
Forward
The LRN operation is defined by the following formulas (the variable names follow the standard Naming Conventions):
LaTex Math image.
LaTex Math image.
where LaTex Math image. is the
local_size
. Formulas are provided for 2D spatial data case.
Backward
The backward propagation computes LaTex Math image., based on LaTex Math image. and LaTex Math image..

Execution Arguments

When executed, the inputs and outputs should be mapped to an execution argument index as specified by the following table.
Primitive input/output
Execution argument index
LaTex Math image.
DNNL_ARG_SRC
LaTex Math image.
DNNL_ARG_DST
workspace
DNNL_ARG_WORKSPACE
LaTex Math image.
DNNL_ARG_DIFF_SRC
LaTex Math image.
DNNL_ARG_DIFF_DST

Implementation Details

General Notes
  1. During training, LRN might or might not require a workspace on forward and backward passes. The behavior is implementation specific. Optimized implementations typically require a workspace and use it to save some intermediate results from the forward pass that accelerate computations on the backward pass. To check whether a workspace is required, query the LRN primitive descriptor for the workspace. Success indicates that the workspace is required and its description will be returned.
  2. The memory format and data type for
    src
     and
    dst
     are assumed to be the same, and in the API are typically referred to as
    data
     (e.g., see
    data_desc
     in dnnl::lrn_forward::desc::desc()). The same holds for
    diff_src
     and
    diff_dst
    . The corresponding memory descriptors are referred to as
    diff_data_desc
    .
Data Type Support
The LRN primitive supports the following combinations of data types:
Propagation
Source / Destination
forward / backward
f32, bf16
forward
f16
There might be hardware and/or implementation specific restrictions. Check the Implementation Limitations section below.
Data Representation
Source, Destination, and Their Gradients
Like most other primitives, the LRN primitive expects the following tensors:
Spatial
Source / Destination
0D
LaTex Math image.
1D
LaTex Math image.
2D
LaTex Math image.
3D
LaTex Math image.
The LRN primitive is optimized for the following memory formats:
Spatial
Logical tensor
Implementations optimized for memory formats
2D
NCHW
dnnl_nchw ( dnnl_abcd ), dnnl_nhwc ( dnnl_acdb ),
optimized^
Here optimized^ means the format that comes out of any preceding compute-intensive primitive.
Post-ops and Attributes
The LRN primitive does not support any post-ops or attributes.

Implementation Limitations

  1. Refer to Data Types for limitations related to data types support.
  2. GPU
    • Supports only 2D spatial case.

Performance Tips

  1. For backward propagation, use the same memory format for
    src
    ,
    diff_dst
    , and
    diff_src
     (the format of the
    diff_dst
     and
    diff_src
     are always the same because of the API). Different formats are functionally supported but lead to highly suboptimal performance.

Example

This C++ API demonstrates how to create and execute a Local response normalization primitive in forward training propagation mode.

Product and Performance Information

1

Performance varies by use, configuration and other factors. Learn more at www.Intel.com/PerformanceIndex.