Local Response Normalization (LRN)
General
The LRN primitive performs a forward or backward local response normalization.
Forward
The LRN operation is defined by the following formulas (the variable names follow the standard Naming Conventions):
LRN across channels :

LRN within channel :

where
is the

local_size
. Formulas are provided for 2D spatial data case.Backward
The backward propagation computes
, based on
and
.



Execution Arguments
When executed, the inputs and outputs should be mapped to an execution argument index as specified by the following table.
Primitive input/output | Execution argument index |
---|---|
![]() | DNNL_ARG_SRC |
![]() | DNNL_ARG_DST |
workspace | DNNL_ARG_WORKSPACE |
![]() | DNNL_ARG_DIFF_SRC |
![]() | DNNL_ARG_DIFF_DST |
Implementation Details
General Notes
- During training, LRN might or might not require a workspace on forward and backward passes. The behavior is implementation specific. Optimized implementations typically require a workspace and use it to save some intermediate results from the forward pass that accelerate computations on the backward pass. To check whether a workspace is required, query the LRN primitive descriptor for the workspace. Success indicates that the workspace is required and its description will be returned.
- The memory format and data type forsrcanddstare assumed to be the same, and in the API are typically referred to asdata(e.g., seedata_descin dnnl::lrn_forward::desc::desc()). The same holds fordiff_srcanddiff_dst. The corresponding memory descriptors are referred to asdiff_data_desc.
Data Type Support
The LRN primitive supports the following combinations of data types:
Propagation | Source / Destination |
---|---|
forward / backward | f32, bf16 |
forward | f16 |
There might be hardware and/or implementation specific restrictions. Check the Implementation Limitations section below.
Data Representation
Source, Destination, and Their Gradients
Like most other primitives, the LRN primitive expects the following tensors:
Spatial | Source / Destination |
---|---|
0D | ![]() |
1D | ![]() |
2D | ![]() |
3D | ![]() |
The LRN primitive is optimized for the following memory formats:
Here optimized^ means the format that comes out of any preceding compute-intensive primitive.
Post-ops and Attributes
The LRN primitive does not support any post-ops or attributes.
Implementation Limitations
- Refer to Data Types for limitations related to data types support.
- GPU
- Supports only 2D spatial case.
Performance Tips
- For backward propagation, use the same memory format forsrc,diff_dst, anddiff_src(the format of thediff_dstanddiff_srcare always the same because of the API). Different formats are functionally supported but lead to highly suboptimal performance.
Example
This C++ API demonstrates how to create and execute a Local response normalization primitive in forward training propagation mode.