Developer Guide and Reference

  • 2022.1
  • 04/11/2022
  • Public Content
Contents

Eltwise

General

Forward
The eltwise primitive applies an operation to every element of the tensor (the variable names follow the standard Naming Conventions):
LaTex Math image.
For notational convenience, in the formulas below we will denote individual element of LaTex Math image., LaTex Math image., LaTex Math image., and LaTex Math image. tensors via s, d, ds, and dd respectively.
The following operations are supported:
LaTex Math image.
LaTex Math image.
LaTex Math image.
Difference Between Forward Training and Forward Inference
There is no difference between the dnnl_forward_training and dnnl_forward_inference propagation kinds.
Backward
The backward propagation computes LaTex Math image. based on LaTex Math image. and LaTex Math image. tensors. However, some operations support a computation using LaTex Math image. memory produced during the forward propagation. Refer to the table above for a list of operations supporting destination as input memory and the corresponding formulas.
Exceptions
The eltwise primitive with algorithm round does not support backward propagation.

Execution Arguments

When executed, the inputs and outputs should be mapped to an execution argument index as specified by the following table.
Primitive input/output
Execution argument index
LaTex Math image.
DNNL_ARG_SRC
LaTex Math image.
DNNL_ARG_DST
LaTex Math image.
DNNL_ARG_DIFF_SRC
LaTex Math image.
DNNL_ARG_DIFF_DST
LaTex Math image.

Implementation Details

General Notes
  1. All eltwise primitives have a common initialization function (e.g., dnnl::eltwise_forward::desc::desc()) which takes both parameters LaTex Math image., and LaTex Math image.. These parameters are ignored if they are unused.
  2. The memory format and data type for LaTex Math image. and LaTex Math image. are assumed to be the same, and in the API are typically referred as
    data
    (e.g., see
    data_desc
    in dnnl::eltwise_forward::desc::desc()). The same holds for LaTex Math image. and LaTex Math image.. The corresponding memory descriptors are referred to as
    diff_data_desc
    .
  3. Both forward and backward propagation support in-place operations, meaning that LaTex Math image. can be used as input and output for forward propagation, and LaTex Math image. can be used as input and output for backward propagation. In case of an in-place operation, the original data will be overwritten. Note, however, that some algorithms for backward propagation require original LaTex Math image., hence the corresponding forward propagation should not be performed in-place for those algorithms. Algorithms that use LaTex Math image. for backward propagation can be safely done in-place.
  4. For some operations it might be beneficial to compute backward propagation based on LaTex Math image., rather than on LaTex Math image., for improved performance.
  5. For logsigmoid original formula LaTex Math image. was replaced by LaTex Math image. for numerical stability.
For operations supporting destination memory as input, LaTex Math image. can be used instead of LaTex Math image. when backward propagation is computed. This enables several performance optimizations (see the tips below).
Data Type Support
The eltwise primitive supports the following combinations of data types:
Propagation
Source / Destination
Intermediate data type
forward / backward
f32, bf16
f32
forward
f16
f16
forward
s32 / s8 / u8
f32
There might be hardware and/or implementation specific restrictions. Check Implementation Limitations section below.
Here the intermediate data type means that the values coming in are first converted to the intermediate data type, then the operation is applied, and finally the result is converted to the output data type.
Data Representation
The eltwise primitive works with arbitrary data tensors. There is no special meaning associated with any logical dimensions.
Post-Ops and Attributes
Propagation
Type
Operation
Description
Restrictions
Forward
Post-op
Applies a Binary operation to the result
General binary post-op restrictions

Implementation Limitations

  1. Refer to Data Types for limitations related to data types support.

Performance Tips

  1. For backward propagation, use the same memory format for LaTex Math image., LaTex Math image., and LaTex Math image. (the format of the LaTex Math image. and LaTex Math image. are always the same because of the API). Different formats are functionally supported but lead to highly suboptimal performance.
  2. Use in-place operations whenever possible (see caveats in General Notes).
  3. As mentioned above for all operations supporting destination memory as input, one can use the LaTex Math image. tensor instead of LaTex Math image.. This enables the following potential optimizations for training:
    • Such operations can be safely done in-place.
    • Moreover, such operations can be fused as a post-op with the previous operation if that operation does not require its LaTex Math image. to compute the backward propagation (e.g., if the convolution operation satisfies these conditions).

Example

This C++ API example demonstrates how to create and execute an Element-wise primitive in forward training propagation mode.

Product and Performance Information

1

Performance varies by use, configuration and other factors. Learn more at www.Intel.com/PerformanceIndex.