General

For notational convenience, in the formulas below we will denote individual element of , , , and tensors via s, d, ds, and dd respectively.

Operation	oneDNN algorithm kind	Forward formula	Backward formula (from src)	Backward formula (from dst)
abs	dnnl_eltwise_abs
bounded_relu	dnnl_eltwise_bounded_relu
clip	dnnl_eltwise_clip
clip_v2	dnnl_eltwise_clip_v2 dnnl_eltwise_clip_v2_use_dst_for_bwd
elu	dnnl_eltwise_elu dnnl_eltwise_elu_use_dst_for_bwd
exp	dnnl_eltwise_exp dnnl_eltwise_exp_use_dst_for_bwd
gelu_erf	dnnl_eltwise_gelu_erf
gelu_tanh	dnnl_eltwise_gelu_tanh
hardswish	dnnl_eltwise_hardswish
linear	dnnl_eltwise_linear
log	dnnl_eltwise_log
logistic	dnnl_eltwise_logistic dnnl_eltwise_logistic_use_dst_for_bwd
logsigmoid	dnnl_eltwise_logsigmoid
mish	dnnl_eltwise_mish
pow	dnnl_eltwise_pow
relu	dnnl_eltwise_relu dnnl_eltwise_relu_use_dst_for_bwd
round	dnnl_eltwise_round
soft_relu	dnnl_eltwise_soft_relu
sqrt	dnnl_eltwise_sqrt dnnl_eltwise_sqrt_use_dst_for_bwd
square	dnnl_eltwise_square
swish	dnnl_eltwise_swish
tanh	dnnl_eltwise_tanh dnnl_eltwise_tanh_use_dst_for_bwd

The backward propagation computes based on and tensors. However, some operations support a computation using memory produced during the forward propagation. Refer to the table above for a list of operations supporting destination as input memory and the corresponding formulas.

Execution Arguments

Primitive input/output	Execution argument index
	DNNL_ARG_SRC
	DNNL_ARG_DST
	DNNL_ARG_DIFF_SRC
	DNNL_ARG_DIFF_DST
	DNNL_ARG_ATTR_MULTIPLE_POST_OP(binary_post_op_position) | DNNL_ARG_SRC_1

Implementation Details

All eltwise primitives have a common initialization function (e.g., dnnl::eltwise_forward::desc::desc()) which takes both parameters , and . These parameters are ignored if they are unused.
The memory format and data type for and are assumed to be the same, and in the API are typically referred as
data
(e.g., see
data_desc
in dnnl::eltwise_forward::desc::desc()). The same holds for and . The corresponding memory descriptors are referred to as
diff_data_desc
.
Both forward and backward propagation support in-place operations, meaning that can be used as input and output for forward propagation, and can be used as input and output for backward propagation. In case of an in-place operation, the original data will be overwritten. Note, however, that some algorithms for backward propagation require original , hence the corresponding forward propagation should not be performed in-place for those algorithms. Algorithms that use for backward propagation can be safely done in-place.
For some operations it might be beneficial to compute backward propagation based on , rather than on , for improved performance.
For logsigmoid original formula was replaced by for numerical stability.

For operations supporting destination memory as input, can be used instead of when backward propagation is computed. This enables several performance optimizations (see the tips below).

Implementation Limitations

Refer to Data Types for limitations related to data types support.

Performance Tips

For backward propagation, use the same memory format for , , and (the format of the and are always the same because of the API). Different formats are functionally supported but lead to highly suboptimal performance.
Use in-place operations whenever possible (see caveats in General Notes).
As mentioned above for all operations supporting destination memory as input, one can use the tensor instead of . This enables the following potential optimizations for training:
Such operations can be safely done in-place.
Moreover, such operations can be fused as a post-op with the previous operation if that operation does not require its to compute the backward propagation (e.g., if the convolution operation satisfies these conditions).

Example