Developer Guide and Reference

  • 2022.1
  • 04/11/2022
  • Public Content
Contents

Inner Product

General

The inner product primitive (sometimes called fully connected) treats each activation in the minibatch as a vector and computes its product with a weights 2D tensor producing a 2D tensor as an output.
Forward
More precisely, let LaTex Math image., LaTex Math image., LaTex Math image. and LaTex Math image. be LaTex Math image., LaTex Math image., LaTex Math image., and LaTex Math image. tensors, respectively (variable names follow the standard Naming Conventions). Then:
LaTex Math image.
In cases where the LaTex Math image. and LaTex Math image. tensors have spatial dimensions, they are flattened to 2D. For example, if they are 4D LaTex Math image. and LaTex Math image. tensors, then the formula above is applied with LaTex Math image.. In such cases, the LaTex Math image. and LaTex Math image. tensors must have equal spatial dimensions (e.g. LaTex Math image. and LaTex Math image. for 4D tensors).
Difference Between Forward Training and Forward Inference
There is no difference between the dnnl::prop_kind::forward_training and dnnl::prop_kind::forward_inference propagation kinds.
Backward
The backward propagation computes LaTex Math image. based on LaTex Math image. and LaTex Math image..
The weights update computes LaTex Math image. and LaTex Math image. based on LaTex Math image. and LaTex Math image..
The optimized memory formats LaTex Math image. and LaTex Math image. might be different on forward propagation, backward propagation, and weights update.

Execution Arguments

When executed, the inputs and outputs should be mapped to an execution argument index as specified by the following table.
Primitive input/output
Execution argument index
LaTex Math image.
DNNL_ARG_SRC
LaTex Math image.
DNNL_ARG_WEIGHTS
LaTex Math image.
DNNL_ARG_BIAS
LaTex Math image.
DNNL_ARG_DST
LaTex Math image.
DNNL_ARG_DIFF_SRC
LaTex Math image.
DNNL_ARG_DIFF_WEIGHTS
LaTex Math image.
DNNL_ARG_DIFF_BIAS
LaTex Math image.
DNNL_ARG_DIFF_DST
LaTex Math image.

Implementation Details

General Notes
N/A.
Data Types
Inner product primitive supports the following combination of data types for source, destination, weights, and bias:
Propagation
Source
Weights
Destination
Bias
forward / backward
f32
f32
f32
f32
forward
f16
f16
f16, u8, s8
f16
forward
u8, s8
s8
u8, s8, s32, bf16, f32
u8, s8, s32, bf16, f32
forward
bf16
bf16
f32, bf16
f32, bf16
backward
f32, bf16
bf16
bf16
weights update
bf16
f32, bf16
bf16
f32, bf16
Data Representation
Like other CNN primitives, the inner product primitive expects the following tensors:
Spatial
Source
Destination
Weights
1D
LaTex Math image.
LaTex Math image.
LaTex Math image.
2D
LaTex Math image.
LaTex Math image.
LaTex Math image.
3D
LaTex Math image.
LaTex Math image.
LaTex Math image.
Memory format of data and weights memory objects is critical for inner product primitive performance. In the oneDNN programming model, inner product primitive is one of the few primitives that support the placeholder format dnnl::memory::format_tag::any (shortened to
any
from now on) and can define data and weight memory objects formats based on the primitive parameters. When using
any
it is necessary to first create an inner product primitive descriptor and then query it for the actual data and weight memory objects formats.
The table below shows the combinations for which plain memory formats the inner product primitive is optimized for. For the destination tensor (which is always LaTex Math image.) the memory format is always dnnl::memory::format_tag::nc (dnnl::memory::format_tag::ab).
Spatial
Source / Weights logical tensor
Implementation optimized for memory formats
0D
NC / OI
0D
NC / OI
1D
NCW / OIW
1D
NCW / OIW
2D
NCHW / OIHW
2D
NCHW / OIHW
3D
NCDHW / OIDHW
3D
NCDHW / OIDHW
Post-Ops and Attributes
Post-ops and attributes enable you to modify the behavior of the inner product primitive by chaining certain operations after the inner product operation. The following post-ops are supported by inner product primitives:
Propagation
Type
Operation
Description
Restrictions
forward
attribute
Output scale
Scales the result of inner product by given scale factor(s)
int8 inner products only
forward
post-op
Applies an Eltwise operation to the result
forward
post-op
Adds the operation result to the destination tensor instead of overwriting it
forward
post-op
Applies a Binary operation to the result
General binary post-op restrictions
To facilitate dynamic quantization, the primitive supports run-time output scales. That means a user could configure attributes with output scales set to the DNNL_RUNTIME_F32_VAL wildcard value instead of the actual scales, if the scales are not known at the primitive descriptor creation stage. In this case, the user must provide the scales as an additional input memory object with argument
DNNL_ARG_ATTR_OUTPUT_SCALES
during the execution stage.

Implementation Limitations

  1. Check Data Types.
  2. The CPU engine does not support
    u8
    or
    s8
    data type for
    dst
    with
    f16
    src
    and
    weights
    .

Performance Tips

  • Use dnnl::memory::format_tag::any for source, weights, and destinations memory format tags when create an inner product primitive to allow the library to choose the most appropriate memory format.

Example

This C++ API example demonstrates how to create and execute an Inner Product primitive.
Key optimizations included in this example:
  • Primitive attributes with fused post-ops;
  • Creation of optimized memory format from the primitive descriptor.

Product and Performance Information

1

Performance varies by use, configuration and other factors. Learn more at www.Intel.com/PerformanceIndex.