Video and Image Processing Suite User Guide

ID 683416
Date 4/04/2022
Document Table of Contents
Give Feedback

12.3. Result of Output Data Type Conversion

After the calculation, the fixed point type of the results must be converted to the integer data type of the output.

This conversion is performed in four stages, in the following order:

  1. Result scaling—You can choose to scale up the results, increasing their range. This is useful to quickly increase the color depth of the output.
    • The available options are a shift of the binary point right –16 to +16 places.
    • This is implemented as a simple shift operation so it does not require multipliers.
  2. Removal of fractional bits—If any fractional bits exist, you can choose to remove them:
    • Truncate to integer—Fractional bits are removed from the data. This is equivalent to rounding towards negative infinity.
    • Round-half up—Round up to the nearest integer. If the fractional bits equal 0.5, rounding is towards positive infinity.
    • Round-half even. Round to the nearest integer. If the fractional bits equal 0.5, rounding is towards the nearest even integer.
  3. Conversion from signed to unsigned—If any negative numbers can exist in the results and the output type is unsigned, you can choose how they are converted:
    • Saturate to the minimum output value (constraining to range).
    • Replace negative numbers with their absolute positive value.
  4. Constrain to range—logic that saturates the results to the minimum and maximum output values is automatically added:
    • If any of the results are not within the minimum and maximum values allowed by the output bits per pixel
    • If any of the results are beyond the range specified by the output Guard bands (optional)