19.3.2. FIR Filter Precision
- You may parameterize the input data to between 8 to 16 bits per color per pixel. The IP treats this data as unsigned integer data. You may enable optional guard bands at the input to keep the data inside a reduced range of values.
- You may parameterize the coefficient data up to a total width of 18 bits per coefficient. The coefficients may be signed or unsigned and contain up to 18 fractional bits.
- You may parameterize the output data to between 8 to 16 bits per color per pixel, and the selected output data width may be different from the input data width.
To convert from the full precision result of the filtering to the selected output precision, the IP first rounds up the value to remove the required number of fraction bits. Then the IP saturates the value. You may select how many fraction bits should be preserved in the final output using the IP parameter editor. As with the input data, the output data is treated as unsigned, so the IP clips any negative values that result from the filtering to 0. Any values greater than the maximum value that can be represented in the selected number of bits per color per pixel are clipped to this maximum value.