Video and Vision Processing Suite Intel® FPGA IP User Guide

ID 683329
Date 12/12/2022

A newer version of this document is available. Customers should click here to go to the newest version.

Document Table of Contents
1. About the Video and Vision Processing Suite 2. Getting Started with the Video and Vision Processing IPs 3. Video and Vision Processing IPs Functional Description 4. Video and Vision Processing IP Interfaces 5. Video and Vision Processing IP Registers 6. Video and Vision Processing IPs Software Programming Model 7. Protocol Converter Intel® FPGA IP 8. 3D LUT Intel® FPGA IP 9. AXI-Stream Broadcaster Intel® FPGA IP 10. Chroma Key Intel® FPGA IP 11. Chroma Resampler Intel® FPGA IP 12. Clipper Intel® FPGA IP 13. Clocked Video Input Intel® FPGA IP 14. Clocked Video to Full-Raster Converter Intel® FPGA IP 15. Clocked Video Output Intel® FPGA IP 16. Color Space Converter Intel® FPGA IP 17. Deinterlacer Intel® FPGA IP 18. FIR Filter Intel® FPGA IP 19. Frame Cleaner Intel® FPGA IP 20. Full-Raster to Clocked Video Converter Intel® FPGA IP 21. Full-Raster to Streaming Converter Intel® FPGA IP 22. Genlock Controller Intel® FPGA IP 23. Generic Crosspoint Intel® FPGA IP 24. Genlock Signal Router Intel® FPGA IP 25. Guard Bands Intel® FPGA IP 26. Interlacer Intel® FPGA IP 27. Mixer Intel® FPGA IP 28. Pixels in Parallel Converter Intel® FPGA IP 29. Scaler Intel® FPGA IP 30. Stream Cleaner Intel® FPGA IP 31. Switch Intel® FPGA IP 32. Tone Mapping Operator Intel® FPGA IP 33. Test Pattern Generator Intel® FPGA IP 34. Video Frame Buffer Intel® FPGA IP 35. Video Streaming FIFO Intel® FPGA IP 36. Video Timing Generator Intel® FPGA IP 37. Warp Intel® FPGA IP 38. Design Security 39. Document Revision History for Video and Vision Processing Suite User Guide

34.3. Video Frame Buffer IP Functional Description

The IP receives incoming video frames from its Intel FPGA streaming video input and writes them to external memory via an Avalon memory-mapped interface.

Received frames are available for output when the Avalon memory-mapped agent accepts the last beat of the last line.

The IP starts producing output frames when you set the OUTPUT_CONTROL bit. It selects which frame to output based on the buffer behavior parameters. You can set buffer behavior so that the IP drops incoming frames if no free buffers are available or repeats an output frame if no new frames exist to read.

Register Behavior

Bit [0] of the INPUT_STATUS register goes high when the IP starts writing the first frame. It goes low after the IP finishes handling the last expected input line, or after receiving the last line of a short frame. It returns high when the IP starts writing the next frame.

Bit [0] of the OUTPUT_STATUS register goes high when the IP starts reading the first frame. It goes low in between output frames and remains low if it has no valid frames to output.

Applications which need to know when the IP receives or creates frames can poll these status registers.

Triple-buffering Behavior

Select triple-buffering by turning on both Enable dropping of input frames and Enable repeating of output frames in the GUI.

For triple-buffering, the frame buffer uses three frame buffers, with base addresses calculated from the base address and interbuffer stride you select. The stride is the separation between frames in memory and is always fixed to the parameterized value regardless of the size of frames the IP receives. Similarly, lines are always spaced in memory according to the interline stride, regardless of the size of the line the IP receives. The IP has three buffers for triple buffering.

Broken Frame Behavior

For full variants, (lite mode off), the IP processes and creates broken frames (frames with incorrect dimensions or with the broken bit in their associated image information packet set) in the same way as good frames. Alternatively, it discards them if you turn on Enable the dropping of broken frames.

For full variants, (lite mode off), the IP detects broken frames by checking the received frame dimensions against the expected dimensions. If the IP receives and measures a broken frame without the broken bit set, the IP sets the broken bit for that frame, if the IP produces it at the output.

For full variants, (lite mode off), broken frames may result in the IP creating frames of dimensions smaller or larger than the specified image information packet and may result in it displaying partial frames.

The IP clips frames with heights or widths greater than the maxima you specify in the GUI to those values.

Double-buffering Behavior

Select double-buffering by turning off both Enable dropping of input frames and Enable repeating of output frames in the GUI.

Double buffering uses only two frame buffers, with base addresses calculated from the base address and interbuffer stride you specify.

If no free buffers are available and a new frame of video is pending at the input, frame writing stalls. The IP holds off the input by de-asserting its TREADY output until a free buffer becomes available.

If the IP finishes creating output from one buffer and the other buffer does not hold a complete, new frame, the IP stalls its output by lowering TVALID.

Auxiliary control packet support

Turn on auxiliary packets by selecting some local storage in the GUI. You can allocate local storage slots up to a maximum of 256 4-beat auxiliary control packets (64 bits of data).

If you allocate no local storage, the IP discards any auxiliary control packets it receives.

If you allocate local storage, every received frame consumes at least one of the 64 bit storage slots. The IP marks slots as empty if a frame has no associated auxiliary control packets.

The IP classes any auxiliary control packets the IP receives as owned by a parent frame according to the following rules in the Intel FPGA Streaming Video Protocol specification, which are reproduced here :

Pairs of image information and end-of-field control packets surround one field of video. The IP associates any auxiliary control packets before the current field’s end-of-field control packet with that field. In addition, the IP associates any auxiliary control packets that occur after the end-of-field control packet of the preceding field with the current field.

Figure 80.  Auxiliary control packets

The figure shows how aux1 and aux2 associates with field 1 and aux3 with field 2.

The IP associates the frame with any auxiliary control packets, if you turn on Drop/repeat auxiliary metapackets with their associated frame. If frame repeating is on, the IP repeats auxiliary packets with their parent frame. If frame dropping is on, the IP drops any auxiliary packets associated with a dropped frame.

If you turn off Drop/repeat auxiliary control metapackets with their associated frame, the IP produces all auxiliary packets once, in the order in which it receives them, regardless of whether it drops or repeats their parent frame.

If the local storage becomes full, the IP raises the auxiliary FIFO overflow sticky bit in the OUTPUT_STATUS register (bit 1) and flushes the contents and buffering starts again. Ensure you have sufficient storage for your application.

Frame statistics

The registers from 0x0144 to 0x0158 contain statistics on the numbers of frames the IP receives, drops, produces, and repeats. The NUM_INVALID_FIELDS register includes frames that the IP indicates as invalid via the broken bit and frames that the IP measures as invalid where their dimensions do not match the expected dimensions.


The frame buffer latency depends on the availability of the external memory interfaces, which may backpressure the IP’s read and write host interfaces via the av_mm_mem_read_host_waitrequest and av_mm_mem_write_host_waitrequest signals.

The streaming video output experiences backpressure via the axi4s_vid_out_tready input. The worst-case latency figures are when the frame buffer experiences no backpressure. Any backpressure increases these latencies by the same amount of cycles.

Worst case latency is on start-up, so these figures show where the frame buffer is writing and then reading the first frame after reset. The latency figures are for when the host interfaces do not have a separate clock, so all interfaces operate from the same clock. Latencies under steady-state conditions are less than or equal to these figures.

The latency figures are the same for both full and lite variants of the IP, as measured from the start of the first data packet of a new frame. The figures show the IP producing the image information packet. In lite variants of the IP, it does not produce this packet but the timing is otherwise identical.

Figure 81. Video frame buffer latencies This figure only shows some signals. The figure shows four of the frame buffer interfaces: The Intel FPGA streaming video input, the Avalon memory-mapped host writer, the Intel FPGA streaming video output and the Avalon memory-mapped host reader. The figure shows the start of a new frame of video on the streaming video input.
Table 578.  Latencies

The table shows latencies between the interfaces.

Initiating event Resultant event Latency between events (measured in clock cycles)
axi4s_vid_in_tuser[0] strobe to indicate start of frame. First av_mm_mem_write_host_write strobe of a frame write. 44
Last av_mm_mem_write_host_write strobe of a frame write. First valid cycle of outgoing image information packet on the streaming output. Full variants only. 43
Last av_mm_mem_write_host_write strobe of a frame write. First av_mm_mem_read_host_read strobe of a frame read. 74
av_mm_mem_read_host_readdatavalid strobe high to return the first frame data. axi4s_vid_out_tuser[0] strobe to indicate start of frame on streaming output. 20