AN 708: Application Note - PCI Express DMA Reference Design Using External Memory

ID 683390
Date 5/17/2017
Public

Reference Design

Figure 1. DMA Reference Design Block Diagram


The reference design uses an external memory connected to the Intel® memory controller that can access up to 128 MB of on-board external memory. The read DMA moves the data from the system memory to the external memory. The write DMA moves the data from the external memory to the system memory.

Because the memory contains a single port, the read and the write data movers cannot access the external memory at the same time. Hence, this reference design does not demonstrate the real capability of the DMA for simultaneous reads and writes. Because the latency of the external memory is higher than that of an on-chip memory, the throughput achieved with an external memory is lower compared to the throughput of on-chip memory.

Figure 2.  DMA Reference Design Qsys Connections


Table 10.  AVMM DMA Reference Design Blocks and Port Descriptions
Function Port Description
AVMM DMA pcie_256_dma This is the 256-bit Avalon Memory Mapped module with DMA. It consists of one read and one write data mover, a RX master, a TX slave, and an internal descriptor controller. The descriptor controller can also be external to the DMA module.
RXM_BAR0 N/A This is an Avalon-MM master port. It passes the memory access from PCIe host to PCIe BAR0. The host uses this port to program the descriptor controller. Because this reference design uses an internal descriptor controller, the port connection is not shown in Qsys. The connection is inside the pcie_256_dma module.
RXM_BAR4 Rxm_BAR4 This is an Avalon-MM master port. It passes the memory access from PCIe host to PCIe BAR4. In the reference design, it connects to one port of the external DDR3 or DDR4 memory. The PCIe host accesses the memory through PCIe BAR4

In a typical application, system software controls this port to initialize random data in the external memory. Software also reads the data back to verify correct operation.

TXS Txs This is an Avalon-MM slave port. In a typical application, an Avalon-MM master controls this port to send memory reads or writes to the PCIe domain.

The descriptor controller uses it to write DMA status back to descriptor space in the PCIe domain when the DMA completes its operation. The descriptor controller also uses this port to send upstream MSI interrupts.

Read Data Mover dma_rd_master This is an Avalon-MM master port.

The Read Data Mover moves data from the PCIe domain to the external DDR3 or DDR4 memory during normal read DMA operation. The Read Data Mover also fetches the descriptors from the PCIe domain and writes them to the FIFO in the Descriptor Controller. There are two separate groups of descriptors, one for write DMA and another for read DMA. Because of these two separate groups, the dma_rd_master port is connected to two ports. It connects to wr_dts_slave for the write DMA descriptor FIFO and rd_dts_slave for the read DMA descriptor FIFO.

Write Data Mover dma_wr_master This is an Avalon-MM master port.

The Write Data Mover reads data from the external DDR3 or DDR4 memory and then writes data to the PCIe domain. In this reference design, because the DDR3 or DDR4 controller has a single port, the Write Data Mover uses the same port as the Read Data Mover.

FIFO in Descriptor Controller wr_dts_slave and rd_dts_slave This is an Avalon-MM slave port for the FIFO in the Descriptor Controller.

When the Read Data Mover fetches the descriptors from system memory, it writes the descriptors to the FIFO using this port.

Because there are two separate groups of descriptors for read and write, there are two ports.

For the write DMA, the FIFO address is from 0x8000_20000x8000_3FFF.

For the read DMA, the FIFO address is from 0x8000_00000x8000_1FFF.

Control in Descriptor Controller wr_dcm_master and rd_dcm_master The control block in the Descriptor Controller has one transmit and one receive port, one for read DMA and another one for write DMA. The receive port is connected to the RXM_BAR0 and the transmit port is connected to the Txs.

The receive path from the RXM_BAR0 connects internally. It is not shown in the DMA Reference Design Qsys Connections figure. For the transmit path, both read and write DMA ports connect to the Txs externally. These ports are visible in the DMA Reference Design Qsys Connections figure.

DDR3 or DDR4 Controller Intel® DDR3 or DDR4 controller This is a single port DDR3 or DDR4 controller. It can access up to 128 MB external memory.