AN794: Arria 10 Low Latency Ethernet 10G MAC and XAUI PHY Reference Design

RSS

AN-794 | 2017.02.01

Arria 10 Low Latency Ethernet 10G MAC and XAUI PHY Reference Design

This reference design demonstrates the Low Latency Ethernet 10G IP solution for Arria^® 10 devices.

This design uses Intel's Low Latency Ethernet 10G Media Access Controller (MAC) and XAUI PHY IP cores with a dual XAUI small form factor pluggable plus (SFP+) high-speed mezzanine card (HSMC) board and FPGA mezzanine card (FMC) to high-speed mezzanine card (HSMC) adapter board on Arria^® 10 FPGA development kit.

The design provides flexible test and demonstration platforms on which you can control, test, and monitor the Ethernet operations using system loopback at various points.

This design offers the following features:

Loopback points that include XGMII and serial physical medium attachment (PMA) interface in the Arria^® 10 FPGA development board, and PMA interface in the Broadcom PHY BCM8727 chip on the Dual XAUI to SFP+ HSMC board.
External optical loopback test at HSMC board SFP+ modules.
Sequential random bursts tests. You can configure the number of packets, payload-data type, and payload size for each burst.
Packet statistics for traffic generator, monitor, MAC transmitter (TX) and MAC receiver (RX).
Packet classification for different frame sizes transmitted and received by the MAC.
Throughput for the traffic received by the traffic monitor.
System Console user interface. This TCL-based user interface allows you to dynamically configure and monitor any registers in the reference design.

Figure 1. System Architecture Overview. This figure shows the high-level overview of the design's system architecture.

Figure 2. Ethernet Subsystem Clocking Scheme. This figure shows the clocking scheme for the Ethernet subsystem.

Figure 3. Design Top Level Reset Scheme. This figure shows the reset scheme. Use the master reset signal (csr_rst_n) to reset the MAC, PHY, MDIO, address decoder, reset synchronizer, and synchronizer. This signal connects to hard reset button.

Design Components

The reference design consists of three main subsystems: 10GBASE-X Ethernet, Dual XAUI to SFP+ HSMC board, and PC and System Console.

Table 1. Subsystem Design Components. This table lists the components of the design's subsystems.
Component	Description
Low Latency Ethernet 10G MAC IP Core	This IP core handles the flow of data through the XAUI PHY IP core. On transmit path, the MAC accepts client frames and constructs Ethernet frames before forwarding them to the PHY layer. Similarly on the receive path, the MAC accepts Ethernet frames through the PHY layer, performs checks and removes the relevant fields before forwarding the frames to the client. For this design, the MAC uses the memory-based statistics counters.
XAUI PHY IP Core	The XAUI PHY IP core sets to soft XAUI by default.
Traffic Controller	The traffic controller consists of: Traffic generator—injects client packet bursts into the MAC TX core. Traffic monitor—receives packet bursts from the MAC RX core. The traffic controller connects to the Avalon-ST single-clock FIFO in the Ethernet subsystem through the Avalon Streaming (Avalon-ST) interface.
Ethernet Packet Generator	This module consists of Avalon Memory-Mapped (Avalon-MM) registers, Ethernet packet generation block, CRC generator, and shift register.
Ethernet Packet Monitor	This module verifies the payload of received packets and collects information from the statistics counters. This consists of Avalon-MM registers and CRC checkers.
MDIO IP Core	This IP core enables you to control the Broadcom PHY BCM8727 chip on the Dual XAUI to SFP+ HSMC board. You can access the external PHY registers through a pair of indirect registers to specify read or write operation, register address, port address, and device address.
JTAG to Avalon Master Bridge	This IP core provides a connection between the System Console and Qsys system through the physical interfaces. The System Console initiates Avalon-MM transactions by sending encoded streams of bytes through the bridge’s physical interfaces.
Reset Controller	This module synchronizes and generates signals as per design requirements.
Avalon-ST Single-Clock FIFO	The Avalon-ST single-clock FIFO buffer receives and transmits data between the MAC and the client. The buffer is 64 bits wide and 512 bits deep. The buffer operates in store-and-forward mode by default. You can configure the buffer to enable the drop-on-error feature. When you enable the drop-on-error feature, the buffer drops the received packets when an error occurs.
Avalon-ST Adapter	This adapter converts the 32-bit Avalon-ST interface to 64 bits and vice verse.
PLL	Arria^® 10 ATX PLL takes a 156.25 MHz input clock from the on-board oscillator as the clock source for the XAUI PHY. Arria^® 10 fPLL takes a 100 MHz input clock from the on-board oscillator and acts as the clock source for the other components in this design.

Design Registers

You can use the registers to edit the design.

Table 2. System Register Map. This table lists all the base addresses for components on the subsystems.
Component	Base Address
Low latency 10G MAC	0x00000000
XAUI PHY	0x00008000
Generator	0x0000C000
Monitor	0x0000C400
Ethernet MDIO	0x0000B000
Avalon-ST single-clock FIFO (RX)	0x00009400
Avalon-ST single-clock FIFO (TX)	0x00009600

Table 3. Generator Register Map. This table lists the generator registers for the traffic controller.
Byte Offset	Name	Width	R/W	Reset Value	Description
0x00	`NUMPKTS`	32	RW	0x0	Number of packet registers. The total number of packets that the traffic generator generates and transmits to the 10GBASE-X Ethernet subsystem components.
0x04	`RANDOMLENGTH`	1	RW	0x0	Enables random length packets up to the maximum size defined by the `PKTLENGTH` register.
0x08	`RANDOMPAYLOAD`	1	RW	0x0	Enables random payload contents.
0x0C	`START`	1	R/W	0x0	Write to this register to start the generation of the Ethernet traffic.
0x10	`STOP`	1	R/W	0x0	Stops the generation of the Ethernet traffic.
0x14	`MACSA0`	32	RW	0x0	Lower 32 bits of the Ethernet frame source address.
0x18	`MACSA1`	16	RW	0x0	Upper 16 bits of the Ethernet frame source address.
0x1C	`MACDA0`	32	RW	0x0	Lower 32 bits of the Ethernet frame destination address.
0x2P	`MACDA1`	16	RW	0x0	Upper 16 bits of the Ethernet frame destination address.
0x24	`TXPKTCNT`	32	RO	0x0	The number of packets that the traffic generator transmits. Read this register when the traffic generator is not active (e.g. after testing).
0x34	`PKTLENGTH`	—	R/W	0x0	The maximum length of any payload when random-sized packets are enabled. Otherwise, this register defines the packet length generated by the traffic generator.

Table 4. Monitor Register Map. This table lists the monitor registers for the traffic controller.
Byte Offset	Name	Width	R/W	Reset Value	Description
0x00	`RXPKTCNT_EXPT`	32	RW	0xffffffff	Number of packets that the traffic monitor expects.
0x04	`RXPKTCNT_GOOD`	32	RO	0x0	Number of good packets received by the traffic monitor.
0x08	`RXPKTCNT_BAD`	32	RO	0x0	Number of packets received with CRC error.
0x0C	`RXBYTECNT_LO32`	32	RO	0x0	Lower 32 bits of the counter for bytes that the traffic monitor receives.
0x10	`RXBYTECNT_HI32`	32	RO	0x0	Upper 32 bits of the counter for bytes that the traffic monitor receives.
0x14	`RXCYCLCNT_LO32`	32	RO	0x0	Lower 32 bits of the counter for cycles that the traffic monitor uses to receive the expected number of packets.
0x18	`RXCYCLCNT_HI32`	32	RO	0x0	Upper 32 bits of the counter for cycles that the traffic monitor uses to receive the expected number of packets
0x1C	`RXCTRL_STATUS`	10	RW/ RO	0x0	Monitor configuration and status register. Bit[0]: Initializes all counters when 1’b1 Bit[1]: Reserved Bit[2]: Read-only—sets when the traffic monitor has received all expected packets

Clock and Reset Signals

The design uses different clock and reset signals for different components.

Table 5. Clock and Reset Signals
Signal	Direction	Width	Description
`refclk1_p`	Input	1	100 MHz clock source used as IOPLL reference clock and Avalon-MM management clock
`csr_clk`	Input	1	Configuration clock for the Avalon-MM interface, frequency is 100 MHz.
`csr_rst_n`	Input	1	Reset Avalon-MM interface.
`tx_312_5_clk`	Input	1	312.5 MHz clock for MAC TX data path.
`rx_312_5_clk`	Input	1	312.5 MHz clock for MAC RX data path.
`tx_156_25_clk`	Input	1	156.25 MHz clock for MAC TX data path.
`rx_156_25_clk`	Input	1	156.25 MHz clock for MAC RX data path.
`tx_rst_n`	Input	1	Active-low reset for MAC TX data path.
`rx_rst_n`	Input	1	Active-low reset for MAC RX data path.
`xgmii_156_25_clk`	Output	1	156.25 MHz output clock from fPLL.
`mac_312_5_clk`	Output	1	312.5 MHz output clock from fPLL.
`pll_ref_clk`	Input	1	Reference clock for ATX PLL, fPLL, and XAUI PHY.

Hardware and Software Requirements

Intel uses the following hardware and software to test the design.

Hardware

Arria^® 10 GX FPGA development kit (revision D)
Dual XAUI to SFP+ HSMC board
FMC to HSMC adapter board
USB Download Cable
SFP+ module with loopback cable

Software

Quartus^® Prime version 16.1 (for hardware testing)
ModelSim simulator

Board Setups

Before you run the design, you need to set up the boards.

Figure 4. Setting Up the Arria^® 10 FPGA Development Board

The development board has a stop button for system console testing operations, and reset buttons for the Ethernet subsystem and the Dual XAUI to SFP+ HSMC board.

Figure 5. Setting Up the Dual XAUI to SFP+ HSMC Board

You need the FMC to HSMC adapter board to connect the Arria^® 10 FPGA development board to the Dual XAUI to SFP+ HSMC board.

Install jumpers at J13 and J14 as shown in the figure above.
You must plug the adapter board and HSMC board into the FMCB of the Arria^® 10 FPGA development board and install an SFP+ module with a loopback cable in the upper SFP+ slot (CH2).
The HSMC board does not require a separate power supply because it draws power from the Arria^® 10 FPGA development board.

Testing the Design

To make sure the design runs well, you must test the design on hardware.

The perform the hardware test, follow these steps:

Download and restore the design.
Launch the Quartus^® Prime software and open the project file (top.qpf).
Click Processing > Start Compilation to compile the design.
Configure the FPGA using the generated configuration file (top.sof).
When configuration completes, open the Clock Control application (arria10GX_10ax115sf45_fpga_v15.1.2\examples\board_test_system\ClockController.exe) and change the frequency for U14 CLK2 to 156.25 MHz.
Reset the Ethernet system and HSMC board using the push button.
You must reset the system whenever you begin a new test.
On the Quartus^® Prime software, click Tools > System Debugging Tools and launch the System Console.
In the System Console command shell, change the directory to "system_console" directory.
Run the following command to initialize the design:
source demo.tcl
Run the required tests using the provided test commands listed in Test Commands.

Test Commands

This reference design provides various Tcl commands to test the Arria^® 10 FPGA development board and the Dual XAUI to SFP+ HSMC board in various loopback modes.

Table 6. TCL Commands. Format for the TCL command test is `TEST <LPBK_POINT> <BURST_SIZE> <NUM_BURSTS>`.
Command	Mode/Values	Description
`LPBK_POINT`	SFPP	Loopback at SFP+ cable
	BCMPMA	Loopback at BCM8727 PMA
	BCMXGXS	Loopback at BCM8727 XGXS
	ALTPMA	Loopback at Intel serial PMA
`BURST_SIZE`	Any integer	Number of packets in the burst.
`NUM_BURSTS`	Any number greater than 0	Specifies the intended number of bursts.

Each test generates a log file in text file format. View the log to ensure that the traffic monitor does not receive bad packets. The log also provides packet classification and statistics by the MAC TX and RX.

Note: Make sure to reset using the push buttons after each test completes.

Table 7. Test Commands
Test	Command
SFP+ Loopback Test (External loopback at the SFP+ cable)	`TEST SFPP 10000 1`
BCM8727 PMA Loopback Test	`TEST BCMPMA 10000 1`
BCM8727 XGXS Loopback Test	`TEST BCMXGXS 10000 1`
PMA Serial Loopback Test	`TEST ALTPMA 10000 1`

Simulating the Design

The design provides a testbench for you to verify the design in loopback mode.

The simulation script uses QUARTUS_ROOTDIR environment variable to access Intel simulation libraries. You must set the QUARTUS_ROOTDIR to point to the Quartus^® Prime installation path after installation.

Note: If you can't find this environment variable, set the variable manually.

Figure 6. Testbench Block Diagram

To run the simulation, follow these steps:

Download and restore the design.
Navigate to the a10_llmac_xaui_project\testbench directory.
In the TCL Console window, type the following command:
do tb_run.tcl
When the simulation completes, the Transcript window displays the statistics of the transmitted packets and received packets generated by the simulator.
Figure 7. TX Statistics

Figure 8. RX Statistics

Reference Design Debug Features

This design includes a Signal Tap II (STP) file (top.stp) to help you self-debug if you encounter any design issues on the hardware.

The STP file contains 2 instances:

Status – monitors the design channel's ready, reset, and PHY statuses.
XGMII – monitors the packet condition at XGMII and Avalon-ST interfaces.

Figure 9. Status Instance. The figure shows that the design channel is ready with resets deasserted.

Figure 10. XGMII Instance. The figure shows the XGMII (10G) and Avalon-ST (client) interface data path signals to monitor and debug packet conditions during transmission and reception.

Revision History

Date	Version	Changes
February 2017	2017.02.01	Initial release.