Adapter installation suggestions
- Install the Network adapter in a slot that matches or exceeds the bus width of the adapter.
- Example 1: if you have a 32-bit PCI adapter put it in a 32-bit or 64-bit PCI or PCI-X* slot.
- Example 2: if you have a 64-bit PCI-X adapter put it in a 64-bit PCI-X slot.
- Example 3: if you have an x4 PCIe* adapter put it in an x4, x8, or x16 PCIe* slot.
For PCI and PCI-X*, install the Network Adapter in the fastest available slot.
||Some PCIe* slots are physically wired with fewer channels than the dimensions of the slot would indicate. In that case, a slot that matches an x8 dimensions would have the functionality of an x4, x2 or x1 slot. Check with your system manufacture.|
- Example 1: if you have a 64-bit PCI adapter put it in a 66 MHz 64-bit PCI slot.
- Example 2: if you have a 64-bit PCI-X adapter put in a 133 MHz (266 or 533 if available) 64-bit PCI-X slot.
Try to install the adapter in a slot on a bus by itself. If add-in cards share a bus, they compete for bus bandwidth.
||The slowest board on a bus dictates the maximum speed of the bus. i.e. When a 66MHz and a 133 MHz add-in card are installed in a 133 MHz bus, then all devices on that bus function at 66 MHz.|
Driver configuration suggestions
Reduce Interrupt Moderation Rate to Low, Minimal, or Off
- Also known as Interrupt Throttle Rate (ITR)
- The default is "Adaptive" for most roles.
- The low latency profile sets the rate to off.
- The storage profiles set the rate to medium.
Enable Jumbo Frames to the largest size supported across the network (4KB, 9KB, or 16KB)
||Decreasing Interrupt Moderation Rate increases CPU utilization.|
Disable Flow Control.
||Enable Jumbo Frames only if devices across the network support them and are configured to use the same frame size.|
- The default is Generate & Respond.
Increase the Transmit Descriptors buffer size.
||Disabling Flow Control can result in dropped frames.|
- The default is 256. Maximum value is 2048.
Increase the Receive Descriptors buffer size.
||Increasing Transmit Descriptors increases system memory usage.|
- The default is 256. Maximum value is 2048.
||Increasing Receive Descriptors increases system memory usage.|
TCP configuration suggestions
- Tune the TCP window size (Applies to Windows* Server editions before Windows Server 2008*)
Optimizing your TCP window size can be complex as every network is different. Documents are available on the Internet that explain the considerations and formulas used to set window size.
Before Windows Server 2008, the network stack used a fixed-size receive-side window. Starting with Windows Server 2008, Windows provides TCP receive window auto-tuning. The registry keywords TcpWindowSize, NumTcbTablePartitions, and MaxHashTableSize, are ignored starting with Windows Server 2008.
- Resource: General Introduction
- Resource: Detailed Information
- See Related topics at the end of this page
Teaming considerations and suggestions
When teaming multiple adapter ports together to maximize bandwidth, the switch needs to be considered. Typically, dynamic or static 802.3ad link aggregation is the preferred teaming mode. But, this teaming mode demands multiple contiguous ports on the switch. Give consideration to port groups on the switch. Typically, a switch has multiple ports grouped together, that are serviced by one PHY. This one PHY can have a limited shared bandwidth for all the ports it supports. This limited bandwidth for a group might not be enough to support full utilization of all ports in the group.
Performance gain can be limited to the bandwidth shared, when the switch shares bandwidth across contiguous ports. Example: Teaming 4 ports on Intel® Gigabit Network Adapters or LAN on motherboards together in an 802.3ad static or dynamic teaming mode. Using this example, 4 gigabit ports share a total PHY bandwidth of 2 Gbps. The ability to group switch ports is dependent on the switch manufacturer and model, and can vary from switch to switch.
Alternative teaming modes can sometimes mitigate these performance limitations, for instance using Adaptive Load Balancing (ALB), including Receive Load Balancing. ALB has no demands on the switch and does not need to be connected to contiguous switch ports. If the link partner has port groups, an ALB team can be connected to any port of the switch. Connecting the ALB team this way, distributes the connections across the available port groups on the switch. This action can increase the overall network bandwidth.
Performance testing considerations
- When copying a file from one system to another (1:1) using one TCP session, throughput is significantly lower than doing multiple simultaneous TCP sessions. Low throughput performance on 1:1 networks is because of latency inherent in a single TCP/IP session. A few file transfer applications support multiple simultaneous TCP streams. Some examples are: bbFTP*, gFTP*, and FDT*.
This graph is intended to show (not guarantee) the performance benefit of using multiple TCP streams. These are actual results from an Intel® 10 Gigabit CX4 Dual Port Server Adapter, using default Advanced settings, under Windows 2008* x64.
- Direct testing of your network interface throughput capabilities can be done by using tools like: iperf*, and Microsoft NTTTCP*. These tools can be configured to use one or more streams.
- When copying a file from one system to another, the hard drives of each system can be a significant bottle neck. Consider using high RPM, higher throughput hard drives, striped RAIDs, or RAM drives in the systems under test.
- Systems under test should connect through a full-line rate, non-blocking switch.
- Theoretical Maximum Bus Throughput:
- PCI Express* (PCIe*) Theoretical Bi-Directional Bus Throughput
|PCI Express Implementation
||Encoded Data Rate
||Unencoded Data Rate|
||4 Gb/sec (0.5 GB/sec)|
||16 Gb/sec (2 GB/sec)|
||32 Gb/sec (4 GB/sec)|
||64 Gb/sec (8 GB/sec)|
- PCI and PCI-X Bus Theoretical Bi-Directional Bus Throughput
|Bus and Frequency
||32-Bit Transfer Rate
||64-Bit Transfer Rate|
||The PCIe* link width can be checked in Windows* through adapter properties. Select the Link Speed tab and click the Identify Adapter button. Intel® PROSet for Windows* Device Manager must be loaded for this utility to function.|
User Guide provides release notes, configuration details, and known issues
Advanced driver settings for Intel® Ethernet 10 Gigabit Server Adapters
Performance Tuning* for 10 Gigabit Ethernet adapters in Linux*
Maximizing File Transfer Performance Using 10Gb Ethernet and Virtualization (Linux* and VMware VSphere*)
Performance Tuning Guidelines* for Windows Server 2012*
Performance Tuning Guidelines for Windows Server 2008 R2*
Performance Tuning Guidelines for Windows Server 2008*
TCP Features* of Windows 2000 and Windows Server 2003*
How to Improve Windows 2003* File Server Performance*
Improving Measured Latency in Linux* for Intel® 82575/82576 or 82598/82599 Ethernet Controllers
Assigning Interrupts to Processor Cores for Intel® 82575/82576 or 82598/82599 Ethernet Controllers
Parameter Talk: TX and RX descriptors explains the effects of changing buffer sizes
Low virtual network I/O performance with Intel 1Gbps NIC using igb driver explains tuning ESXi performance for certain workloads
This applies to: