SuperComputing 2011
Moving Towards Terabit/Sec Transfers

Designing a 40GE FAST Data Transfer Kit

Transfering data beyond 10GE requires careful server design and selection of various components. This involves selection of server chassis, motherboard, I/O controllers, network cards and network cables. Servers supporting Intel Westmere or earlier processors involves a bridge chipset where most of the addone cards are terminated, some motherboards have dual chipsets for increased bandwidth.

This is different in the case of SandyBridge or IVY based systems where PCIe Lanes run straight from processors and chipset is mostly not required thus providing the lowest possible latency. This section lists experiences using the latest PCIe Generation 3 based server platforms and offers details to design a complete system.

Mainly there are four sections:

1. Hardware Selection 2. Test Setup 3. Optimization 4. Results

Hardware Selection:

This section lists the 40GE capable hardware available to our group.


Vendors like Dell and SuperMicro are now offering SandyBridge E5 based servers with multiple PCIe Gen3 x8 or x16 slots.

SuperMicro: We have built custom SuperMicro Servers using X9DRi/F motherboards for best performance, see more details in test setup.

Dell: R720xd is the latest offering with PCIe Gen3 slots. However the front disks connects to PERC RAID controlled via the expander backplane which reduces the overall I/O bandwidth.

Network Cards

Currently Mellanox is the only vendor offering PCIe Gen2 and Gen3 40GE NICs. There are two variants available, ConnectX-3 EN and ConnectX-3 VPI. Later can be used either as 40GE Ethernet or IB FDR mode operating at 56 Gbps. Make sure to use the latest firmware and drivers.

Network Switches

Dell-Force10: Z9000, S4810

Mellanox: SX1036 (Ethernet Only), SX6036 (Ethernet + IB)

Disk Controller

LSI: MegaRaid 9265-8i (6Gbps per port) with Fast Path enabled for SSD drives.


SATA-3 (6Gbps)

OCZ: Enterprise rated Deneva 2, or for testing Vertex 3

Other PCIe based SSDs available in the market (not yet tested):

  • Fusion IO: ioDrive2
  • OCZ: CM84
  • Intel:Series 910