Designing a 40GE FAST Data Transfer Kit
Transfering data beyond 10GE requires careful server design and selection of various components. This involves selection of server chassis, motherboard, I/O controllers, network cards and network cables. Servers supporting Intel Westmere or earlier processors involves a bridge chipset where most of the addone cards are terminated, some motherboards have dual chipsets for increased bandwidth.
This is different in the case of SandyBridge or IVY based systems where PCIe Lanes run straight from processors and chipset is mostly not required thus providing the lowest possible latency. This section lists experiences using the latest PCIe Generation 3 based server platforms and offers details to design a complete system.
Mainly there are four sections:
This section lists the 40GE capable hardware available to our group.
Vendors like Dell and SuperMicro are now offering SandyBridge E5 based servers with multiple PCIe Gen3 x8 or x16 slots.
Dell: R720xd is the latest offering with PCIe Gen3 slots. However the front disks connects to PERC RAID controlled via the expander backplane which reduces the overall I/O bandwidth.
Currently Mellanox is the only vendor offering PCIe Gen2 and Gen3 40GE NICs. There are two variants available, ConnectX-3 EN and ConnectX-3 VPI. Later can be used either as 40GE Ethernet or IB FDR mode operating at 56 Gbps. Make sure to use the latest firmware and drivers.
LSI: MegaRaid 9265-8i (6Gbps per port) with Fast Path enabled for SSD drives.
OCZ: Enterprise rated Deneva 2, or for testing Vertex 3
Other PCIe based SSDs available in the market (not yet tested):
- Fusion IO: ioDrive2
- OCZ: CM84
- Intel:Series 910