HECN Team Demonstrates High-Speed Networking at SC21

At the SuperComputing 2021 (SC21) event, the SESDA HECN team demonstrated the use of NVMe over Fabric over TCP (NVMe-OF/TCP) technology across a 400-GigE wide-area network (WAN) infrastructure using 200-GigE Mellanox NICs, NVMe SSD drives, and PCIe Gen4 x16 slots in the demo servers.
The new NVMe-OF/TCP technology is now fully supported by recent Linux kernels which allowed showcasing very high performance disk-to-memory network data transfers between a pair of high performance 4×200-GigE SuperMicro AMD EPYC NVMe servers in the MAX Suite that was routed across the 400-GigE ESnet/Internet2 WAN path to the StarLight Booth at SC21 and then looped to the MAX Suite. An aggregate throughput of 24.35 GB/s (194.8 Gigabits per second (Gbps)) was achieved with 64 parallel reads across 16 NVMe drives, with only a moderate level of system CPU utilization on the server. When looping the traffic at StarLight instead of SC21, which reduced the RTT from 19.7 ms to 14.7 ms, aggregate throughput of 25.11 GB/s (200.9 Gbps) was achieved. On purely local transfers between the two servers, an aggregate throughput of 44.53 GB/s (356.2 Gbps) was reached.

Comments are closed.