Building a 100G RoCE ring of three DGX Spark nodes without a switch Print

Three DGX Spark nodes can be connected directly to each other in a ring, with no switch in the high-speed fabric. Each pair of nodes is joined by two independent physical Ethernet links at 100 Gbit/s each, giving 200 Gbit/s of nominal bandwidth per side of the triangle and six physical links in total. This article covers the addressing scheme, the configuration steps, the pitfalls we hit, and the throughput you can realistically expect.

All node names and IP addresses below are examples from the RFC 1918 private range. Substitute your own.

1. Addressing the six direct links

Every direct link gets its own /24 subnet — this keeps routing unambiguous and makes each link individually testable.

Note the crossing pattern: an f0 port on one node faces an f1 port on the other.

Each Ethernet interface has a matching RDMA device — rocep1s0f0 ↔ enp1s0f0np0, rocep1s0f1 ↔ enp1s0f1np1, roceP2p1s0f0 ↔ enP2p1s0f0np0, roceP2p1s0f1 ↔ enP2p1s0f1np1. Verify the mapping before testing; picking the wrong RDMA device is an easy way to measure the wrong link.

The management network stays separate, on its own interface per node.

2. Software

Base: NVIDIA DGX Spark Playbooks, with playbooks for the ring topology and NCCL.

Important: if the ring is already cabled and addressed, do not run the playbook's --run-setup — it rewrites the network configuration. Adapt the setup script and supply your own cluster config instead.

Per node: OpenMPI 4.1.6, NCCL 2.29.7+cuda13.0, nccl-tests 2.20.0. First sanity check is mpirun hostname returning all three node names; NCCL should then discover four RDMA interfaces per node and select NET/IB.

3. Firewall: the TCP/18515 pitfall

Under UFW, inter-node MPI traffic is blocked by default and must be permitted explicitly. The mistake that costs the most time is writing 18515 as the source port in the rule. It is the destination port on the receiving side — with the port in the source position the rule looks correct and the connection still never establishes. Fix the direction and ib_write_bw connects immediately.

4. MTU

Raising the Ethernet MTU from 1500 to 9000 raises the RDMA active MTU from 1024 to 4096. Confirm with a DF-flagged 9000-byte ICMP packet. Note that ip link set only holds until reboot — persist the change in the interface configuration.

The measurable effect on throughput was none: 99.62 → 99.63 Gbit/s. Message size and queue-pair count matter far more than MTU here.

5. How to verify

1. ICMP across all six direct links.
2. Routes, RoCE v2 GID index 3 and the RDMA-device match on the pair under test.
3. ib_write_bw on a single link (RoCE v2, GID index 3, RC transport).
4. NCCL all_gather_perf with 3 ranks, one per node, sizes 1 MiB–16 GiB, 5 warmup and 20 measured iterations — expect wrong = 0 and out of bounds = 0 at every size.
5. Hardware counters during the NCCL run, to confirm all four ports carry traffic.

6. What to expect

Single link, ib_write_bw:

A single link reaches practically the full 100 Gbit/s.

NCCL all_gather across three nodes at 16 GiB: out-of-place algbw 31.43 GB/s / busbw 20.95 GB/s; in-place algbw 33.28 GB/s / busbw 22.19 GB/s. Mean busbw across all sizes was 15.4871 GB/s — an average, not a peak, pulled down by the small sizes; a dedicated 16 GiB run averaged 21.5495 GB/s. Hardware counters showed roughly 93–95 Gbit/s of RDMA traffic on each of the four ports, about 186 Gbit/s TX and RX in total.

7. One caveat

The operating system reports 200000 Mb/s for these ports while the confirmed physical rate is 100 Gbit/s per link. We did not investigate the source of that discrepancy. Treat the OS-reported link speed on this hardware as unreliable and measure instead.