VXLAN: one remote IP, multiple destination ports

A host running a userspace dataplane may have several workers receiving VXLAN traffic. Each worker listens on a different UDP port, but they share the host’s underlay IP and handle traffic for the same overlay segment:

Remote host: 192.0.2.10
  worker A -> UDP 4789, VNI 100
  worker B -> UDP 4790, VNI 100

The sender needs to distribute flows across those workers. The destination is not just an IP address: it is an IP-and-port pair.

Linux already supports VXLAN FDB entries backed by nexthop groups. Linux 7.3 adds an optional destination port to each FDB nexthop, with configuration support in iproute2 7.3.0.

What changes

Previously, VXLAN transmission through an FDB nexthop group used the destination port configured on the VXLAN device. Group members could select different remote IPs, but not different destination ports.

With the new NHA_DST_PORT attribute, each member can specify its own port. After selecting a nexthop, VXLAN uses both its remote IP and its destination port.

This allows a single VXLAN device and nexthop group to reach multiple workers on the same remote host.

A configuration example

Assume an existing VXLAN device named vx0, configured for VNI 100, and underlay connectivity to 192.0.2.10.

Create one FDB nexthop per receiving worker:

ip nexthop add id 1 via 192.0.2.10 fdb dst_port 4789
ip nexthop add id 2 via 192.0.2.10 fdb dst_port 4790

Put them in an equal-weight group:

ip nexthop add id 10 group 1/2 fdb

Then point a destination MAC’s FDB entry at that group:

bridge fdb add 00:11:22:33:44:55 dev vx0 self static nhid 10

Traffic to that MAC uses the group’s flow-based selection to choose a worker. The selected member determines whether the outer UDP destination port is 4789 or 4790. This is flow distribution, not packet-by-packet round robin.

The receiving workers must be prepared to handle the traffic assigned to them; this feature does not provide worker health checks or coordinate their state.

Why keep the same VNI?

The VNI and UDP destination port serve different purposes here:

  • VNI 100 identifies the overlay segment.
  • UDP port 4789 or 4790 selects the receiving worker.

There is no need to create a separate overlay segment just to address another worker serving the same segment.

Existing behavior stays the same

The port is optional. A nexthop without dst_port continues to use the VXLAN device’s configured destination port:

ip nexthop add id 3 via 192.0.2.10 fdb

The new attribute is accepted only on FDB nexthops with a gateway address, and an explicit port must be nonzero. It changes the sender’s destination-port selection; it does not configure listeners on the remote host.

For deployments that distinguish receivers by UDP port, this removes the need for separate sending VXLAN devices solely to select those ports.

Patch series and review discussion

VXLAN: one remote IP, multiple destination ports

Leave a Reply