Experimental IPv6-only network at APRICOT 2024

APRICOT is the Internet Network Operators Summit for the Asia Pacific region. Every year, a conference wireless network is built to provide connectivity for hundreds of delegates, with a separate "IPv6-only" SSID as an alternative for people to try. This year, we decided to experiment with a new approach, by using some of the recent mechanisms designed for "IPv6-mostly" networks to build a better "IPv6-only" network.

"IPv6-mostly" is a way to gracefully sunset IPv4 on dual-stack access networks. The work in this area has been driven in part by Google's enterprise network, which has become so large that they ran out of RFC 1918 private addresses.

Technically, there are two pieces to this:

Once both of these conditions are true, the client configures itself with a CLAT - a client-side NAT46 translator, with a hidden private IPv4 address. Any IPv4 application traffic is routed through this translator, carried across the network as IPv6 until it reaches the NAT64 (PLAT), where it is converted back to IPv4. This whole mechanism is called 464XLAT.

The end result is that you can interact with IPv4 resources - even using IPv4 literals, like "ping 8.8.8.8" - when running on an IPv6-only network. In effect, your IPv6 network doubles up as a large block of private addresses behind a NAT. A big advantage of this approach is that there is no need to use DNS64 to fake up AAAA records for IPv4-only destinations.

"IPv6-mostly" is supported by modern versions of macOS (13+), iOS and Android. Any other clients will simply continue with regular dual-stack operation, but overall the usage of your DHCPv4 address pools will go down.

For APRICOT, we wanted to build a pure IPv6-only network, not dual-stack. But we also wanted to enable the CLAT in client devices that support it, to get maximum compatibility with IPv4. Here's how we went about it: the pieces were all built inside an Ubuntu 22.04 virtual machine running on a compact Intel NUC computer.

Firstly, we needed a DHCPv4 server which would respond to clients which requested option 108, granting them permission to run IPv6-only. Regular DHCP servers like ISC and KEA are quite happy to do that. However, we also did not want to respond to clients who didn't support option 108; if we did, we'd have to offer them an IPv4 address, and we'd be back to a dual-stack network.

I couldn't find an off-the-shelf DHCPv4 server which was capable of working this way, so I found a modular DHCP server in Go called coredhcp and created a new plugin to implement the desired behavior. This has now been merged into the main codebase.

Secondly, I needed to send router advertisements with the PREF64 option. The conference routers were Arista Layer 3 switches, and although they have this feature in very recent firmware, it wasn't available in the version we were using.

Therefore, I used Linux's radvd to perform the router advertisements. This feature was not available in the latest released version, only git HEAD, so I had to compile radvd from source. Since it's not possible for one router to send advertisements on behalf of another, this meant that the VM where radvd was running also had to act as the gateway for the IPv6-only network, turning the VM into a router for IPv6 traffic.

Thirdly, I needed a NAT64 translator somewhere on the network. This could potentially have been done on a conference router, but since the IPv6 traffic was having to pass through the VM anyway, I decided to implement NAT64 on the VM as well. I did this using jool, a kernel module for NAT64.

The basic reachability to the IPv4 Internet via NAT64 seemed to work well. However, we were plagued by clients being repeatedly kicked off the IPv6-only network by the Cisco wireless controller, at seemingly random intervals. Strangely, this didn't seem to affect clients on the main conference SSID on the same access points. It turns out there were several underlying issues.

Firstly, the client private CLAT address 192.0.0.2 was leaking out as the source IP address on various multicast packets (such as multicast DNS, and Chrome doing service discovery on UDP port 1900). Logs showed that the wireless controller had a feature called "IP Theft or Reuse" which would add client MAC addresses to an exclusion list if it saw the same source IP address from multiple clients. We were able to turn that option off.

Secondly, some clients were being kicked off regularly every 10 minutes. The Cisco WLC had a setting where the APs would perform dynamic channel reassignment every 10 minutes. We increased it to 24 hours.

Finally, when the conference was nearly over, we discovered another per-SSID setting "IPv4 DHCP required" which we also turned off. We believe that would have fixed the remaining problems.

There was one other major issue, which was that clients would lose the ability to reach IPv4 destinations for a few minutes at a time, without being kicked off the wireless network - IPv6 connectivity continued to work. Using tcpdump, we saw that IPv6 neighbor discovery on the VM was forgetting about the CLAT's IPv6 address. It turns out that the version of Jool in the Ubuntu 22.04 package repositories is old (v4.1.7) and this is a known problem. It was straightforward to upgrade this to the latest release, v4.1.11.

Apart from these problems, the CLAT/NAT64 mechanism worked very well for those devices that supported it. Remaining issues were minor: traceroute to an IPv4 destination showed only "*" for every hop, and the macOS ssh client didn't work when given the "-4" flag (although it did work with an IPv4 literal address). Otherwise, it was just like being on a dual-stack network.

From DHCP logs, I found that 142 unique devices had attempted to use the IPv6-only SSID, and of those, 115 (81%) supported DHCP option 108. That's a surprisingly high proportion, representing a high usage of Apple laptops, iOS phones, and Android phones amongst delegates. Those should have gotten a good experience from the network, if it weren't for the wireless disconnection issues.

The other 27 devices would have had a much worse experience. They got no DHCPv4 response, so they retried repeatedly, configured themselves with an IPv4 link-local address (169.254.x.x), and would only have been able to reach Internet sites with IPv6 addresses.

We could have improved compatibility for these clients somewhat by providing a DNS64 service, which fakes up AAAA records for DNS names which have only A records. However, these DNS settings would have applied to all hosts on the network, meaning that even those clients supporting option 108 would also have been exposed to fake DNS responses. I felt that the experiment was more useful without it, as the NAT64/DNS64 combination is already well known and tested.

IPv6-only using the IPv6-mostly mechanisms works surprisingly well, and is only going to improve over time as Windows and Linux add support for it.

Personally, I'd be quite happy to run this way at home, except that my Mikrotik router does not have any NAT64 capability. (RouterOS versions 7.8 and later do have the PREF64 router advertisement option though)

For more information, check out: