[Dnsmasq-discuss] DHCPv6 doesn't work on Linux interfaces enslaved to a VRF

Sun Oct 8 22:05:53 UTC 2023

On 07/10/2023 14:02, Luci Stanescu via Dnsmasq-discuss wrote:
> Hi,
> 
> I've discovered that DHCPv6 doesn't work on Linux interfaces enslaved to 
> a VRF. Now, I believe this to be a bug in the kernel and I've reported 
> it, but in case you'd like to implement a workaround in dnsmasq, this is 
> quite trivial, as I'll explain in a bit.
> 
> The issue is that when a datagram is received from an interface enslaved 
> to a VRF device, the sin6_scope_id of the msg_name field returned from 
> recvmsg() points to the interface index of the VRF device, instead of 
> the enslaved device. Unfortunately, this is completely useless when the 
> source address is a link-local address, as a subsequent sendmsg() which 
> specifies that scope will fail with ENETUNREACH, as expected, 
> considering the interface index of the enslaved device would have to be 
> specified as the scope (there can of course be multiple interfaces 
> enslaved to a single VRF device).
> 
> With DHCPv6, a DHCPSOLICIT is received from a link-local address and 
> DHCPADVERTISE is sent to the source of that address, with a scope 
> specified according to the scope from the msg_name field returned by 
> recvmsg(). I've debugged this using strace, as dnsmasq doesn't print any 
> errors when the send fails. Here is the recvmsg() call:
> 
> recvmsg(6, {msg_name={sa_family=AF_INET6, sin6_port=htons(546), 
> sin6_flowinfo=htonl(0), inet_pton(AF_INET6, "fe80::216:3eff:fed0:4e7d", 
> &sin6_addr), sin6_scope_id=if_nametoindex("myvrf")}, msg_namelen=28, 
> msg_iov=[{iov_base="\1\203\273\n\0\1\0\16\0\1\0\1,\262\320k\0\26>\320N}\0\6\0\10\0\27\0\30\0'"..., iov_len=548}], msg_iovlen=1, msg_control=[{cmsg_len=36, cmsg_level=SOL_IPV6, cmsg_type=0x32}], msg_controllen=40, msg_flags=0}, MSG_PEEK|MSG_TRUNC) = 56
> 
> and the sending of the response later on:
> 
> sendto(6, 
> "\2\203\273\n\0\1\0\16\0\1\0\1,\262\320k\0\26>\320N}\0\2\0\16\0\1\0\1,\262"..., 114, 0, {sa_family=AF_INET6, sin6_port=htons(546), sin6_flowinfo=htonl(0), inet_pton(AF_INET6, "fe80::216:3eff:fed0:4e7d", &sin6_addr), sin6_scope_id=if_nametoindex("myvrf")}, 28) = -1 ENETUNREACH (Network is unreachable)
> 
> Please notice that the scope is the index of the VRF master device, so 
> the sendto() call is certain to fail.
> 
> When reporting the issue as a kernel bug, I reproduced the issue using 
> local communication with unicast and a couple of simple Python scripts. 
> Here's reproduction using local communication, but with multicast, to 
> make it closer to home:
> 
> First, set up a VRF device and a veth pair, with one end enslaved to the 
> VRF master (on which we'll be receiving datagrams) and the other end 
> used to send datagrams.
> 
> ip link add myvrf type vrf table 42
> ip link set myvrf up
> ip link add veth1 type veth peer name veth2
> ip link set veth1 master myvrf up
> ip link set veth2 up
> 
> # ip link sh dev myvrf
> 110: myvrf: <NOARP,MASTER,UP,LOWER_UP> mtu 65575 qdisc noqueue state UP 
> mode DEFAULT group default qlen 1000
>      link/ether da:ca:c9:2b:6e:02 brd ff:ff:ff:ff:ff:ff
> # ip addr sh dev veth1
> 112: veth1 at veth2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc 
> noqueue master myvrf state UP group default qlen 1000
>      link/ether 32:63:cf:f5:08:35 brd ff:ff:ff:ff:ff:ff
>      inet6 fe80::3063:cfff:fef5:835/64 scope link
>         valid_lft forever preferred_lft forever
> # ip addr sh dev veth2
> 111: veth2 at veth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc 
> noqueue state UP group default qlen 1000
>      link/ether 1a:8f:5a:85:3c:c0 brd ff:ff:ff:ff:ff:ff
>      inet6 fe80::188f:5aff:fe85:3cc0/64 scope link
>         valid_lft forever preferred_lft forever
> 
> The receiver:
> import socket
> import struct
> 
> s = socket.socket(socket.AF_INET6, socket.SOCK_DGRAM, socket.IPPROTO_UDP)
> s.setsockopt(socket.IPPROTO_IPV6, socket.IPV6_RECVPKTINFO, 1)
> s.setsockopt(socket.SOL_SOCKET, socket.SO_BINDTODEVICE, b'veth1')
> s.bind(('', 2000, 0, 0))
> mreq = struct.pack('@16sI', socket.inet_pton(socket.AF_INET6, 
> 'ff02::1:2'), socket.if_nametoindex('veth1'))
> s.setsockopt(socket.IPPROTO_IPV6, socket.IPV6_JOIN_GROUP, mreq)
> 
> while True:
>      data, cmsg_list, flags, source = s.recvmsg(4096, 4096)
>      for level, type, cmsg_data in cmsg_list:
>          if level == socket.IPPROTO_IPV6 and type == socket.IPV6_PKTINFO:
>              dest_address, dest_scope = struct.unpack('@16sI', cmsg_data)
>              dest_address = socket.inet_ntop(socket.AF_INET6, dest_address)
>              dest_scope = socket.if_indextoname(dest_scope)
>              print("PKTINFO destination {} {}".format(dest_address, 
> dest_scope))
>      source_address, source_port, source_flow, source_scope = source
>      source_scope = socket.if_indextoname(source_scope)
>      print("name source {} {}".format(source_address, source_scope))
> 
> And the sender:
> import socket
> 
> s = socket.socket(socket.AF_INET6, socket.SOCK_DGRAM, socket.IPPROTO_UDP)
> dest = ('ff02::1:2', 2000, 0, socket.if_nametoindex('veth2'))
> s.sendto(b'foo', dest)
> 
> The receiver will print:
> PKTINFO destination ff02::1:2 veth1
> name source fe80::188f:5aff:fe85:3cc0 myvrf
> 
> Please notice that the receiver gets the right address, the one 
> associated to veth2, but the scope identifies the VRF master. However, 
> I've noticed that the scope in PKTINFO actually identifies the index of 
> the actual interface on which the datagram was received, the VRF slave 
> veth1.
> 
> As I mentioned, I believe this is a bug in the kernel and I've opened a 
> bug report for that. But, considering that dnsmasq seems to already 
> request IPV6_PKTINFO (0x32) in recvmsg() as shown in the strace above 
> (msg_control=[{cmsg_len=36, cmsg_level=SOL_IPV6, cmsg_type=0x32}]), I 
> believe a workaround of using the scope from there would work just fine 
> and would be trivial to implement.
> 
> -- 
> Luci Stanescu
> 

Thanks for a very clear and thorough explanation.

Three things come to mind.

1) Even if this is a kernel bug, kernel bugs fixes take a long time to 
spread, so working around them in dnsmasq is a good thing to do, as long 
as it doesn't leave us with long-term technical debt. This wouldn't be 
the first time a kernel bug has been worked around.

2) https://docs.kernel.org/networking/vrf.html says:

Applications that are to work within a VRF need to bind their socket to 
the VRF device:
setsockopt(sd, SOL_SOCKET, SO_BINDTODEVICE, dev, strlen(dev)+1);
or to specify the output device using cmsg and IP_PKTINFO.

Which kind of implies that this might not be a kernel bug, rather we're 
just not doing what's required to work with VRF.

Setting the device to send to using IP_PKTINFO rather than relying on 
the flowinfo field in the destination address would be quite possible, 
and the above implies that it will work. This brings us on to

3) IPv4. Does DHCPv4 work with VRF devices? It would be nice to test, 
and fix any similar problems in the same patch. Interestingly, the 
DHCPv4 code already sets the outgoing device via IP_PKTINFO (there being 
no flowinfo field in an IPv4 sockaddr) so it stands a chance of just 
working.

Copying the inferface index into the flowinfo of the destination or 
setting IP_PKTINFO are both easy patches to make and try. The difficult 
bit is being sure that they won't break existing installations.

Cheers,

Simon.