[Dnsmasq-discuss] nftables netlink cache initialization failure with dnsmasq

[Monib]

Hello,

An OpenWRT user here who has been trying to set up split tunneling
using https://docs.openwrt.melmac.net/pbr/, which uses dnsmasq and
nftables, but I am having some issues.

I am encountering an error: "netlink: Error: cache initialization
failed: Protocol error"

The issue starts happening semi-randomly but seems to occur when too
many DNS requests are made in a short period. Once it appears, the
relevant nftables sets stop being populated by dnsmasq.

I reported this on the nftables mailing list:
https://lore.kernel.org/netfilter-devel/aBpv9rBirbFkpWvB@calendula/T/#t

They pointed out this:

> EPROTO can be reported by libmnl with netlink sequence problems.
>
> Quickly browsing dnsmasq code, it looks like there is a pool of child processes that are sharing a single nft_ctx handle to handle events, two or more child processes are racing.
>
> I can expand libnftables(3) manpage to clarify this.

To be frank, I don’t understand what most of this means, but it seems
like this needs to be addressed from dnsmasq side?

Here is a strace of dnsmasq capturing the error:
https://github.com/user-attachments/files/19410818/strace.log.tar.gz

Here's what I see in openwrt logs:

Sun Mar 23 17:52:24 2025 daemon.err dnsmasq[4]: nftset inet fw4
pbr_wg_xray_4_dst_ip_cfg066ff5 netlink: Error: cache initialization
failed: Protocol error
Sun Mar 23 17:52:33 2025 daemon.err dnsmasq[4]: nftset inet fw4
pbr_wg_xray_4_dst_ip_cfg046ff5 netlink: Error: cache initialization
failed: Protocol error
Sun Mar 23 17:52:58 2025 daemon.err dnsmasq[4]: nftset inet fw4
pbr_wg_xray_4_dst_ip_cfg066ff5 netlink: Error: cache initialization
failed: Protocol error
Sun Mar 23 17:54:08 2025 daemon.err dnsmasq[4]: nftset inet fw4
pbr_wg_xray_4_dst_ip_cfg066ff5 netlink: Error: cache initialization
failed: Protocol error
Sun Mar 23 17:54:22 2025 daemon.err dnsmasq[4]: nftset inet fw4
pbr_wg_xray_4_dst_ip_cfg076ff5 netlink: Error: cache initialization
failed: Protocol error
Sun Mar 23 17:54:22 2025 daemon.err dnsmasq[4877]: nftset inet fw4
pbr_wg_xray_4_dst_ip_cfg076ff5 netlink: Error: cache initialization
failed: Protocol error
Sun Mar 23 17:54:53 2025 daemon.err dnsmasq[4]: nftset inet fw4
pbr_wg_xray_4_dst_ip_cfg066ff5 netlink: Error: cache initialization
failed: Protocol error
... (many similar entries)

Hardware:
- Router: Linksys EA8100
- SoC: MediaTek MT7621 ver:1 eco:3

Software:
- OpenWRT Version: 24.10.0 (r28427-6df0e3d02a)
- Kernel: 6.6.73
- Dnsmasq: 2.90 (Compile options: IPv6 GNU-getopt no-DBus UBus no-i18n
no-IDN DHCP DHCPv6 no-Lua TFTP conntrack no-ipset nftset auth
cryptohash DNSSEC no-ID loop-detect inotify dumpfile)
- Nftables: v1.1.1 (Commodore Bullmoose #2)

Please let me know if you require any more information or have any questions.

Thanks,
LoV432