[Dnsmasq-discuss] Malformed NSEC/NSEC3 Can Hang dnsmasq

[Daniel Anderson]

A malformed signed NSEC/NSEC3 response can make DNSSEC-enabled dnsmasq stop answering queries and spin at high CPU.

Code location: src/dnssec.c, prove_non_existence_nsec() around the NSEC bitmap loop, and the NSEC3 exact-match proof path.

The bug is in the DNSSEC bitmap parser. NSEC bitmap windows are encoded as: one-byte window number, one-byte bitmap length, then bitmap data. dnsmasq checks the full block length with p[1] + 2, but advances only by p[1]:
rdlen -= p[1]; p += p[1]; 

If p[1] == 0, neither value changes, so validation loops forever. I reproduced this with a controlled upstream DNS server returning a signed malformed NSEC proof for hang. A. After the response, dnsmasq stopped answering follow-up queries and remained running at high CPU until killed.

Suggested fix: reject zero-length bitmap windows and advance by p[1] + 2.

I am attaching the POC and the patch.  Copy files to a director on your computer and run using ```./run-live-poc.sh dnssec2```

Note: This POC showcases two bugs and I’ll be filing the next bug in a few minutes. Also, I found this bug while testing the N184 Bug and Vulnerability Scanning Suite (https://github.com/MillaFleurs/N184).

Thank you,

Dan

-------------- next part --------------
A non-text attachment was scrubbed...
Name: dnssec_upstream.py
Type: text/x-python-script
Size: 10495 bytes
Desc: not available
URL: <http://lists.thekelleys.org.uk/pipermail/dnsmasq-discuss/attachments/20260511/55a62694/attachment-0001.bin>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: README.md
Type: text/markdown
Size: 1385 bytes
Desc: not available
URL: <http://lists.thekelleys.org.uk/pipermail/dnsmasq-discuss/attachments/20260511/55a62694/attachment-0001.md>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: run-live-poc.sh
Type: application/octet-stream
Size: 4747 bytes
Desc: not available
URL: <http://lists.thekelleys.org.uk/pipermail/dnsmasq-discuss/attachments/20260511/55a62694/attachment-0002.obj>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: patch.diff
Type: application/octet-stream
Size: 1503 bytes
Desc: not available
URL: <http://lists.thekelleys.org.uk/pipermail/dnsmasq-discuss/attachments/20260511/55a62694/attachment-0003.obj>