[Dnsmasq-discuss] Partial denial of service with dnsmasq on resource constrained systems
tony.ambardar at gmail.com
Fri Apr 16 09:48:51 UTC 2021
On Fri, 16 Apr 2021 at 02:21, Kevin 'ldir' Darbyshire-Bryant
<ldir at darbyshire-bryant.me.uk> wrote:
> > On 14 Apr 2021, at 00:34, Simon Kelley <simon at thekelleys.org.uk> wrote:
> > Tagging onto the end of the thread just to report the results of my
> > research.
> > This started because of problems with the OOM killer in a
> > resource-constrained system that was prompting OOM death when it spawned
> > sub-processes to handle TCP connections. I proposed a trick of putting
> > the large in-memory dataset in sub-process, so that the main dnsmasq
> > process which forks to handle TCP requests stays small. Reading around,
> > that probably won't work: the OOM killer weights size of all children
> > when looking for a victim. I think that trying to outsmart the OOM
> > killer is probably a hiding to nothing. It's possible for the OS to
> > protect critical daemons using oom_adj and friends, and supporting that
> > in OpenWRT looks like a better way to go.
> > We then moved onto the fact that adblocking involves thousands of lines of
> > local=/example.com/
> > and the code supporting that didn't really envisage such large numbers.
> > I think improvements can be made there, and I'll look at doing that in
> > more detail.
> Hi Simon,
> Thanks for picking this up again. There are multiple interlinked problems:
> (ab)use of local=/example.com/ or similar for adblock lists which leads to memory usage & (apparently) slower response time due to the linear search of ’servers’ handling the local domain. I’m sure the ‘local’ list could be put into some sort of tree/hash structure to speed that up but the memory consumption will inherently be there to some (hopefully optimised) extent. The list has to exist :-)
> TCP requests causing a fork of dnsmasq with all that ‘local/server’ list memory usage, up to 21 times. I’m looking at my APU 2 running openwrt at the moment with a 46000 line (small) adblock ‘address=/foo.bar/‘ list - dnsmasq consuming circa 12MB. 21*12MB 252MB isn’t going to cause my APU to sweat memory wise but a lesser device could very well ask ‘Where am I going to find this extra 228MB of memory from then?’.
> I don’t agree that your ‘large dataset sub-process with small tcp handling children’ won’t work. Yes linux oom killer takes into account all the children, but the tcp children are going to be much(much!) smaller, presumably the size of a basic dnsmasq instance of which most of it will be program text. A base dnsmasq on said APU2 takes 2.5MB (the above example would then total 12MB + 20*2.5=62MB) That’s quite a difference on a constrained system between asking for lumps of 12MB vs 2.5MB. Whether this approach makes sense from a latency perspective I don’t know… I’m assuming that large process is acting as an ‘address validator’ and hence the requesting children will need to wait for its ‘yes/no’ answer subject to process scheduling etc.
> Tony mentioned another possible solution which avoids the sub-process malarky of marking the ’server list’ memory in some way that the OS knows it’s shared and therefore don’t need to have ‘real memory’ for all those tcp sub-processes. I’m sure he’ll be along in a minute to explain further.
Hi Simon, Kevin,
I wrote a test program that emulates dnsmasq execution when loading
large blocklists and then forking children to handle TCP DNS requests.
It allocates memory and initializes it, then sets it read-only, and
finally forks a number of child processes. At each step, it prints
committed memory from /proc/meminfo, tracking the risk of OOM. The
option "--private" uses private anonymous memory and reproduces normal
dnsmasq behaviour; it can easily trigger OOM on small memory systems
(e.g. 128MB). The other option "--shared" demonstrates how using
shared anonymous memory reduces the overall commit and can work on
even small systems.
I also captured some log files from running on a "big" 8GB system and
a "small" 128MB OpenWrt system. Using shared memory on the "small"
system, I can allocate 64 MB without issue, while doing the same using
private memory is a reliable way to trigger OOM.
For more details see the head commit and related files here:
More information about the Dnsmasq-discuss