[Dnsmasq-discuss] Dnsmasq with Gigantic hosts file

[Jan 'RedBully' Seiffert]

Simon Kelley wrote:
> Jan 'RedBully' Seiffert wrote:
[snip 700k hosts file]
>> If you are talking about a traditional hosts file:
>> AFAIK dnsmasq uses a hash table for such lookups, so basically even a
>> large number should not be a problem. I for example have 10.000
>> "bad-hosts" in an extra file read by dnsmasq (50k where also already
>> mentioned here on the list), and my DNS lookups are "lightning fast" (I
>> also run dnsmasq with nice -1 + some traffic shaping rules prioritizing
>> DNS packets, but thats to reduce latencies).
>> But 700k may drive it to it's limits...
>> First you will need a little RAM for it (my dnsmasq uses 2MB ATM, so
>> _very_ roughly * 70 ~= 140MB), but secondly scalability-issues in the
>> hash table may surface (hash collision, but you may fix it locally by
>> setting a higher number of hash buckets in the source code and recompile).
>>
>> But the ultimate answer to this is only known by Simon.
> 
> Releases before 2.35 will choke reading a file this big. The code was
> re-written in 2.35 to make it usable. (and also to re-size the hash
> table based on the size of /etc/hosts, which addresses Jan's point).
> 
Ahhh, it's good to have someone knowing the code in and out ;)
I looked myself into 2.36 cache.c and yes, this should solve it.

> I'd expect the next choke-point to be reverse (address->names) DNS
> lookups, which are not hashed. I don't of anybody who has hit that brick
> wall yet, but 700,000 might. You will have to test. The lookup time
> should scale linearly with the size of the hosts file for reverse
> lookups, and much less than linearly for forward lookups.
> 
While it is common that most of these entries will point to one address
(be it 127.0.0.1 or a local LAN machine) and F_REVERSE not set for them,
it's a pity it will still slow down reverse lookups...

Maybe the following patch will help:
Maintain a tree for reverse lookups.
Patch is compile tested only, i don't know if this will work, I surely
dropped the ball somewhere. Also because the cache logic 'as is' is
quite complex IMHO (but that's maybe just my brain always needing a
moment wrapping around hash tables when looking into the impl.).

> Cheers,
> 
> Simon.
> 
Greeting
	Jan

-- 
ASCII a stupid question,
get a stupid ANSI
-------------- next part --------------
A non-text attachment was scrubbed...
Name: dnsmasq236_reverse_tree.diff
Type: text/x-patch
Size: 8495 bytes
Desc: not available
Url : http://lists.thekelleys.org.uk/pipermail/dnsmasq-discuss/attachments/20070129/84acdda2/dnsmasq236_reverse_tree.bin