[Dnsmasq-discuss] Dnsmasq with Gigantic hosts file

[Simon Kelley]

Jan 'RedBully' Seiffert wrote:
> Jason wrote:
>> Hello all,
>>
> Hi
> 
>> I'm considering replacing squidguard, a filtering redirector for the
>> squid cache proxy, with a BIG hosts file and dnsmasq (716,093
>> entries!).  Currently, my network serves less than 10 clients(dns only,
>> no dhcp), but may reach 200 in the next few years.  Will this work. 
>> It'd be nice to consolidate the functions and have the option to
>> shutdown the squid cache.
> 
> If you are talking about a traditional hosts file:
> AFAIK dnsmasq uses a hash table for such lookups, so basically even a
> large number should not be a problem. I for example have 10.000
> "bad-hosts" in an extra file read by dnsmasq (50k where also already
> mentioned here on the list), and my DNS lookups are "lightning fast" (I
> also run dnsmasq with nice -1 + some traffic shaping rules prioritizing
> DNS packets, but thats to reduce latencies).
> But 700k may drive it to it's limits...
> First you will need a little RAM for it (my dnsmasq uses 2MB ATM, so
> _very_ roughly * 70 ~= 140MB), but secondly scalability-issues in the
> hash table may surface (hash collision, but you may fix it locally by
> setting a higher number of hash buckets in the source code and recompile).
> 
> But the ultimate answer to this is only known by Simon.

Releases before 2.35 will choke reading a file this big. The code was
re-written in 2.35 to make it usable. (and also to re-size the hash
table based on the size of /etc/hosts, which addresses Jan's point).

I'd expect the next choke-point to be reverse (address->names) DNS
lookups, which are not hashed. I don't of anybody who has hit that brick
wall yet, but 700,000 might. You will have to test. The lookup time
should scale linearly with the size of the hosts file for reverse
lookups, and much less than linearly for forward lookups.

Cheers,

Simon.