[Dnsmasq-discuss] Dnsmasq with Gigantic hosts file

[Simon Kelley]

Jason wrote:
> Hello all,
> 
> I'm considering replacing squidguard, a filtering redirector for the 
> squid cache proxy, with a BIG hosts file and dnsmasq (716,093 
> entries!).  Currently, my network serves less than 10 clients(dns only, 
> no dhcp), but may reach 200 in the next few years.  Will this work.  
> It'd be nice to consolidate the functions and have the option to 
> shutdown the squid cache.  Also, squidguard can block a specific url 
> inside a domain, ie, www.goodsite.com/badarea/badstuff.html.  Can 
> dnsmasq emulate this?  I know this is outside the design intent of 
> dnsmasq, but it's not a bad application. 
> 

OK, I did some tests with a 700k-lines hosts file. All the names were 
synthetically created, of the form name1, name2 etc so there may be 
hash-function effects not captured by this, but I doubt they are large.

This is on my Thinkpad t40p:
srk at spike:~$ cat /proc/cpuinfo
processor       : 0
vendor_id       : GenuineIntel
cpu family      : 6
model           : 9
model name      : Intel(R) Pentium(R) M processor 1600MHz
stepping        : 5
cpu MHz         : 1594.996
cache size      : 1024 KB

So a reasonably fast beast: much faster a Linksys lunchbox.

Time to read the hosts file at startup is just over four seconds.

Forward lookups for names in the hosts file are too fast to measure, dig 
says zero or one millisecond.

Reverse lookups for in-cache addresses are about 70 milliseconds on 
dnsmasq-2.36. For addresses not in the cache, that 70 milliseconds gets 
added to the time taken by the upstream server. (Which is larger, about 
200ms minimum for my ISP and a cable-modem).

With the changes Jan and I have been discussing the time for reverse 
lookups drops from 70ms to 17ms. On a fast machine, that's worth having. 
On a Linksys MIPS box, it probably makes the difference between 
practical and not practical.

Executive summary: 700000 names in /etc/hosts needs a fast processor on 
2.35 or 2.36. When 2.37 is released (soon, to fix a bug in 2.36) then 
700000 names should be fine on anything. (But the process size is about 
40MB, so you'd better have a reasonable amount of memory.)

Cheers,

Simon.