[Dnsmasq-discuss] reading /etc/hosts takes 6 minutes !?!?!

[Simon Kelley]

koko wrote:
> rasz at capek:/etc$ cat /etc/hosts | grep 127 | uniq | wc -l
> 52076
> rasz at capek:/etc$ sudo /etc/init.d/dnsmasq restart
> rasz at capek:/etc$ cat /var/log/syslog
> 
> Oct 27 08:19:12 capek dnsmasq[8060]: exiting on receipt of SIGTERM
> Oct 27 08:19:12 capek dnsmasq[8223]: started, version 2.33 cachesize 150
> Oct 27 08:19:12 capek dnsmasq[8223]: compile time options: IPv6
> GNU-getopt no-ISC-leasefile DBus I18N
> Oct 27 08:19:12 capek dnsmasq[8223]: reading /etc/resolv.conf
> Oct 27 08:19:12 capek dnsmasq[8223]: using nameserver 212.76.39.211#53
> Oct 27 08:19:12 capek dnsmasq[8223]: using nameserver 212.76.39.205#53
> Oct 27 08:19:12 capek dnsmasq[8223]: ignoring nameserver 127.0.0.1 -
> local interface
> Oct 27 08:24:55 capek dnsmasq[8223]: read /etc/hosts - 52076 addresses
> 
> Is this normal behaviour? this is 6 minutes of constant 100% cpu. Is
> 50k of entries just too much for it?
> 
> rasz at capek:/etc$ cat /proc/cpuinfo | grep Hz
> cpu MHz         : 2057.069
> rasz at capek:/etc$ cat /proc/meminfo | grep mTo
> MemTotal:       515868 kB
> 

All of dnsmasq is written on the principle that premature optimisation 
is the root of all evil. The /etc/hosts code is a good example: every 
address read is checked against all the others to ensure that there is 
no duplication of the reverse records. That's an instant n-squared 
algorithm, which is fine for a few hundred entries, but not (it seems) 
50000.

A question: are most of the addresses in your /etc/hosts the same, maybe 
0.0.0.0 or 127.0.0.1, for ad-blocking? It seems a good quess that most 
large /etc/hosts files would be of that form, and it's trivial to 
optimse for that case.

Cheers,

Simon.