[Dnsmasq-discuss] How does DNSMASQ handle large concurrent configure file updating request requests

Simon Kelley simon at thekelleys.org.uk
Fri Jan 16 11:24:57 GMT 2015


-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256



On 16/01/15 09:25, Yongkang You wrote:
> On Thu, Jan 15, 2015 at 9:56 PM, Simon Kelley
> <simon at thekelleys.org.uk> wrote:
> 
>> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
>> 
>> On 15/01/15 07:47, Yongkang You wrote:
>>> Hi Simon,
>>> 
>>> Thanks for replying. Batching the changes is a very good idea,
>>> but it is not easy to implement in IaaS, since the VMs
>>> creation processes are handled separately.
>>> 
>>>> From my observation, the burst 10k SIGUSER1 signals will
>>>> cause dnsmasq
>>> occupying 100% CPUs for several hours. The workload comes from
>>> 10k reload events, although the 10k DHCP config entries have
>>> been successfully updated in first tens of minutes.
>>> 
>> 
>> Wow, that's a lot of CPU. How are you synchronising changes to
>> the config files which you are re-reading?  That last thing you
>> want is race conditions where changes are missed, or half-written
>> files read.
>> 
>> Is there one file per VM (ie ~10000 files, just some of which
>> change)? we might have to get more smart about only readng stuff
>> which changes, especially as I think there's processing going on
>> at the moment which is O(n^2) in the number of configs.
>> 
>> 
>> Cheers,
>> 
>> Simon.
>> 
> 
> 
> Hi Simon,
> 
> 10k VMs share same config file. The operation logic is like: 1.
> update the config file, by adding some specific mac/ip lines. (This
> step is mutual between different VM updating) 2. send SIGUSER1 to
> dnsmasq process.
> 
> So, yes you are right. The step 2 might happen when some other VM
> is adding config. This might cause some problem.
> 
> After all entries are recorded in config file, there might be 40k
> lines. Will it become slow when trigger SIGUSER1?
> 

There are a couple of problems here.

The first is updating the config file asychronously to telling dnsmasq
to re-read it. Sooner or later, dnsmasq is going to read the file when
it's incomplete, especially if you re-write it from scratch as a
database dump, rather than appending new entries to the end.

The second problem is that it's clearly not tenable to re-read such a
large configuration every time it changes, as you've demonstrated.

There may be further problems with such a large amount of
configuration, or a large number of DHCP leases, but they can be
tackled as they arise, let's try and sort this first.

Here's one possible solution.

dhcp-hostfile can take a directory instead of a filename, so instead
of having one  file with 40k lines in it, have a directory with a file
for each host. (Aside, 40k files is a big directory, but it can be
split into multiple directories, if needed). By itself, this will go
backwards, since opening 40k small files to read them all will be even
slower, so we need an addition to dnsmasq.

The addition to dnsmasq is to use inotify to detect when a file is
added or written. The development branch of dnsmasq already has code
to do this for resolv files. With this change, there's no need to us a
signal, and no need to re-read all the files. When a file appears in
the directory or is modified _and_is_closed_ dnsmasq will wake up and
read just that file. Much better, this fixes the race condition too.

The simple way to implement this simply adds new configuration from
new files, if a file gets changed or deleted, then the dhcp-hosts that
were created by its old contents are not deleted, and SIGUSR1 is still
needed to re-read the config from scratch and garbage collect old
entries. More complex implementation would keep track which dhcp-hosts
are associated with which files, and delete old versions when a file
changes. I'd rather not have to do the more complex version, is it can
be avoided.

Comments?


Cheers,

Simon.



-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1

iQIcBAEBCAAGBQJUuPUDAAoJEBXN2mrhkTWi3bwP/1TR/k9IWKcGRU1gCsYV26gu
4SQ69z0441eseCGT+U1VFkiFpWbsQIZEYQ//JzGysC+ieev5ysIOfbHeeS6kYJ4f
mrBrD77pGe2DIjB2uny0eoB074MgRv62QHCZI8DbAtUKAV3WRpAMwfSWaQTYgFAb
xGX5NMu7Hzw8CEuJEontQktPZ2wbJ490LRmuYQ7Z5K3lmzlC3UQ7w/CG/Nptpjlx
rPNP+FkCoh6UeCzhAIZdwDCo1LCl3h2HKvKOZ2DlcW39unoPp+1xfrhe5/OPvlmf
QRS2y+AQmC4wLHF7wLaUNzw2gbCrEpXtvwbznQdQvvW0CJNqNEIGkgVKjeue5tW7
WOlY7yliptmg9G8zQtsuYzJwF0TjItdHA0mqjmWQ4ElTZ8T8tQEN9KtAod99O461
9wGkmRYHfSVOEjwrc05z3RjaqpqV70Bdd+TIvClI+COMxcNGU5JJTeXejKT4LbNz
2YtVXTi5/QGSyH99EPlA3NWEGNYx41uXjrNTD575CEllD/AkGHUDQSXEbk8XA1ZZ
MqqmkPXUUrT5bGjbppWGHHF4YbsukW9FMuISY9CYS8/LFQc0NQWDHoaN3cIPR+bg
qI7t8j/Ne9Wgv/P5lBw7hlhTG5f8fsUParwQT3mGbPiVdlf+eoWOwI7b94R4YGXy
h1cRPp4HAkwe2nBlAMF4
=/O2j
-----END PGP SIGNATURE-----



More information about the Dnsmasq-discuss mailing list