[Dnsmasq-discuss] Slow DHCP Performance with 7500 Static Reservations

[Mike Ruiz]

Hello All,

We use dnsmasq as a very effective replacement for the ISC software, thus far with great success. However, we have run into occasional performance problems at scale.

These seem to manifest as a general slowdown of the request -> reply process, which can sometimes exceed 90 seconds from request to reply.

The problems only cause issues when there is a penalty for missing a request, e.g. when we boot an entire rack of servers. In the default configuration, many servers will not attempt to retry the boot sequence, forcing us to detect hung machines and issue a remote reboot. We often have to do this several times in the frequent case of building more than a few hundred servers in a single batch.

Here are the symptoms:

1. We never have any issue with even mass installation as long as the configuration contains less than a few thousand static entries and a few hundred subnets.

2. At some point, dhcp will slow down: this sequence was taken with 7383 static host entries defined:

---------------------------------------------------------------------------
  TIME: 19:43:47.480189
    IP: > (00:01:e8:92:fd:41) >  (00:21:9b:a2:e4:47)
    OP: 1 (BOOTPREQUEST)
 HTYPE: 1 (Ethernet)
  HLEN: 6
  HOPS: 1
   XID: 2472c541

<SNIP>

---------------------------------------------------------------------------
  TIME: 19:45:35.963209
    IP: > (00:21:9b:a2:e4:47) >  (00:00:5e:00:01:8d)
    OP: 2 (BOOTPREPLY)
 HTYPE: 1 (Ethernet)
  HLEN: 6
  HOPS: 1
   XID: 2472c541

<SNIP>

3. If I reduce the number of static entries to, for example, 2408, response time returns to sub-second.

General configuration notes:

This request is handled by a relay, and we have the following config options in play:

no-ping
no-hosts
no-resolv
cache-size=0
dhcp-lease-max=20000
dhcp-authoritative
conf-dir=/etc/dnsmasq.d
domain=sekret.zynga.com
port=0

Fast:

# ls -la /etc/dnsmasq.conf
-rw-r--r-- 1 root root 408222 Jun 14 20:20 /etc/dnsmasq.conf
# grep dhcp-range /etc/dnsmasq.conf | wc
     80      80    6340
# grep dhcp-host /etc/dnsmasq.conf | wc
   2408    2410  209798

Slow:

# ls -la /etc/dnsmasq.conf
-rw-r--r-- 1 root root 1205132 Jun 14 20:31 /etc/dnsmasq.conf
# grep dhcp-range /etc/dnsmasq.conf | wc
     80      80    6340
# grep dhcp-host /etc/dnsmasq.conf | wc
   7383    7387  650118

Are there any obvious inflection points that would cause the server to drop in performance by a few orders of magnitude with lots of hosts defined? Are there any recommendations for tuning, beyond introducing more dnsmasq servers?

-Mike
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.thekelleys.org.uk/pipermail/dnsmasq-discuss/attachments/20110614/c9c64894/attachment.htm