<div dir="ltr">Hello,<div><br></div><div>This is a follow-up on an issue I posted on August 9. I had the suspicion that dnsmasq was not caching for custom upstream domains, when I was routing requests to Consul. It turns out I did my testing wrong, and was not able to notice that the issue was happening only when Consul was in the upstream.</div><div><br></div><div>Turns out it the real cause was that Consul reports itself as not supporting recursive queries, and dnsmasq does not cache non-recursive responses.<br></div><div><br></div><div>Sorry for the confusion.<br></div><div><br></div><div>Here's some more info from someone who ran into the same issue:</div><div><a href="https://groups.google.com/forum/#!topic/consul-tool/NQDkxOj5Bks">https://groups.google.com/forum/#!topic/consul-tool/NQDkxOj5Bks</a></div><div><br></div><div>Thanks Petr Menšík for the initial response!</div><div><br></div><div>Paulo</div><div><br><div class="gmail_quote"><div dir="ltr">On Wed, Aug 9, 2017 at 2:34 PM Paulo Bittencourt <<a href="mailto:paulo@wattpad.com">paulo@wattpad.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div class="gmail_quote"><div>Hi Peter,</div><div><br></div><div>Sorry for the late reply on this - I was on vacation.</div><div><br></div><div>I am continuing the investigation with the caching behavior of dnsmasq when querying Consul. I verified the TTL on the responses from Consul and it is '5s', the value which we've configured into the Consul agent.</div><div><br></div><div>Here is the DNS response from the Consul (the IP and port are different from the previous examples as I've set up a test environment for this, but this is still hitting a Consul agent directly):</div><div>---</div><div><div>/ # dig @<a href="http://100.64.0.10" target="_blank">100.64.0.10</a> consul.service.consul</div><div><br></div><div>; <<>> DiG 9.10.4-P8 <<>> @<a href="http://100.64.0.10" target="_blank">100.64.0.10</a> consul.service.consul</div><div>; (1 server found)</div><div>;; global options: +cmd</div><div>;; Got answer:</div><div>;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 28354</div><div>;; flags: qr aa rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 0</div><div><br></div><div>;; QUESTION SECTION:</div><div>;consul.service.consul.<span style="white-space:pre-wrap"> </span>IN<span style="white-space:pre-wrap"> </span>A</div><div><br></div><div>;; ANSWER SECTION:</div><div>consul.service.consul.<span style="white-space:pre-wrap"> </span>5<span style="white-space:pre-wrap"> </span>IN<span style="white-space:pre-wrap"> </span>A<span style="white-space:pre-wrap"> </span>100.96.24.9</div><div><br></div><div>;; Query time: 1 msec</div><div>;; SERVER: 100.64.0.10#53(100.64.0.10)</div><div>;; WHEN: Wed Aug 09 18:32:01 UTC 2017</div><div>;; MSG SIZE rcvd: 55</div><div>---<br></div></div><div><br></div><div>The behavior I was seeing is that it would not cache Consul replies but would cache other domains. The only difference I could see between the domains is that the 'consul' upstream server was configured via the '--server flag', while the other upstreams came from '/etc/resolv.conf'.</div><div><br></div><div>As a test, I set up three dnsmasq instances as follows:</div><div><br></div><div>* INSTANCE A: uses resolv.conf setting Consul as upstream for all domains</div><div> Command: dnsmasq --log-facility=- --log-queries --port=1053</div><div><br></div><div>* INSTANCE B: uses --server flag to set Consul as upstream for the '.consul' domain</div><div> Command: dnsmasq --log-facility=- --log-queries --port=2053 --no-resolv --server=/consul/<a href="http://100.64.0.10" target="_blank">100.64.0.10</a></div><div><br></div><div><div>* INSTANCE C: uses --server flag to set Consul as upstream all domains</div><div> Command: dnsmasq --log-facility=- --log-queries --port=3053 --no-resolv --server=100.64.0.10</div></div><div><br></div><div>By following the query logs and inspecting the output TTLs, I can see that instance A caches the results (for 5 seconds) while instances B and C do not cache any results. If I query repeatedly, I can see the TTLs returned by instance A gradually decrease from 5s to 1s until the entry falls out of the cache. Instances B and C always return a 5s TTL (which is what Consul always returns).</div><div><br></div><div>As far as I can tell, dnsmasq is not doing any caching when the upstream is set via the --server flag.</div><div><br></div><div>Should we consider this a bug, or maybe a known limitation that should be documented?</div><div><br></div><div>Thanks again for your help.</div><div><br></div><div>Paulo</div><div><br></div><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Date: Thu, 13 Jul 2017 15:27:37 +0200<br>
From: Petr Men??k <<a href="mailto:pemensik@redhat.com" target="_blank">pemensik@redhat.com</a>><br>
To: <a href="mailto:dnsmasq-discuss@lists.thekelleys.org.uk" target="_blank">dnsmasq-discuss@lists.thekelleys.org.uk</a><br>
Subject: Re: [Dnsmasq-discuss] Caching with custom upstream domain<br>
Message-ID: <<a href="mailto:db8d6d0d-8a3b-e553-c47c-3bec8bf64d65@redhat.com" target="_blank">db8d6d0d-8a3b-e553-c47c-3bec8bf64d65@redhat.com</a>><br>
Content-Type: text/plain; charset=utf-8<br>
<br>
Hello Paulo,<br>
<br>
I did a quick test but it seems it does cache the responses.<br>
<br>
It would help if you could show us dig query result<br>
<br>
$ dig -p 8600 @<a href="http://127.0.0.1" rel="noreferrer" target="_blank">127.0.0.1</a> consul.service.consul<br>
<br>
It depends on what TTL is used for replies from Consul server. If TTL in<br>
second column is 0, it should not be cached by dnsmasq. You would have<br>
to change consul service in that case.<br>
<br>
Or use min-cache-ttl=120 or higher value. But that TTL might have good<br>
reason, it would be better to fix the service.<br>
<br>
Dne 12.7.2017 v 16:20 Paulo Bittencourt napsal(a):<br>
> Hello!<br>
><br>
> We are using dnsmasq for local DNS caching on all our hosts.<br>
><br>
> We've set up a custom upstream domain to route DNS requests for Consul<br>
> to the local Consul agent, using this config directive:<br>
><br>
> server=/consul/<a href="http://127.0.0.1#8600" rel="noreferrer" target="_blank">127.0.0.1#8600</a> <<a href="http://127.0.0.1#8600" rel="noreferrer" target="_blank">http://127.0.0.1#8600</a>><br>
><br>
> Looking at the load the Consul agent, we were suspecting that dnsmasq<br>
> was not caching the replies from Consul. After enabling query logging<br>
> on dnsmasq, it looks like this is the case.<br>
><br>
> If I query any other domain repeatedly, (eg. <a href="http://example.com" rel="noreferrer" target="_blank">example.com</a><br>
> <<a href="http://example.com" rel="noreferrer" target="_blank">http://example.com</a>>, <a href="http://google.com" rel="noreferrer" target="_blank">google.com</a> <<a href="http://google.com" rel="noreferrer" target="_blank">http://google.com</a>>), it logs that it's<br>
> using the cache:<br>
><br>
> Jul 12 13:56:56 ip-10-184-50-103 dnsmasq[18119]: query[A] <a href="http://www.google.ca" rel="noreferrer" target="_blank">www.google.ca</a><br>
> <<a href="http://www.google.ca" rel="noreferrer" target="_blank">http://www.google.ca</a>> from 127.0.0.1<br>
> Jul 12 13:56:56 ip-10-184-50-103 dnsmasq[18119]: cached <a href="http://www.google.ca" rel="noreferrer" target="_blank">www.google.ca</a><br>
> <<a href="http://www.google.ca" rel="noreferrer" target="_blank">http://www.google.ca</a>> is 172.217.5.227<br>
> ... repeats ...<br>
><br>
> If I query a Consul domain repeatedly, it doesn't seem to use the cache:<br>
><br>
> Jul 12 13:57:55 ip-10-184-50-103 dnsmasq[18119]: query[A]<br>
> consul.service.consul from 127.0.0.1<br>
> Jul 12 13:57:55 ip-10-184-50-103 dnsmasq[18119]: forwarded<br>
> consul.service.consul to 127.0.0.1<br>
> Jul 12 13:57:55 ip-10-184-50-103 dnsmasq[18119]: reply<br>
> consul.service.consul is 10.144.40.2<br>
> Jul 12 13:57:55 ip-10-184-50-103 dnsmasq[18119]: reply<br>
> consul.service.consul is 10.51.148.85<br>
> Jul 12 13:57:55 ip-10-184-50-103 dnsmasq[18119]: reply<br>
> consul.service.consul is 10.37.226.205<br>
> ... repeats ...<br>
><br>
> I've searched for any documentation regarding caching behavior for<br>
> custom upstream domains and I haven't found anything.<br>
><br>
> Any ideas?<br>
><br>
> Thanks,<br>
> Paulo<br>
><br>
><br>
> _______________________________________________<br>
> Dnsmasq-discuss mailing list<br>
> <a href="mailto:Dnsmasq-discuss@lists.thekelleys.org.uk" target="_blank">Dnsmasq-discuss@lists.thekelleys.org.uk</a><br>
> <a href="http://lists.thekelleys.org.uk/mailman/listinfo/dnsmasq-discuss" rel="noreferrer" target="_blank">http://lists.thekelleys.org.uk/mailman/listinfo/dnsmasq-discuss</a><br>
></blockquote></div><div dir="ltr"><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><br></blockquote></div></div></div><div dir="ltr"><div dir="ltr"><div dir="ltr"><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<br>
--<br>
--<br>
Petr Men??k<br>
Software Engineer<br>
Red Hat, <a href="http://www.redhat.com/" rel="noreferrer" target="_blank">http://www.redhat.com/</a><br>
email: <a href="mailto:pemensik@redhat.com" target="_blank">pemensik@redhat.com</a> PGP: 65C6C973<br></blockquote></div></div></div></div></blockquote><div><br></div><div> </div></div></div></div>