<div dir="ltr">Ok, I have an interesting update coming from a very different direction. Here is this whole big post I was just about to start a new thread for, thinking it was a separate issue. It turns out that removing query-port=0 completely solves the bad behavior described below. (I tried both with the unpatched 2.76, and with today's head of master `make install`ed; same result).<div><div><br></div><div>====================</div><div><br></div><div><div>dnsmasq is listening on 192.168.42.1, and is configured to use 127.8.8.8, which is a properly functioning DNAT to 8.8.8.8. </div><div><br></div><div>This setup works perfectly when I play the role of dnsmasq with `host <a href="http://example.com">example.com</a> 127.8.8.8`. Bizarrely, dnsmasq also works..... *exactly once*. Then it starts returning REFUSED without even attempting to contact 127.8.8.8. Restarting dnsmasq gets you your single resolution opportunity back. dnsmasq is configured with no-resolv, server=127.8.8.8, query-port=0, and DNSMASQ_EXCEPT=lo.</div><div><br></div><div>host <a href="http://sdfgdsfgd.google.com">sdfgdsfgd.google.com</a> 192.168.42.1 from the machine running dnsmasq succeeds only the first time each dnsmasq restart</div><div>host <a href="http://sdfgdsfgd.google.com">sdfgdsfgd.google.com</a> 192.168.42.1 from a client machine succeeds only the first time each dnsmasq restart</div><div>host <a href="http://sdfgdsfgd.google.com">sdfgdsfgd.google.com</a> 127.8.8.8 from the machine running dnsmasq always succeeds (sanity check)</div><div>host <a href="http://sdfgdsfgd.google.com">sdfgdsfgd.google.com</a> 127.8.8.9 from the machine running dnsmasq fails (sanity check)</div><div>host <a href="http://sdfgdsfgd.google.com">sdfgdsfgd.google.com</a> 8.8.8.8 from a client machine always succeeds (sanity check)</div><div>("sdfgdsfgd" means I banged on the keyboard to avoid any possible caching. Success is an NXDOMAIN, failure is REFUSED.)</div><div><br></div><div>I am sure the initial success really is taking the path I want; tcpdump shows 4 packets:</div><div>192.168.42.1:54321-><a href="http://192.168.42.1:53">192.168.42.1:53</a></div><div>127.0.0.1->8.8.8.8</div><div>127.8.8.8->127.0.0.1</div><div>192.168.42.1:53->192.168.42.1</div><div>For the failed resolution, tcpdump shows no traffic involving 127.8.8.8 or 8.8.8.8, just the request and REFUSED on 192.168.42.1.</div><div><br></div><div>In dnsmasq's logs, successes show up as a "query[A]" line followed by a "forwarded to 127.8.8.8", and then some "reply" lines. Failures show up as just the query[A], with nothing else.</div><div><br></div><div>PS: I guessed that DNSMASQ_EXCEPT in /etc/default/dnsmasq would do the same thing as except-interface in dnsmasq.conf. However, with just except-interface=lo, resolv.conf still gets set to 127.0.0.1, whereas DNSMASQ_EXCEPT=lo sees it get set to the normal 8.8.8.8 8.8.4.4. Is that a bug, or does DNSMASQ_EXCEPT have some other meaning I'm not aware of?</div></div></div></div><br><div class="gmail_quote"><div dir="ltr">On Tue, Apr 10, 2018 at 8:01 PM Fred Douglas <<a href="mailto:fredlas@google.com">fredlas@google.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">I built and installed 2.80test1, and I still see the same "cannot make/bind socket: Address already in use" in the logs. I should probably mention I've had DNSMASQ_EXCEPT=lo in /etc/default/dnsmasq most of this time, but I also tried without it just now - no difference. This is on a Raspberry Pi running the current Raspbian. I will throw in some logging and try to investigate further sometime in the next few days.</div><br><div class="gmail_quote"><div dir="ltr">On Tue, Apr 10, 2018 at 4:49 PM Simon Kelley <<a href="mailto:simon@thekelleys.org.uk" target="_blank">simon@thekelleys.org.uk</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">I still can't reproduce this, but tracing through the code, I found two<br>
different problems which might have a bearing. I just pushed the fixes<br>
to these to git, and tagged 2.80test1<br>
<br>
The fixes (top two at<br>
<a href="http://thekelleys.org.uk/gitweb/?p=dnsmasq.git;a=summary" rel="noreferrer" target="_blank">http://thekelleys.org.uk/gitweb/?p=dnsmasq.git;a=summary</a>) should apply<br>
to 2.76, if that's the easiest way to test, otherwise build from the<br>
2.80test1 tarball.<br>
<br>
I'd be very interested to know if they help.<br>
<br>
<br>
Cheers,<br>
<br>
Simon.<br>
<br>
<br>
On 09/04/18 17:01, Fred Douglas wrote:<br>
> Ok, got the output of log-queries=extra. It is indeed the bind at<br>
> program start:<br>
> <br>
> reading /run/dnsmasq/resolv.conf<br>
> ignoring nameserver 8.8.8.8 - cannot make/bind socket: Address already<br>
> in use<br>
> ignoring nameserver 8.8.4.4 - cannot make/bind socket: Address already<br>
> in use<br>
> <br>
> That's query-port!=0. With =0 or unset, you get<br>
> reading /run/dnsmasq/resolv.conf<br>
> using nameserver 8.8.8.8#53<br>
> using nameserver 8.8.4.4#53<br>
> <br>
> "ignoring nameserver - cannot make/bind" is printed when the<br>
> allocate_sfd function fails to allocate a socket set. allocate_sfd<br>
> returns null early when !daemon->osport, which I guess is why<br>
> query-port=0 sees the same good behavior as query-port unset. So, I<br>
> would guess the problem is in allocate_sfd.<br>
> <br>
> dnsmasq does not exit after that error happens, and I assume sees itself<br>
> as not having access to any resolvers, causing the REFUSEDs.<br>
> <br>
> On Sat, Apr 7, 2018 at 6:45 PM Simon Kelley <<a href="mailto:simon@thekelleys.org.uk" target="_blank">simon@thekelleys.org.uk</a><br>
> <mailto:<a href="mailto:simon@thekelleys.org.uk" target="_blank">simon@thekelleys.org.uk</a>>> wrote:<br>
> <br>
> On 07/04/18 14:47, Fred Douglas wrote:<br>
> > Thanks for the explanation of REFUSED's meaning! I bet it's that<br>
> the UDP sends are outright failing; I suspect that something is<br>
> going wrong with the bind at program start. I'll take a look at the<br>
> logs and report back on Monday.<br>
> <br>
> When using port-randomisation, dnsmasq has to create and bind sockets<br>
> for each upstream interaction. Once you nail the port number using<br>
> query-port, it doesn't need to do that and will create and bind a<br>
> single socket at startup which it uses thereafter. A failure of that<br>
> process should cause a fatal error and abort at start-up.<br>
> <br>
> ><br>
> ><br>
> > For now, though, I can pretty confidently say I'm not accidentally<br>
> blocking the packets. All of my iptables rules are either for TCP,<br>
> not for the interface that goes to the internet (eth0), or are<br>
> matching UDP ports that these experiments aren't using.<br>
> ><br>
> ><br>
> > I used query-port=0, observed the unchanging source port of the<br>
> (successful) resolutions, restarted dnsmasq with<br>
> query-port=that_port - and got the error. Even if I was getting<br>
> unlucky, and that attempt and my other attempts in the ephemeral<br>
> range were failing because the port happened to be in use when<br>
> dnsmasq tried to bind it, that shouldn't be the case for the lower<br>
> numbered ports I was trying. (I'm not making any other changes in<br>
> between these experiments, either, just changing query-port in<br>
> dnsmasq.conf to commented, 0, or non-0, and then `service dnsmasq<br>
> restart`.)<br>
> <br>
> <br>
> running dnsmasq under strace (run dnsmasq with the -d option) would be<br>
> useful, to see exactly what system calls it's making.<br>
> <br>
> You have used --log-queries to make sure this REFUSED return code isn't<br>
> coming from usptream, haven't you?<br>
> <br>
> <br>
> Cheers,<br>
> <br>
> Simon.<br>
> <br>
> <br>
> ><br>
> >> Just tried a simple test, and didn't see the same behaviour.<br>
> >><br>
> >> Use log-queries to check that the process is really failing in<br>
> dnsmasq,<br>
> >> ie the problem is not REFUSED answers from upstream. A REFUSED answer<br>
> >> from dnsmasq only occurs if either there are no possible upstream<br>
> server<br>
> >> to forward to, or if attempts to send UDP packets to all upstream<br>
> >> servers fail immediately, at kernel level. You're not accidentally<br>
> >> blocking packets from you special port, are you?<br>
> >><br>
> >><br>
> >> Cheers,<br>
> >><br>
> >> Simon.<br>
> >><br>
> >>On 06/04/18 21:08, Fred Douglas wrote:<br>
> >>>/I would like dnsmasq to stick to a single source port for its<br>
> requests, />>/so that I can differentiate them from other DNS<br>
> requests going out the />>/same interface. />>//>>/The query-port<br>
> option works as advertised when set to 0 (i.e. picks a />>/single<br>
> random port and sticks to it). Any other value, however - below<br>
> />>/1024, a little above 1024, way up in the 50000s - causes dnsmasq<br>
> to />>/respond to all queries with a "REFUSED" (DNS error code 5).<br>
> />>//>>/My dnsmasq.conf is empty other than query-port, and I<br>
> haven't made any />>/other weird changes to the system that should<br>
> be relevant. This is />>/Debian's current [2.76+whatever security<br>
> patches] version of dnsmasq. />>//>>/Does anyone else get this<br>
> behavior? />>//>>/Fred<br>
> />>//>>//>>/_______________________________________________<br>
> />>/Dnsmasq-discuss mailing list />>/Dnsmasq-discuss at<br>
> <a href="http://lists.thekelleys.org.uk" rel="noreferrer" target="_blank">lists.thekelleys.org.uk</a> <<a href="http://lists.thekelleys.org.uk" rel="noreferrer" target="_blank">http://lists.thekelleys.org.uk</a>><br>
> > <<a href="http://lists.thekelleys.org.uk/mailman/listinfo/dnsmasq-discuss" rel="noreferrer" target="_blank">http://lists.thekelleys.org.uk/mailman/listinfo/dnsmasq-discuss</a>><br>
> />>/<a href="http://lists.thekelleys.org.uk/mailman/listinfo/dnsmasq-discuss" rel="noreferrer" target="_blank">http://lists.thekelleys.org.uk/mailman/listinfo/dnsmasq-discuss</a> />><br>
> ><br>
> ><br>
> ><br>
> > _______________________________________________<br>
> > Dnsmasq-discuss mailing list<br>
> > <a href="mailto:Dnsmasq-discuss@lists.thekelleys.org.uk" target="_blank">Dnsmasq-discuss@lists.thekelleys.org.uk</a><br>
> <mailto:<a href="mailto:Dnsmasq-discuss@lists.thekelleys.org.uk" target="_blank">Dnsmasq-discuss@lists.thekelleys.org.uk</a>><br>
> > <a href="http://lists.thekelleys.org.uk/mailman/listinfo/dnsmasq-discuss" rel="noreferrer" target="_blank">http://lists.thekelleys.org.uk/mailman/listinfo/dnsmasq-discuss</a><br>
> ><br>
> <br>
> <br>
> _______________________________________________<br>
> Dnsmasq-discuss mailing list<br>
> <a href="mailto:Dnsmasq-discuss@lists.thekelleys.org.uk" target="_blank">Dnsmasq-discuss@lists.thekelleys.org.uk</a><br>
> <mailto:<a href="mailto:Dnsmasq-discuss@lists.thekelleys.org.uk" target="_blank">Dnsmasq-discuss@lists.thekelleys.org.uk</a>><br>
> <a href="http://lists.thekelleys.org.uk/mailman/listinfo/dnsmasq-discuss" rel="noreferrer" target="_blank">http://lists.thekelleys.org.uk/mailman/listinfo/dnsmasq-discuss</a><br>
> <br>
</blockquote></div></blockquote></div>