<div dir="ltr"><div dir="ltr">> What value did you use?<div><br></div><div>I went brute force and used 1M. The default on this arm based device was also 212992.<br><br>root@MH7601:~# cat /proc/sys/net/core/wmem_default<br>1048576<br></div><div><br></div><div>I agree that is a lot but given the arp queue length has 101 entries, that is a lot of packets (especially if that might mean 101 hosts - not sure if the arp/neigh queue is per host or per request).</div><div><br></div><div>root@MH7601:~# cat /proc/sys/net/ipv4/neigh/default/unres_qlen<br>101<br></div><div><br></div><div>This is a very controlled environment, there are only about 30 sockets open at any time. This approach won't suit most people but it saved me from crafting a patch into openwrt.</div><div><br></div><div>Thanks,<br>Tom</div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Mon, May 16, 2022 at 10:52 AM Simon Kelley <<a href="mailto:simon@thekelleys.org.uk">simon@thekelleys.org.uk</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">What value did you use?<br>
<br>
On my Ubuntu desktop, /proc/sys/net/core/wmem_default and wmem_max are <br>
both 212992 which is a fair few DNS replies.<br>
<br>
<br>
Simon.<br>
<br>
<br>
On 16/05/2022 18:34, Tom Keddie wrote:<br>
> Hi Simon,<br>
> <br>
> Thanks for your response. I don't have the detailed logs but it's a <br>
> noisy qa wireless environment where clients are coming and going a lot. <br>
> eg. In syslog I could see instances where we would get a DHCP request <br>
> and then a L2 wireless disassociate message would appear immediately <br>
> afterwards, that response isn't going to be deliverable as unicast <br>
> (although for dhcp it might fall back to broadcast eventually).<br>
> <br>
> As we know, DNS isn't logged in such a manner but you could see the same <br>
> scenario unfolding where we get a bunch of dns requests, the client <br>
> drops off immediately afterwards and the responses can't be delivered. <br>
> When there's a lot of requests or a lot of clients you can see how the <br>
> socket buffer would fill.<br>
> <br>
> Increasing the socket buffers as I described below allowed the test to <br>
> run for the required 96 hours, without it we weren't making it past the <br>
> 48 hour mark.<br>
> <br>
> A dynamic solution might work provided it was carefully bound to prevent <br>
> DoS. If you have something you'd like us to test I probably arrange a <br>
> time slot, it's a busy setup that needs lots of hardware though.<br>
> <br>
> Thanks,<br>
> Tom Keddie<br>
> <br>
> ps. this is a controlled environment (as much as you can control wifi), <br>
> there are no malicious actors nor intent in this scenario. It's a soak <br>
> test with a large variety of clients all doing busy work like video <br>
> streaming etc.<br>
> <br>
> <br>
> On Fri, May 13, 2022 at 12:48 PM Simon Kelley <<a href="mailto:simon@thekelleys.org.uk" target="_blank">simon@thekelleys.org.uk</a> <br>
> <mailto:<a href="mailto:simon@thekelleys.org.uk" target="_blank">simon@thekelleys.org.uk</a>>> wrote:<br>
> <br>
> <br>
> <br>
> On 10/05/2022 16:40, Tom Keddie via Dnsmasq-discuss wrote:<br>
> > Hi All,<br>
> ><br>
> > I think you're saying that it's not surprising that dnsmasq<br>
> is not<br>
> > reading from the socket because the send queue is also full.<br>
> ><br>
> ><br>
> > As per this thread on netdev<br>
> ><br>
> (<a href="https://lore.kernel.org/netdev/CABUuw65R3or9HeHsMT_isVx1f-7B6eCPPdr+bNR6f6wbKPnHOQ@mail.gmail.com/" rel="noreferrer" target="_blank">https://lore.kernel.org/netdev/CABUuw65R3or9HeHsMT_isVx1f-7B6eCPPdr+bNR6f6wbKPnHOQ@mail.gmail.com/</a><br>
> <<a href="https://lore.kernel.org/netdev/CABUuw65R3or9HeHsMT_isVx1f-7B6eCPPdr+bNR6f6wbKPnHOQ@mail.gmail.com/" rel="noreferrer" target="_blank">https://lore.kernel.org/netdev/CABUuw65R3or9HeHsMT_isVx1f-7B6eCPPdr+bNR6f6wbKPnHOQ@mail.gmail.com/</a>><br>
> <br>
> ><br>
> <<a href="https://lore.kernel.org/netdev/CABUuw65R3or9HeHsMT_isVx1f-7B6eCPPdr+bNR6f6wbKPnHOQ@mail.gmail.com/" rel="noreferrer" target="_blank">https://lore.kernel.org/netdev/CABUuw65R3or9HeHsMT_isVx1f-7B6eCPPdr+bNR6f6wbKPnHOQ@mail.gmail.com/</a><br>
> <<a href="https://lore.kernel.org/netdev/CABUuw65R3or9HeHsMT_isVx1f-7B6eCPPdr+bNR6f6wbKPnHOQ@mail.gmail.com/" rel="noreferrer" target="_blank">https://lore.kernel.org/netdev/CABUuw65R3or9HeHsMT_isVx1f-7B6eCPPdr+bNR6f6wbKPnHOQ@mail.gmail.com/</a>>>)<br>
> <br>
> > it seems we were consuming the socket send buffer with pending<br>
> packets<br>
> > waiting for ARP responses that were never coming. This was causing<br>
> > failures sending to devices that were still live.<br>
> ><br>
> > As per that thread we increased the /proc/sys/net/core/wmem_default<br>
> > value so all sockets will have larger send buffers (the device<br>
> has very<br>
> > few sockets in use). It might be useful to add dnsmasq config<br>
> options to<br>
> > increase SO_SNDBUF on the dhcp and dns sockets to allow more<br>
> granular<br>
> > control.<br>
> ><br>
> > Thanks, Tom Keddie<br>
> <br>
> So queries are being received, and answered, but the reply is being<br>
> dropped by the kernel because the send queue is full of replies to dead<br>
> hosts? If the hosts are dead, where are the queries coming from to<br>
> generate these blocked replies?<br>
> <br>
> It might be sensible to automatically increase the send queue length<br>
> when a packer send gets EAGAIN. at least the first time, but I'd<br>
> like to<br>
> understand exactly what's going on first.<br>
> <br>
> <br>
> Simon.<br>
> <br>
> ><br>
> > _______________________________________________<br>
> > Dnsmasq-discuss mailing list<br>
> > <a href="mailto:Dnsmasq-discuss@lists.thekelleys.org.uk" target="_blank">Dnsmasq-discuss@lists.thekelleys.org.uk</a><br>
> <mailto:<a href="mailto:Dnsmasq-discuss@lists.thekelleys.org.uk" target="_blank">Dnsmasq-discuss@lists.thekelleys.org.uk</a>><br>
> ><br>
> <a href="https://lists.thekelleys.org.uk/cgi-bin/mailman/listinfo/dnsmasq-discuss" rel="noreferrer" target="_blank">https://lists.thekelleys.org.uk/cgi-bin/mailman/listinfo/dnsmasq-discuss</a><br>
> <<a href="https://lists.thekelleys.org.uk/cgi-bin/mailman/listinfo/dnsmasq-discuss" rel="noreferrer" target="_blank">https://lists.thekelleys.org.uk/cgi-bin/mailman/listinfo/dnsmasq-discuss</a>><br>
> <br>
</blockquote></div></div>