<div dir="ltr"><div>Hello,</div><div><br></div><div>I'm battling a little bit with a weird PXE Boot issue - i can pxeboot ubuntu 20.04 just fine but not 21.04. When booting 20.04 (which works just fine) the behaviour i see is:</div><div><br></div><div>1. transfers vmlinuz kernel image via tftp - OK<br></div><div>2. transfers initrd image via tftp - OK</div><div>3. Kernel boots, mounts the initial ramdisk then that's the end of tftp stuff, everything thereafter happens over http and it's fine</div><div><br></div><div>However, when booting 21.04, the behaviour i see is:</div><div><br></div><div>
<div>1. transfers vmlinuz kernel image via tftp - OK<br></div><div>2. transfers initrd image via tftp - FAILS, always near the end of the file transfer<br></div>
</div><div><br></div><div>The 20.04's initrd is 80mb in size, 21.04's is 99mb.</div><div><br></div><div>I chased some logs but without much luck - just saw a failure to transfer the file. I tried disabling dnsmasq's block size negotiation and that setting halfed the throughput of the transfers but it still failed near the end of the initrd transfer.</div><div><br></div><div>So then I decided to do a tcpdump from the server which is running dnsmasq and I can't fully wrap my head around what's happening:</div><div><br></div><div>Initially, i see lots of:</div><div><br></div><div> 20:08:48.751726 IP server.62770 > pxe-client-machine.1173: UDP, length 1412<br> 20:08:48.753260 IP pxe-client-machine.1173 > server.62770: UDP, length 4 <br></div><div>... repeated ...</div><div><br></div><div>But then the behaviour changes near the end of the transfer:<br></div><div><br></div><div>20:08:48.762027 IP server.62770 > pxe-cleint-machine.1173: UDP, length 1412<br>20:08:49.580276 IP pxe-cleint-machine.1173 > server.62770: UDP, length 4 <br>20:08:50.083707 IP server.62770 > pxe-cleint-machine.1173: UDP, length 1412<br>20:08:50.405353 IP pxe-cleint-machine.1173 > server.62770: UDP, length 4 <br>20:08:51.227949 IP pxe-cleint-machine.1173 > server.62770: UDP, length 4 <br>20:08:52.024222 IP server.62770 > pxe-cleint-machine.1173: UDP, length 1412<br>20:08:52.051790 IP pxe-cleint-machine.1173 > server.62770: UDP, length 4 <br>20:08:52.875666 IP pxe-cleint-machine.1173 > server.62770: UDP, length 4 <br>20:08:53.699587 IP pxe-cleint-machine.1173 > server.62770: UDP, length 4 <br>20:08:54.523490 IP pxe-cleint-machine.1173 > server.62770: UDP, length 4 <br>20:08:55.016630 IP server.62770 > pxe-cleint-machine.1173: UDP, length 1412<br>20:08:55.348003 IP pxe-cleint-machine.1173 > server.62770: UDP, length 4 <br>20:08:56.171466 IP pxe-cleint-machine.1173 > server.62770: UDP, length 4 <br>20:08:56.995009 IP pxe-cleint-machine.1173 > server.62770: UDP, length 4 <br>20:08:57.819022 IP pxe-cleint-machine.1173 > server.62770: UDP, length 4 <br>20:08:58.071977 IP server.62770 > pxe-cleint-machine.1173: UDP, length 1412<br></div><div><span class="gmail-pl-c"><span class="gmail-pl-c"></span></span></div><div><br></div><div>The sequence of 1412 byte transfer (this is below the MTU i have configured on this route) immediately followed by 4 byte ack / reply breaks down and I see duplicated 4 byte replies - which suggests unreliable network maybe?</div><div><br></div><div>I really doubt the unreliable network idea though, this is always exactly reproducible and the thing that makes me totally discount this unreliable network idea is the tcpdump is being captured from the server, so on the server i am seeing those retried 4 byte replies - at least i'm seeing them in tcpdump so dnsmasq should be getting them, i think.</div><div><br></div><div>Anyway, this culminates in the PXE firmware in the client nic flaking out and invoking a reboot of the machine but just before that, i see this in the tcpdump:<br></div><div><br></div><div>20:09:18.416590 IP server > pxe-cleint-machine: ICMP server udp port 62770 unreachable, length 40<br>20:09:23.359873 IP pxe-cleint-machine.1173 > server.62770: UDP, length 4 <br>20:09:23.359897 IP server > pxe-cleint-machine: ICMP server udp port 62770 unreachable, length 40<br>20:09:29.126861 IP pxe-cleint-machine.1173 > server.62770: UDP, length 4 <br></div><div><br></div><div>The server is telling the client that the ephemeral port used by dnsmasq to send the file is unreachable.</div><div><br></div><div>My current hypothesis is that there's some kind of failure in dnsmasq related to the length of the file. The very last packet sent by dnsmasq is still 1412 bytes long but the initrd is not a multiple of 1412 bytes so i think the final packet should be smaller?</div><div><br></div><div>The server host is a freebsd 13.0 machine running dnsmasq 2.85 binary from the freebsd ports collection.</div><div><br></div><div>> Dnsmasq version 2.85 Copyright (c) 2000-2021 Simon Kelley <br>> Compile time options: IPv6 GNU-getopt no-DBus no-UBus i18n IDN2 DHCP DHCPv6 no-Lua TFTP no-conntrack ipset auth cryptohash DNSSEC loop-detect no-inotify dumpfile<br></div><div><br></div><div>Both client and server are Dell t3610's. My config is here: <a href="https://github.com/craigjperry2/home-network/blob/main/roles/home-server/files/dnsmasq.conf">https://github.com/craigjperry2/home-network/blob/main/roles/home-server/files/dnsmasq.conf</a></div><div><br></div><div>
I'm not really sure of the best way to proceed debugging this, any hints, tips or suggestions? <br></div><div><br></div><div>All the best,</div><div><br></div><div>Craig</div><div><br></div><div><br></div></div>