[Dnsmasq-discuss] Extremely long startup times when using libidn2 (and proposed fix)

[Gustaf Ullberg]

Thanks Petr for having a look at this.
Since IDN processing do turns uppercase letters into lowercase, I
deliberately left uppercase letters out.
I think your approach to put everything in check_name() makes sense, even
if that function grows and maybe is starting to become a bit hard to read.

I took your patch and made a couple of changes:
- IDN processing is performed if there are uppercase letters present in the
name (unless we have an old version of libidn2 and there is an underscore
in the name).
- IDN processing is always performed if there are non-ascii characters in
the name, no matter if there are underscores or not (a non-processed name
containing non-ascii characters sounds dangerous).

The size of the blacklist is about 230000 lines, and I agree that it would
make sense to also file a bug on libidn2.

Den ons 8 sep. 2021 kl 15:25 skrev Petr Menšík <pemensik at redhat.com>:

> I think your check should also accept uppercase ASCII letters. Anyway,
> similar check is already done in check_names, which is there to skip names
> containing underscore with older libidn2 versions. I guess it could return
> 2 also in case ascii-only characters were detected, instead of checking the
> name again in another loop.
>
> Attached alternative change, which would process only names not only ascii
> names. Changes check_names to return 2 when IDN should be used. Printing
> ascii names should be safe, even when they contain characters not allowed
> by hostnames. Such as _, +, = or whatever garbage is present. As long as it
> is readable in logs, it should not matter.
>
> How many lines does your dnsmasq.blacklist.txt contain? Those differences
> are significant. Maybe bug should be filled on libidn2. Conversion from
> ascii-only name to ascii name should not take too long even if it was
> called.
> On 9/6/21 3:27 PM, Gustaf Ullberg wrote:
>
> Hi Simon and dnsmasq contributors,
>
> I am running dnsmasq with a blocklist from
>
> https://github.com/notracking/hosts-blocklists/blob/master/dnsmasq/dnsmasq.blacklist.txt
>
> I have noticed that building dnsmasq with libidn2 support (which my distro
> does) can cause extreme slowdowns. The slowdowns seem to come from the call
> to idn2_to_ascii_lz in canonicalise() being very slow.
>
> idn2_to_ascii_lz is run on every domain name in the blocklist to encode
> special characters, and this is surprisingly slow even when there are no
> special characters. I developed a patch (attached to this email) that
> checks a domain name for other characters than . - a-z 0-9. If any such
> character is found, the domain name will be encoded. If no such character
> is found the domain name will not be encoded (as encoding won't change it).
> This removes most of the overhead of using libidn2. Unless you find any
> problems with this approach, I wish the patch can be mainlined.
>
> Some benchmarks on a Raspberry Pi (slow, but probably not an uncommon
> device for running dnsmasq) running ArchLinux and dnsmasq git master:
>
> # Without libidn2: Acceptable speed
> > make
> > time ./src/dnsmasq -C dnsmasq.blacklist.txt --test
> dnsmasq: syntax check OK.
>
> real 0m3.699s
> user 0m3.468s
> sys 0m0.200s
>
>
>
> # With libidn2: To slow to be usable
> > make COPTS="-DHAVE_LIBIDN2"
> > time ./src/dnsmasq -C dnsmasq.blacklist.txt --test
> dnsmasq: syntax check OK.
>
> real 1m6.921s
> user 0m59.509s
> sys 0m0.606s
>
>
> # With libidn2 and attached patch: Back to acceptable speed
> > git am 0001-Avoid-IDN-translations-when-not-needed.patch
> > make COPTS="-DHAVE_LIBIDN2"
> > time ./src/dnsmasq -C dnsmasq.blacklist.txt --test
> dnsmasq: syntax check OK.
>
> real 0m3.903s
> user 0m3.643s
> sys 0m0.219s
>
> Best regards,
> Gustaf
>
> _______________________________________________
> Dnsmasq-discuss mailing listDnsmasq-discuss at lists.thekelleys.org.ukhttps://lists.thekelleys.org.uk/cgi-bin/mailman/listinfo/dnsmasq-discuss
>
> --
> Petr Menšík
> Software Engineer
> Red Hat, http://www.redhat.com/
> email: pemensik at redhat.com
> PGP: DFCF908DB7C87E8E529925BC4931CA5B6C9FC5CB
>
> _______________________________________________
> Dnsmasq-discuss mailing list
> Dnsmasq-discuss at lists.thekelleys.org.uk
> https://lists.thekelleys.org.uk/cgi-bin/mailman/listinfo/dnsmasq-discuss
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.thekelleys.org.uk/pipermail/dnsmasq-discuss/attachments/20210908/a1c1222e/attachment-0001.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: 0001-check_name-determines-if-IDN-processing-is-needed.patch
Type: text/x-patch
Size: 3612 bytes
Desc: not available
URL: <http://lists.thekelleys.org.uk/pipermail/dnsmasq-discuss/attachments/20210908/a1c1222e/attachment-0001.bin>