Hi onefang,

thank you for your detailed explanation!

onefang wrote on 06.11.19 04:00:
[...]
>
> For each IP found in the DNS lookups I do the same HEAD probe using the
> IP in the URL and the mirrors original domain name in a "Host = "
> header. This is where things go wrong for
> https://devuan.packet-gain.de. https://mirror.stinpriza.org has the
> exact same issue.
[...]
> Correct, the checking of https://devuan.packet-gain.de/ gives no
> errors. However checking http://95.216.15.86/ with the HTTP header
> "Host = devuan.packet-gain.de" gives a result of -
>
> 421 HTTP/1.1 421 Misdirected Request.
>
> Which would result in no file downloaded, thus it is flagged as an
> error. Well 34 errors, one for each file. There are also warnings
> that are flagged, things that will give the correct result, but that
> probably could be cleaned up at the mirror end (or some that need
> mirror_list.txt to be cleaned up).
[...]
>
> The reason why I check IPs in URLs with "Host = " headers is that is how
> the DNS round robin works, which I also check, and so it's possible that
> some of the mirrors themselves might do the same. As I said above, I'm
> probing into every little nook and cranny, this is one of those nooks.

Given that HTTPS delivery can't and won't possibly work with the DNR-RR,
I think it would be reasonable to expect the DNS-RR entries to only give
meaningful results with plain HTTP requests. Actually, I am quite
surprised you're not getting this kind of "errors" on more mirrors.

> I had never seen a 421 HTTP code before, and I didn't know what it
> was. https://serverfault.com/questions/916724/421-misdirected-request
> might be helpful, it seems to suggest some issue in a complex multi
> host environment. Maybe that might shed some light on this issue.

It is indeed a multi host setup, with separate SSL certificates, one
for each served (sub-)domain. Any attempt to request via HTTPS an URL
using the numerical IP address instead of the FQDN will result in some
kind of error or warning, either client-side or server-side.

> So there maybe some glitch in my checking logic that only two mirrors
> trigger, or there maybe is some real issue on those two mirrors. You'll
> be able to double check things at your end and let me know if this is a
> real issue for you to fix, or give me more details so that maybe I can
> figure out what I'm doing wrong and how I can fix it.
[...]

I'll look into it more in-depth, but as far as I can tell from a cursory
inspection of the configuration there is no real issue on the server side.
Some restrictions are set reasonably tight, and that is fully intentional.
Right now I'm quite hesitant to loosen any restrictions WRT cross-domain
SSL access that could potentially compromise the entire server setup.

> I have attached the checking logs for devuan.packet-gain.de, one is the
> log for checking devuan.packet-gain.de, the other is the 95.216.15.86
> with "Host = devuan.packet-gain.de" check. I hope these are readable,
> though I intend to try to make them more readable in the future.

Thank you for the logs. They show just what I would have expected though.

If I were in your position, for numerical IPs I'd only keep the HTTP checks
in place and drop the HTTPS tests, as these are IMHO asking for trouble in
the form of a lot of noise in the logs. Of course, for the FQDNs as listed
in mirrors.txt HTTPS should be expected to work as advertised and should be
tested by all means!

I conclusion (as I see it and to the best of my knowledge, please correct
me if I'm wrong): You are currently testing a certain feature, namely
cross-domain HTTPS requests, that was never advertised to work.

HTH.
Best regards,

Urban

--
Sapere aude!