On Mon, 09 Sep 2019 22:21:05 +0200 Evilham said :
> Hey,
>
> thank you again for the email, I commented some on IRC a few days
> ago and updated the list as necessary in some points.
>
> On dv., set. 06 2019, onefang wrote:
>
> > I've been developing my mirror checker script for some time now,
> > and I
> > spent most of the last day testing it and fixing up things /
> > adding
> > better checks. This is the second week this weekly update has
> > given
> > lots of errors. I'll make some comparisons with these results.
> >
> > For the few mirrors I don't mention here, my script agrees with
> > this
> > email.
> >
> > My script doesn't yet support Updated and Integrity tests. It
> > also
> > doesn't yet produce this email, nor the similar web page. All
> > of these
> > are TODO items.
>
> TBH, I'd be less interested in an email that says "this is what
> works and what doesn't *right now, this time on Friday*" than on
> "this week, these were the downtimes / times out of sync of the
> mirrors".
>
> There could be some issues on the current checker, but also it can
> be a matter of timing and that's why some mirrors appear as
> "faulty" on this picture that is sent every week.

You had already mentioned generating a prometheus metrics file, which I
had already added as a TODO item on the issue tracker at
https://sledjhamr.org/mantisbt/project_page.php?project_id=13

For myself I would also add RRD (I use Collectd and CGraphz) and likely
an interface for Icinga/Nagios. So yes, running the script often and
collecting results is on the TODO. Also generating a web page similar
to the current mirror status web page, but with links so you can drill
down on the details. I might pull my finger out and figure out how
CGaphz dashboard works and public guest accounts, so I can show the
world some pretty graphs.

The big problem with running the tests "often" is something I mentioned
to you before, downloading a few files from all the mirrors for
integrity checking consumed 4GB of bandwidth per test. Something I'm
uncomfortable doing from home, in Australia, where their are limits on
bandwidth usage per month. My last home I had 30 GB per month quota,
it wouldn't take many 4GB download sessions to use that all up.
There'll be a big warning for other people using my script if they run
those particular tests.

So that sort of test I'll be moving to my server, which is in Europe,
where bandwidth is a lot cheaper, cheap enough for me to run a high
speed Devuan mirror on it. B-)

Actual timing of these tests will be configurable, with a weekly
summary email.

> > On Fri, 6 Sep 2019 16:37:01 +0100 Devuan Mirrors said :
> >

> >> mirror.koddos.net/devuan/devuan....
> >>       http: [OK] https: [OK] DNS-RR: [OK] Updated: [SKIP] 
> >>       Integrity:
> >> [OK]  

> >
> > The official mirror list says "only usable via deb.devuan.org",
> > so my
> > mirror checker doesn't bother to check it directly. It does
> > check it
> > via the deb.devuan.org DNS-RR. The HTTP tests and DNS-RR tests
> > passed.
> > My script doesn't do HTTPS checks on mirrors accessed via the
> > deb.devuan.org DNS-RR, as the mirrors should in theory NOT have
> > the
> > deb.devuan.org certificate. During testing I noticed that some
> > of them
> > "work" anyway.
> >
> > In not so sure what to do about this, flag them as WARNING since
> > they
> > should not do that? Check further to see if there is some sort
> > of
> > valid cert that somehow passes my checks?
>
> Probably something fishy in your checker script, I'd adventure to
> suggest it only checks that a TLS connection is offered and the
> right content is served, and not the validity / trust chain of the
> cert.

That's entirely possible. I traced an endless loop in the script to
the fact that the version of LuaSocket on ASCII doesn't really do
HTTPS, but silently converts it to HTTP, which caused a loop for the
mirror that redirects HTTP to HTTPS. I added LuaSec and use that in
conjunction with LuaSocket. I have not retested the DNS-RR with HTTPS
since I fixed that.

I saw a commit to LuaSocket recently that fixed it to properly support
HTTPS, but I'm sticking to versions of things that are available from
ASCII for this script.

> This mirror however does not serve the right content (even with
> the wrong cert) over HTTPS, maybe some other server does, it
> wouldn't be crazy.

As I mentioned, I'm not testing integrity yet, that's coming soon.

> But this particular one is indeed misconfigured over IPv6, IIRC I
> mentioned that exclusively checking based on a hostname is
> dangerous because these situations are awfully common (or
> redirections to other hosts, ...):

Also not testing IPv6 yet, since my new home doesn't have that. I'll
likely have that later in the year. My server has IPv6, and I'll be
doing more testing there soonish.

As we had discussed when I started this mirror checker project, I look
up IPs, and CNAMES, and do checks with Host: headers on every IP I
find. Probing at each step all the way down any redirect chains.

> # host mirror.koddos.net
> mirror.koddos.net has address 31.220.0.151
> mirror.koddos.net has IPv6 address 2001:590:3803::31:151
>
> # openssl s_client -servername deb.devuan.org -connect
> 31.220.0.151:443
> Shows: subject=OU = Domain Control Validated, CN = *.koddos.net
>
> $ curl -Ik -H 'Host: deb.devuan.org'
> https://31.220.0.151/mirror_list.txt
> HTTP/1.1 404 Not Found
>
> # openssl s_client -servername deb.devuan.org -connect
> [2001:590:3803::31:151]:443
> Times out, as does:
> # curl -6 http://mirror.koddos.net/devuan/devuan
>

> >> devuan.ipacct.com/devuan....
> >>       http: [OK] https: [OK] DNS-RR: [OK] Updated: [SKIP] 
> >>       Integrity:
> >> [OK]  

> >
> > The official mirror list says this mirror doesn't support HTTPS,
> > so I
> > don't test that. It's also not on the DNS-RR.
>
>
> For the class of issues "HTTPS supported but not listed":
> ATM the list is maintained manually, so if changes are not
> notified, they are not reflected. The list was updated with this
> class of issues unless I missed some.

For things like this I'll be saying "https: [N/A]", which will mean the
mirror list says it's not supported. So just as this email prompted
some mirror admins to say "hey, we do support HTTPS now", this should
prompt similar reports, and thus we can manually update the list.

Maybe I could test HTTPS anyway, then report "https: [OK, fix list]" or
some such.

> For the class of issues "RR is supported but it hasn't been
> added": maybe we should fix that :-). I'd feel much more confident
> adding things if we had that email with a % of proper responses /
> uptime over the week.

Yes, we chatted about that in IRC. You told me the test to add to the
RR is for it to work reliably, and speedily. I'll be adding some
minimal speed tests to my script. It's downloading things for other
tests anyway, I can time them and log them.

> >> ==== faulty mirrors: ====
> >>
> >> pkgmaster.devuan.org (https)
> >
> > All tests passed, including HTTPS.
>
> Given this is the reference server, it was possibly a network
> hiccup. As mentioned, the email is based on taking a picture of
> current state and therefore these things can happen.

Yep, I often get different results in different test runs, and then have
to figure out if the different result is coz I had just broken
something, or there's a new hiccup elsewhere I just detected. Testing
this sort of thing keeps you on your toes. B-)

> >> mirror.4isp.it (DNS-RR)
> >
> > The official mirror list says "temporarily-offline", so my
> > mirror
> > checker doesn't bother to check it. I should probably add an
> > option to
> > check it anyway, so the admin of this mirror knows what works
> > and what
> > doesn't.
>
> It was removed of the RR because it was being problematic; we
> never got an answer so I just changed the state to "offline".

That's an argument for "test it anyway". If it starts showing
consistently good results, we could change it back to "Active: yes".
Though feedback to "Are you having problems" questions is also good.

> >> dist-mirror.fem.tu-ilmenau.de/devuan (http)
> >> dist-mirror.fem.tu-ilmenau.de/devuan (https)
> >
> > This mirror has some issues due to it automatically redirecting
> > HTTP to
> > HTTPS. I spent most of the day trying to sort this out, not
> > sure
> > exactly where the problem was. Likely it shouldn't be doing
> > that, since
> > not everyone has apt-transport-https installed. If you don't
> > have that
> > installed and you use this mirror, things will probably break.
> > It
> > should work fine if you actually use HTTPS with
> > apt-transport-https.
> > I'd give this one a WARNING instead of an ERROR, but it's
> > debatable
> > that maybe this should be flagged as an ERROR.
>
> apt-transport-https comes by default on buster/beowulf, so I'd
> expect this to be more common over time (also given the history of
> apt issues that have proven that there is some value in HTTPS for
> these things as well).

This is the one that was causing the loop during testing. The mirror
would redirect HTTP to HTTPS, LuaSocket would silently convert HTTPS
requests to HTTP, then the mirror would redirect the HTTP to HTTPS, and
around we go again. Now I'm using LuaSec for HTTPS, that loop doesn't
happen.

> Redirection to HTTPS is only a *problem* if the used hostname is
> deb.devuan.org, otherwise the user is free to pick a different
> mirror that does plain HTTP.

OK, so this would be ERROR if we are checking deb.devuan.org, but a
WARNING otherwise. I already have other WARNINGS, they are things that
would be nice if they didn't happen, but not show stoppers. So if the
HTTP only using user is having problems, they check the mirror status
web page, see the "WARNING: redirects HTTP to HTTPS", and knows to
switch mirrors.

> >> devuan.bio.lmu.de (DNS-RR)
> >
> > The official mirror list says "Active: not yet", so my mirror
> > checker doesn't bother to check it.
>
> That should be 'Active: yes'. It is part of the RR. Fixed.

Thanks.

> > I'll keep working on my script. It may replace the script that
> > generates this email, and the similar web page one day.
>
> Thank you for this, looking forward to seeing how it evolves, I
> hope these changes / comments were useful.

Next on my TODO is to write proper documentation, which will include a
high level theory of operation, so you can check I'm doing the correct
things. And will likely include a little reorganisation of the script
internals to make things a bit more obvious.

--
A big old stinking pile of genius that no one wants
coz there are too many silver coated monkeys in the world.