Dear developers,

as I mentioned some days ago, there is a whole class of potential issues
we are not considering in the mirror checks.

You can find a first version of my go at it here:
https://git.devuan.org/evilham/mirror-checker


It is fairly well documented and is working just fine, there are a
couple things left to do though.

I published an example of the (current) output here:

https://evilham.com/d1mc.example.json

Abstractly, what this is doing is checking by using any combination of
(IP, PORT) associated with a given hostname and protocol.
It also does not blindly follow redirects, but instead recursively
applies the trick of expanding each hostname to a list of (IP, PORT) pairs.
Notice that SRV records (RFC2782) as described in a previous email allow
for odd things, like having an HTTP request be resolved to a totally
different host on a non-standard port.

The default check is running against ascii main Release, Packages.gz
files as well as a devuan package and a debian package. It runs in a
decent amount of time.

Output is JSON because it's machine readable and was trivial to add, I
can look into other options at some point.
The important bit is the root { "status": "(KO|OK)" }, which should show
OK, but currently is not.

Reason for status being "KO" is that at the moment I ran the test (about
10 minutes ago), debian.bio.lmu.de is not responding via IPv6, therefore
people with IPv6 connectivity will randomly have issues installing
packages that are not forked.

Notice as well that the entry for the offending host has a
"manual_check" entry, that tells you how to reproduce the problem:

# curl -I -H 'Host: debian.bio.lmu.de'
'http://[2001:4ca0:4300::1:22]/debian/pool/main/a/apt/apt-doc_1.4.8_all.deb'
curl: (7) Failed to connect to 2001:4ca0:4300::1:22 port 80: Connection
timed out

As you can see, these checks do address very real issues.

I duplicate the help text from the package below for informational
purposes. Do let me know if there are things that can be done
significantly better.



python -m DevuanMirrorChecker OPERATION

Where OPERATION is one of:

--help: Show this help page.

--show-hosts HOSTNAME [PROTOCOL [PORT]]:
Query DNS for all hosts involved in serving PROTOCOL for HOSTNAME.
This always resolves HOSTNAME for A and AAAA records, following any
CNAMES if needed.
Additionally, if PROTOCOL is passed, SRV records are retrieved as per
RFC2782 and then those hostnames are resolved.

Output is a list of (IP, port) where port is that defined in the SRV
record for IPs coming from SRV, defaulting to PORT for IPs resolved
from HOSTNAME.

  Examples:
    python -m DevuanMirrorChecker --show-hosts deb.devuan.org http
    python -m DevuanMirrorChecker --show-hosts deb.debian.org http

--main [FILE_1, ..., FILE_N]
Check devuan's mirror network for sync with pkgmaster.

FILE_X arguments must be a path in pkgmaster.

NOTE: Only file content is being checked.
Currently this relies on a correct connection to pkgmaster.

  Example:
    python -m DevuanMirrorChecker --main /merged/pool/DEVUAN/\
    main/b/base-files/base-files_9.9+devuan2.5.all.deb

--
Evilham