Dear onefang,

thank you for apt-panopticon, and especially for running it frequently on
all mirrors. It has already proven to be very useful to me, and I like the
near-realtime reporting. I'm trying to establish a habit of checking the
report somewhat regularly so I can tell in a timely manner if something is
wrong with my mirror.

The reason I'm replying here is that there were actually a bunch of
warnings that were displayed for my mirror, which I have finally all fixed.
The better part of the warnings were about the debian redirect for partial
mirrors, and in particular the way it is set up with lighttpd. I checked
the list and noticed there is at least one more mirror suffering from the
same issue, so I'd like to share how to solve it here:

The redirects in my mirror's lighttpd configuration (and probably in the
other lighttpd-based mirrors as well, since my configuration was used for
the lighttpd example) are set up like this:

    url.redirect = (
        "^/merged/pool/DEVUAN/(.*)" => "/devuan/pool/$1",
        "^/merged/pool/DEBIAN/(.*)" => "
https://deb.debian.org/debian/pool/$1",
        "^/merged/pool/DEBIAN-SECURITY/(.*)" => "
https://deb.debian.org/debian-security/pool/$1"
    )

The script, in its URL sanitation check sends a request with multiple
slashes, which I couldn't reproduce on my side at first, since my client
was smart/sneaky enough to remove superfluous slashes before issuing the
request to the server. Using the same command as your script, I realized
that lighttpd chokes on the multiple slashes because it performs
redirection *before* sanitizing the URLs. I'm not sure why lighttpd would
do it in this order, since that means the regexes will have to account for
all sorts of perfectly valid garbage (like
"/.//.///.////./foo//.///../index.html"). Anyways, I changed the redirects
to

    url.redirect = (
        "^[/]+merged[/]+pool[/]+DEVUAN[/]+(.*)" => "/devuan/pool/$1",
        "^[/]+merged[/]+pool[/]+DEBIAN[/]+(.*)" => "
https://deb.debian.org/debian/pool/$1",
        "^[/]+merged[/]+pool[/]+DEBIAN-SECURITY[/]+(.*)" => "
https://deb.debian.org/debian-security/pool/$1"
    )

which means that it now works for your test cases, but it would still choke
on funky requests like the one mentioned above which is why I consider this
more of a workaround than a solution. I'm trying to see if theres an easy
way around it, otherwise I might be willing to give nginx a try instead. In
the meanwhile, other lighttpd-based mirrors (at a glimpse, at least
mirror.devuan.de seems affected) might want to adopt the same redirects.

The other issue that I've had with my mirror was that it ran out of disk
space - it was originally only 40GB which I upgraded to 100GB a couple of
months ago. I just upgraded to 200GB today and I'm hoping that should
suffice for now. Is there any indication for an upper limit on the disk
usage? I think debian is hosting five distributions at a time
(oldoldstable, oldstable, stable, testing, unstable) and we're at four
(oldstable thru unstable) at the moment, so my guess would be at least a
25% increase? The reason I'm asking is because I'm considering to add more
storage and upgrade to a complete mirror, without the debian redirect.
Could you shed some light on the storage situation on pkgmaster.devuan.org,
maybe some storage recommendations for a partial and a full mirror? I
realize there are several factors this depends upon, so a conservative
guess would be fine. :-)

Thank you!


Best regards,
Matthias



2020年3月7日(土) 0:40 onefang <onefang_devuan@???>:

> Alas life got in the way and has delayed all my projects. So I'm sending
> this email much later than I had planned.
>
> apt-panopticon is basically ready for an alpha release.
>
> With the blessing of the other Devuan devs (at least no one complained at
> the time, or since) Evilham and I have decided to use the results of
> apt-panopticon to decide which mirror servers are on the DNS-RR and which
> are not. https://pkgmaster.devuan.org/mirror_list.txt will be updated
> for things like "certificate expired, so they no longer support HTTPS",
> "have been doing nothing but timeouts for sometime, so mark as offline",
> etc. The DNS-RR will be modified to reflect serious issues.
>
> Evilham wants me to go one step further, and automate things. His idea
> was to run apt-panopticon once a minute, and report the results to a
> backend that will automatically add or remove mirror servers to DNS-RR
> depending on yet to be discussed pass conditions. On the fastest server
> it is currently running on the average run time is slightly less than one
> minute, so I think maybe doing this every five minutes. The goal is that
> when someone uses the DNS-RR, they get as perfect an experience as we can
> manage.
>
> Pass conditions are likely to be something like - passes all tests with
> no warnings (except HTTPS that DNS-RR can't handle anyway), 100% up time,
> 100% updated. Very few currently meet those conditions.
>
> Apt-panopticon is running on two different servers now, plus my desktop,
> and I'm awaiting a friend to get IPv6 for her server before I run it
> there. Mostly coz running it on an actual mirror server means that
> particular mirror gets disk speed tested instead of network speed, which
> is not fair on the other mirrors.
>
> So on my own mirror it runs every ten minutes, and shows ludicrous speed
> for itself -
>
> https://sledjhamr.org/apt-panopticon/results/Report-web.html
>
> On a small server that rrq kindly let me use, it runs once an hour,
> mostly coz it's a single core VM, running it more often will bog down his
> server -
>
> https://borta.devuan.dev/apt-panopticon/results/Report-web.html
>
> Could all mirror admins PLEASE PLEASE PLEASE actually monitor one of
> those, and actually correct any issues for your server, or at least talk
> to me about it if you think the script is wrong. I'm kinda disappointed
> that there has been almost no positive changes since I started telling
> you all about this. There is way too many warnings and errors. If you
> hover your mouse over the column titles, a short explanation of what is
> being tested, and why will pop up. Remember, this is now the official
> indication of how healthy our mirror system is.
>
> --
> A big old stinking pile of genius that no one wants
> coz there are too many silver coated monkeys in the world.
> _______________________________________________
> devuan-mirrors mailing list
> devuan-mirrors@???
> https://mailinglists.dyne.org/cgi-bin/mailman/listinfo/devuan-mirrors
>