|:: Re: [devuan-mirrors] [devuan-dev] I…|
|This message is part of the following thread:|
|the complete thread tree sorted by date|
|Matthias Kruk at|
> Dear onefang, > thank you for apt-panopticon, and especially for running it frequently on > all mirrors. It has already proven to be very useful to me, and I like the > near-realtime reporting. I'm trying to establish a habit of checking the > report somewhat regularly so I can tell in a timely manner if something is > wrong with my mirror.
> The reason I'm replying here is that there were actually a bunch of > warnings that were displayed for my mirror, which I have finally all > fixed.
> The better part of the warnings were about the debian redirect for > partial mirrors, and in particular the way it is set up with lighttpd. I > checked the list and noticed there is at least one more mirror suffering > from the same issue, so I'd like to share how to solve it here: > The redirects in my mirror's lighttpd configuration (and probably in the > other lighttpd-based mirrors as well, since my configuration was used for > the lighttpd example) are set up like this: > url.redirect = ( > "^/merged/pool/DEVUAN/(.*)" => "/devuan/pool/$1", > "^/merged/pool/DEBIAN/(.*)" => > "https://deb.debian.org/debian/pool/$1", > "^/merged/pool/DEBIAN-SECURITY/(.*)" => > "https://deb.debian.org/debian-security/pool/$1" > ) > The script, in its URL sanitation check sends a request with multiple > slashes, which I couldn't reproduce on my side at first, since my client > was smart/sneaky enough to remove superfluous slashes before issuing the > request to the server. Using the same command as your script, I realized > that lighttpd chokes on the multiple slashes because it performs > redirection *before* sanitizing the URLs. I'm not sure why lighttpd would > do it in this order, since that means the regexes will have to account for > all sorts of perfectly valid garbage (like > "/.//.///.////./foo//.///../index.html"). Anyways, I changed the redirects > to > url.redirect = ( > "^[/]+merged[/]+pool[/]+DEVUAN[/]+(.*)" => "/devuan/pool/$1", > "^[/]+merged[/]+pool[/]+DEBIAN[/]+(.*)" => > "https://deb.debian.org/debian/pool/$1", > "^[/]+merged[/]+pool[/]+DEBIAN-SECURITY[/]+(.*)" => > "https://deb.debian.org/debian-security/pool/$1" > ) > which means that it now works for your test cases, but it would still > choke on funky requests like the one mentioned above which is why I > consider this more of a workaround than a solution. I'm trying to see if > theres an easy way around it, otherwise I might be willing to give nginx a > try instead. In the meanwhile, other lighttpd-based mirrors (at a glimpse, > at least mirror.devuan.de seems affected) might want to adopt the same > redirects.
> The other issue that I've had with my mirror was that it ran out of disk > space - it was originally only 40GB which I upgraded to 100GB a couple of > months ago. I just upgraded to 200GB today and I'm hoping that should > suffice for now. Is there any indication for an upper limit on the disk > usage? I think debian is hosting five distributions at a time > (oldoldstable, oldstable, stable, testing, unstable) and we're at four > (oldstable thru unstable) at the moment, so my guess would be at least a > 25% increase? The reason I'm asking is because I'm considering to add more > storage and upgrade to a complete mirror, without the debian redirect. > Could you shed some light on the storage situation on > pkgmaster.devuan.org, maybe some storage recommendations for a partial > and a full mirror? I realize there are several factors this depends upon, > so a conservative guess would be fine. :-)
> Thank you! > Best regards, > Matthias > 2020年3月7日(土) 0:40 onefang <onefang_devuan@???>:
> Alas life got in the way and has delayed all my projects. So I'm > sending > this email much later than I had planned.
> apt-panopticon is basically ready for an alpha release.
> With the blessing of the other Devuan devs (at least no one complained > at > the time, or since) Evilham and I have decided to use the results of > apt-panopticon to decide which mirror servers are on the DNS-RR and > which > are not. https://pkgmaster.devuan.org/mirror_list.txt will be > updated > for things like "certificate expired, so they no longer support HTTPS", > "have been doing nothing but timeouts for sometime, so mark as offline", > etc. The DNS-RR will be modified to reflect serious issues.
> Evilham wants me to go one step further, and automate things. His idea > was to run apt-panopticon once a minute, and report the results to a > backend that will automatically add or remove mirror servers to DNS-RR > depending on yet to be discussed pass conditions. On the fastest server > it is currently running on the average run time is slightly less than > one > minute, so I think maybe doing this every five minutes. The goal is > that > when someone uses the DNS-RR, they get as perfect an experience as we > can > manage.
> Pass conditions are likely to be something like - passes all tests with > no warnings (except HTTPS that DNS-RR can't handle anyway), 100% up > time, > 100% updated. Very few currently meet those conditions.
> Apt-panopticon is running on two different servers now, plus my desktop, > and I'm awaiting a friend to get IPv6 for her server before I run it > there. Mostly coz running it on an actual mirror server means that > particular mirror gets disk speed tested instead of network speed, which > is not fair on the other mirrors.
> So on my own mirror it runs every ten minutes, and shows ludicrous speed > for itself -
> On a small server that rrq kindly let me use, it runs once an hour, > mostly coz it's a single core VM, running it more often will bog down > his > server -
> Could all mirror admins PLEASE PLEASE PLEASE actually monitor one of > those, and actually correct any issues for your server, or at least talk > to me about it if you think the script is wrong. I'm kinda disappointed > that there has been almost no positive changes since I started telling > you all about this. There is way too many warnings and errors. If you > hover your mouse over the column titles, a short explanation of what is > being tested, and why will pop up. Remember, this is now the official > indication of how healthy our mirror system is.
> -- > A big old stinking pile of genius that no one wants > coz there are too many silver coated monkeys in the world. > _______________________________________________ > devuan-mirrors mailing list > devuan-mirrors@??? > https://mailinglists.dyne.org/cgi-bin/mailman/listinfo/devuan-mirrors
> Visible links > 1. https://deb.debian.org/debian/pool/$1 > 2. https://deb.debian.org/debian-security/pool/$1 > 3. https://deb.debian.org/debian/pool/$1 > 4. https://deb.debian.org/debian-security/pool/$1 > 5. http://mirror.devuan.de/ > 6. http://pkgmaster.devuan.org/ > 7. mailto:email@example.com > 8. https://pkgmaster.devuan.org/mirror_list.txt > 9. https://sledjhamr.org/apt-panopticon/results/Report-web.html > 10. https://borta.devuan.dev/apt-panopticon/results/Report-web.html > 11. mailto:firstname.lastname@example.org > 12. https://mailinglists.dyne.org/cgi-bin/mailman/listinfo/devuan-mirrors
|This message was posted to the following mailing lists:|
Mailing List Info | Nearby Messages
|Re: [devuan-mirrors] [devuan-dev] Introducing apt-panopticon, my Devuan mirror checker script.||[devuan-dev] bug#406: apt-2.0.0: Runs very, very slow after update to version 2.0|
Mailing List Info | Nearby Messages
|Re: [devuan-mirrors] [devuan-dev] Introducing apt-panopticon, my Devuan mirror checker script.|
|dyne.org open discussions |
administrated by dyne.org hackers
|Lurker (version 2.3)|