http://deb.devuan.org/robots.txt also exists now. http://deb.devuan.org/
should return that index.html file mentioned below.

Those two things plus a few firewall rules reduced things to "lost in the
noise". The firewall rules block things claiming to be Claude and semrush.

I started adding checking for those two files to apt-panopticon. Then
decided that it needed a bit of a rewrite. lol

So for the last week you may have noticed every now and then something
with a user agent resembling "apt-panopticon version 0.0 rewrite, using
LuaSocket 3.0.0" trying to download some pages. Eventually it'll replace
the current apt-panopticon instances.

On 2026-02-10 13:49:16, onefang wrote:
> http://deb.devuan.org/index.html now exists, please serve that file and
> not some bespoke custom version, or an autogenerated index, when you
> serve deb.devuan.org and it's CC friends.
>
> Thanks.
>
> On 2026-02-10 04:10:49, onefang wrote:
> > What people do with their own domains is their own business, but
> > deb.devuan.org is the Devuan teams business, it's our domain. The Devuan
> > team decides what goes on http://deb.devuan.org/ The same applies to the
> > CC.deb.devuan.org domains.
> >
> > So all Devuan package mirrors should be serving the exact same thing when
> > responding to http://deb.devuan.org/ and the country code versions.
> >
> > We have https://pkginfo.devuan.org/ for searching for packages, so we
> > don't really need the search bots scanning us. The AI bots found non
> > Devuan things on some of our mirrors at http://deb.devuan.org/, hence why
> > they keep scanning http://deb.devuan.org/ for non Devuan package repos.
> >
> > So we decided to just stop the bots from scanning http://deb.devuan.org/
> > which is why there is now a http://deb.devuan.org/robots.txt that tells
> > all the bots that bother to obey it not to scan http://deb.devuan.org/
> >
> > For those bots that ignore robots.txt there will soon be a
> > http://deb.devuan.org/index.html file. This will include links to the
> > package repo stuff and no other links. "http://deb.devuan.org/" should
> > serve this index.html.
> >
> > In this way the behaving bots just wont scan anything, and the
> > misbehaving bots will only find Devuan package repo, nothing else, when
> > they scan http://deb.devuan.org/
> >
> > Hopefully within a week the bots will have forgotten the non Devuan stuff
> > they have been scanning http://deb.devuan.org/ for, and they wont be
> > bogging down our mirrors and filling our logs with 404s. I'm certainly
> > tired of the same non existent files being scanned for 7000 times a day.
> >
> > This should also help with the bots scanning Devuan package mirrors for
> > non existant packages, that are not at deb.devuan.org, but that can still
> > leak a bit.
> >
> > Cross our fingers, hope this helps.