On 2025-11-28 08:54:43, David Niklas via Dng wrote:
> On Thu, 27 Nov 2025 12:05:28 +0000
> g4sra <g4sra@???> wrote:
> > > ClaudeBot is currently doing something really stupid. It somehow has
> > > come to the conclusion that my Devuan mirror mirrors EVERYTHING!
> >
> > Welcome to the AI future.
> > anthropic.com have been having bad press since last year and don't
> > appear to care how much upset they cause.
> >
> > Using Agent Headers or robots.txt is pointless as the aggressor cannot
> > be trusted to be honest. I am currently looking for some form of
> > proactive\reactive dynamic firewall blocking for this type of 'attack'.
> > I am also investigating if I can tarpit (without causing myself
> > significant overhead) to actually slow down probes onto my network by
> > causing heightened resource usage at the aggressor's end.
> >
> > If you know of anything that may help please give me a head-up.
>
> The easiest thing, and I have seen other OSS admins do this before AI
> came along, is to make some of these non-existant paths trigger a
> blacklist for that IP. Then no matter what it requests the AI just gets a
> response like, "You've been blacklisted. If you think you've been blocked
> in error please reach out to ..."

As I mentioned, an arms race. Some of the AI bots are resorting to DDoS
type tactics, using an army of IPs and otherwise trying to hide their
activities from blockers.

> As for why 404 is not respected, some hosts report this when the file
> actually exists. So you can't trust 404 on some hosts. The web is broken
> and AI is just better exposing it.

Asking me for the same files thousands of times per day, and not getting
the files, how is that any sort of sane response to broken 404s? Probing
around in other directories looking for these and other files I don't
have from other things I don't mirror. Artificial Insanity.