-- snip --

> > Using Agent Headers or robots.txt is pointless as the aggressor cannot be trusted to be honest. I am currently looking for some form of proactive\reactive dynamic firewall blocking for this type of 'attack'.
> > I am also investigating if I can tarpit (without causing myself significant overhead) to actually slow down probes onto my network by causing heightened resource usage at the aggressor's end.
> >

> > If you know of anything that may help please give me a head-up.
>

>

> One of my ideas is to delay the 404 report. Normal people probably wont
> even notice if the 404 report took a few seconds. It may slow down the
> bots, but maybe not.

I am aware of the tcp window size setting used during http\https but do not know enough about it to determine detrimental side effects. Was thinking a window size of one with a one second ACK delay. Obviously the connection socket would be 'stuck' open and there is a danger the host could run out of free sockets, other than that I dont have a clue.