onefang <onefang_devuan@???> wrote:

> A total of 6126 sites probed the server 
>    1.122.146.15
>    1.141.148.168
>    1.145.123.157

>
> Then lists the other 6123 IPs. After the thousands of IPs, it has -
>
>

> Requests with error response codes
>    400 Bad Request
>       /: 22 Time(s)
>       null: 22 Time(s)
>       *: 4 Time(s)
>       ../../proc/: 4 Time(s)
>    403 Forbidden
>       /drupal/?q=node/add: 71 Time(s)
>       /drupal/user/2981/xmlrpc.php: 3 Time(s)
>       /drupal/user/2981/edit/xmlrpc.php: 2 Time(s)
>       /drupal/search/node/Popular%2Bbroadband%2B ... Bconfigurations: 1 Time(s)
>    404 Not Found
>       /merged/dists/ascii/InRelease: 1795 Time(s)

>
> And thousands of others in other categories.

I’m personally not sure if much, or even any, of this is really important. If you run a server that’s internet accessible, assume every man and his dog will be trying to break in for nefarious purposes.
When we moved a few years ago, I knew we’d be without internet for a week or two. So I asked my brother to host my mail server for a while till I got sorted out. It wasn’t long before he was asking WTF my server was doing as it was triggering alerts in his router firewall which had conservative restrictions on connections. I had to explain that it wasn’t what my server was doing, it was the hordes of criminals out there attacking it that were causing his alerts.
With a previous work hat on I worked for a small IT services company. For our small hosting facility, I had to set up region based blocklists at the border gateway. “Certain countries” were particularly active in attacking our servers, so it was simplest to just block them entirely. Worst were the attacks on our IP PBX - we could see Mbps of SIP registration requests from single IPs trying to brute-force accounts.

And there’s only one way to avoid such attacks - don’t host anything ! Unless “just don’t” is a viable/acceptable option, you just have to accept that you need to handle ****s.


> I guess the message that ASCII got moved to the archive server
> hasn't gotten through to a lot of people that are still running ASCII.

Doesn’t matter if there is only one person (or even no-one) in the world who hasn’t got the message - the bad guys simply try all potential options against everyone in the hope that they’ll find something they can abuse.


> There will also be lots of errors for URLs that I don't have, looking
> like people are trying to crack my server.

Yes, the bad guys simply try anything that might be on a web server knowing that only some very very small percentage of those attempts will succeed. It’s simply a matter of scale - it’s “cheap” to hit a web server with all known possibilities, especially when they are using someone else’s resources (i.e. a compromised computer), so why bother narrowing things down at all ?



> Would be nice if fail2ban didn't fail2ban, the only rule that actually works is the one I wrote myself. Any one got clue? I could use some.

Can you clarify what you mean - I’m guessing that it’s failing to ban stuff ?
The most obvious gotcha is that all the built in rules are disabled by default, and typically you’d want to customise the defaults (e.g. error counts, ban times) anyway.
But another gotcha is that IIRC adding an IP to the blocklist doesn’t stop already open connections.

So an attacker that opens a connection, then tries hundreds of hacks using that one connection won’t get stopped. For that to work, you need to edit what it does (same with any firewall) - but it comes with a performance penalty. Typical firewall rules tend to have “accept packets for already open connections” as the first rule. This means that the mass of packets that make up a typical session need very little by way of processing in the firewall rules, hence little load and packet delay.
Again from memory, fail2ban installs it’s rules in the “when opening a new connection, do this ...” chain. The performance gain from running the rules against just the new connections vs every packet of all connections is significant. But the downside is that the rules only stop new connections.
The alternative way around the problem is to add the IP to the blocklist, then actively remove it from the conntrack (connection tracking) table or otherwise killing the active connection.
Another alternative is to use the service itself. E.g. Postfix has an option to set the maximum number of errors it will accept before it shuts down a connection. So instead of an attacker being able to open a connection and try hundreds, thousands, of potential email addresses - by setting a limit, Postfix will close the connection, and when the attacker resumes by opening a new connection, fail2ban will have done it’s job.


Another trick I’ve used is to employ a second ban. Fail2ban can be “irritating” when you ban yourself - so ideally you don’t want it to be too severe. Many attackers will simply move on to their next target once banned, but some will keep coming back for days or weeks. It’s possible to write a rule that simply says “if this IP has been banned 3 times in the last 3 hours, then ban it for a day”. Accidentally triggering a ban yourself won’t lock you out for long, but the persistent attacker will only get a few goes/day. You can also use such a rule to block all traffic from an IP, not just for one service if you want to - though in my experience it’s different attackers working on different services (i.e. someone trying to hack your mail server for sending spam, probably isn’t also attacking your web server; and someone attacking your web server to host phishing resources to direct people to, probably isn’t attacking your mail server).


> I'm considering switching to nginx some day, though not sure if that’ll help with this.

Not really, you’ll just have different logs for the same attacks.


Simon