Online list unusable due to bot traffic

[mabley]

Looking for context or advice.

Starting about 6 months ago, our Guests filter for the Users Online list went from a consistent 20 pages of guest IPs to regularly 100+, and now more often 500+ pages for a 24 hour period. That's upwards of 25,000 unique IPs hitting our site every day.

I do a few things to try and mitigate unwanted traffic:
- Reviewed and updated our robots.txt (ha ha)
- Catch spam account registrations with a module (that's working great)
- Loading IPs from obvious crawlers/scrapers/bots into bans

None of this seems to make a dent. Only one of our boards allow guest posting and we honestly get only one or two spam posts a month, so that's fine. I'm just so weirded out by the massive escalation in bots that I can't make a dent in.

(Not sure where to put this post; please move if needed.)

[Kindred]

There's a few threads already discussing the influx of bots.

Short of it -- add a large bot list to your blocked set in htaccess

[shawnb61]

You might want to use this as a starting point:
https://github.com/sbulen/SMF-bot-hygiene

[mabley]

Much appreciated - I'll pursue these leads