Looking for context or advice.
Starting about 6 months ago, our Guests filter for the Users Online list went from a consistent 20 pages of guest IPs to regularly 100+, and now more often 500+ pages for a 24 hour period. That's upwards of 25,000 unique IPs hitting our site every day.
I do a few things to try and mitigate unwanted traffic:
- Reviewed and updated our robots.txt (ha ha)
- Catch spam account registrations with a module (that's working great)
- Loading IPs from obvious crawlers/scrapers/bots into bans
None of this seems to make a dent. Only one of our boards allow guest posting and we honestly get only one or two spam posts a month, so that's fine. I'm just so weirded out by the massive escalation in bots that I can't make a dent in.
(Not sure where to put this post; please move if needed.)
There's a few threads already discussing the influx of bots.
Short of it -- add a large bot list to your blocked set in htaccess
You might want to use this as a starting point:
https://github.com/sbulen/SMF-bot-hygiene
Much appreciated - I'll pursue these leads