Forum getting swarmed by 500+ guests?

[Rhindeer]

Sooo, I'm having a strange issue in that my forum (http://spiritsoftheearth.net/smf) is getting swarmed by guests. No registrations, as I have a spam mod installed. However, I was able to trace them back to Amazon.

I tried banning a range of them, but that ended up banning some actual users. So I undid that.

Anyway, the issue is that it's causing errors because it's maxing out the "max_questions" resource.

Is there a way to resolve this? I'm not actually sure what to do since my host has already raised "max_questions" to the limit.

I attached an image showing some of the guests as an example. As I send this there are 520 guests. Ahhh!You cannot view this attachment.

[shawnb61]

I assume you mean "max_connections".

I'm seeing the same thing - and for me, there has been an explosion of new bots recently.  You need to find a good way to identify & block those bots.  In some cases, existing bots are just not honoring crawl-delays. 

I've been periodically reviewing the actual web access logs to identify the bots, and adding entries to .htaccess to block what I find.

It's been kind of like whack-a-mole...  Every time I think I have it covered, a new bot shows up.  Some of them unabashedly AI related. 

One recent surprise is GoogleOther.  This fairly new bot has been hammering my site - and with very strange hackish behavior (many many thousands of attempts at verification codes?!?!?).  And yes, I've confirmed they are Google IPs.  Not honoring robots.txt at all....
https://thriveagency.com/news/google-introduces-new-googlebot-web-crawler-named-googleother-what-you-need-to-know/

WTF Google...

Anyway @vbgamer45 generously shared a sample .htaccess here:
https://www.simplemachines.org/community/index.php?msg=4170375

Other helpful reading:
https://www.imperva.com/blog/most-active-good-bots/
https://radar.cloudflare.com/traffic/verified-bots
https://github.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/blob/master/_generator_lists/bad-user-agents.list

Hope this helps.

[Rhindeer]

I assume you mean "max_connections".

I'm seeing the same thing - and for me, there has been an explosion of new bots recently.  You need to find a good way to identify & block those bots.  In some cases, existing bots are just not honoring crawl-delays. 

I've been periodically reviewing the actual web access logs to identify the bots, and adding entries to .htaccess to block what I find.

It's been kind of like whack-a-mole...  Every time I think I have it covered, a new bot shows up.  Some of them unabashedly AI related. 

One recent surprise is GoogleOther.  This fairly new bot has been hammering my site - and with very strange hackish behavior (many many thousands of attempts at verification codes?!?!?).  And yes, I've confirmed they are Google IPs.  Not honoring robots.txt at all....
https://thriveagency.com/news/google-introduces-new-googlebot-web-crawler-named-googleother-what-you-need-to-know/

WTF Google...

Anyway @vbgamer45 generously shared a sample .htaccess here:
https://www.simplemachines.org/community/index.php?msg=4170375

Other helpful reading:
https://www.imperva.com/blog/most-active-good-bots/
https://radar.cloudflare.com/traffic/verified-bots
https://github.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/blob/master/_generator_lists/bad-user-agents.list

Hope this helps.

Thank you so much for this! This really does help! Though I admit I've never edited the htaccess file before. The code that was linked--where would I put it in this file? It's my current htaccess file. I apologize, I'm a n00b when it comes to a lot of this. D:

Code Select Expand

<IfModule mod_rewrite.c>
RewriteEngine On
RewriteBase /
RewriteCond %{REQUEST_FILENAME} !-f
RewriteCond %{REQUEST_FILENAME} !-d
RewriteRule . /index.php [L]
</IfModule>
Also strangely enough it wasn't "max_connections", it was "max_questions"! I've had the max_connections error in this past, but this one keeps popping up as max_questions.

But oof! Glad to know I'm not alone in this! Well, kinda--I'm sorry everyone else has to deal with the bot plight too. Dx The hackish behavior is disturbing. So far I haven't seen any of that, thank goodness.

[Aleksi "Lex" Kilpinen]

Unrelated discussion about Who's Online and guests split to it's own topic Who's Online (Was: re: Forum getting swarmed by 500+ guests?)

[shawnb61]

Thank you so much for this! This really does help! Though I admit I've never edited the htaccess file before. The code that was linked--where would I put it in this file? It's my current htaccess file. I apologize, I'm a n00b when it comes to a lot of this. D:

Code Select Expand

Code Select Expand

<IfModule mod_rewrite.c>
RewriteEngine On
RewriteBase /
RewriteCond %{REQUEST_FILENAME} !-f
RewriteCond %{REQUEST_FILENAME} !-d
RewriteRule . /index.php [L]
</IfModule>

I would place the new code below the block displayed above.  Then TEST...  First off, make sure you haven't cut yourself off!  (Typos can really mess things up...)  You should still be able to access the site as normal.

I test using Chrome, because it has a nice little interface for updating your user-agent for testing purposes.  In Chrome, open the dev console.  Open the "Network Conditions" tab.  In the User agent section, if you uncheck the "Use browser default" box, then specify a "Custom" user agent, you can type anything in there.  Type each bad guy, navigate to your site, and you should be forbidden from accessing the site.
https://developer.chrome.com/docs/devtools/device-mode/override-user-agent

(When done testing, change it back to the browser default if you actually use Chrome...)

Also strangely enough it wasn't "max_connections", it was "max_questions"! I've had the max_connections error in this past, but this one keeps popping up as max_questions.

Interesting.  Apparently "max_questions" is in fact a standard per-hour limit.  Learn something new every day.
https://dev.mysql.com/doc/refman/8.0/en/user-resources.html

[Rhindeer]

Thank you so much for this! This really does help! Though I admit I've never edited the htaccess file before. The code that was linked--where would I put it in this file? It's my current htaccess file. I apologize, I'm a n00b when it comes to a lot of this. D:

Code Select Expand

Code Select Expand

<IfModule mod_rewrite.c>
RewriteEngine On
RewriteBase /
RewriteCond %{REQUEST_FILENAME} !-f
RewriteCond %{REQUEST_FILENAME} !-d
RewriteRule . /index.php [L]
</IfModule>

I would place the new code below the block displayed above.  Then TEST...  First off, make sure you haven't cut yourself off!  (Typos can really mess things up...)  You should still be able to access the site as normal.

I just tried putting it in my htaccess file (pasted it directly under the above code) and it gave me a 500 internal error, so I removed it and it's back to normal. D: So not quite sure how to edit this file, ahh!

For reference I was using this suggested code:

Code Select Expand

<Location />
<Limit GET POST PUT>

# Begin Bad Bot Blocking
BrowserMatchNoCase OmniExplorer_Bot/6.11.1 bad_bot
BrowserMatchNoCase omniexplorer_bot bad_bot
BrowserMatchNoCase Baiduspider bad_bot
BrowserMatchNoCase Baiduspider/2.0 bad_bot
BrowserMatchNoCase yandex bad_bot
BrowserMatchNoCase yandeximages bad_bot
BrowserMatchNoCase Spinn3r bad_bot
BrowserMatchNoCase sogou bad_bot
BrowserMatchNoCase Sogouwebspider/3.0 bad_bot
BrowserMatchNoCase Sogouwebspider/4.0 bad_bot
BrowserMatchNoCase sosospider+ bad_bot
BrowserMatchNoCase jikespider bad_bot
BrowserMatchNoCase ia_archiver bad_bot
BrowserMatchNoCase PaperLiBot bad_bot
BrowserMatchNoCase ahrefsbot bad_bot
BrowserMatchNoCase ahrefsbot/1.0 bad_bot
BrowserMatchNoCase SiteBot/0.1 bad_bot
BrowserMatchNoCase DNS-Digger/1.0 bad_bot
BrowserMatchNoCase DNS-Digger-Explorer/1.0 bad_bot
BrowserMatchNoCase boardreader bad_bot
BrowserMatchNoCase radian6 bad_bot
BrowserMatchNoCase R6_FeedFetcher bad_bot
BrowserMatchNoCase R6_CommentReader bad_bot
BrowserMatchNoCase ScoutJet bad_bot
BrowserMatchNoCase ezooms bad_bot
BrowserMatchNoCase CC-rget/5.818 bad_bot
BrowserMatchNoCase libwww-perl/5.813 bad_bot
BrowserMatchNoCase magpie-crawler 1.1 bad_bot
BrowserMatchNoCase jakarta bad_bot
BrowserMatchNoCase discobot/1.0 bad_bot
BrowserMatchNoCase MJ12bot bad_bot
BrowserMatchNoCase MJ12bot/v1.2.0 bad_bot
BrowserMatchNoCase MJ12bot/v1.2.5 bad_bot
BrowserMatchNoCase SemrushBot/0.9 bad_bot
BrowserMatchNoCase MLBot bad_bot
BrowserMatchNoCase butterfly bad_bot
BrowserMatchNoCase SeznamBot/3.0 bad_bot
BrowserMatchNoCase HuaweiSymantecSpider bad_bot
BrowserMatchNoCase Exabot/2.0 bad_bot
BrowserMatchNoCase netseer/0.1 bad_bot
BrowserMatchNoCase NetSeer crawler/2.0 bad_bot
BrowserMatchNoCase NetSeer/Nutch-0.9 bad_bot
BrowserMatchNoCase psbot/0.1 bad_bot
BrowserMatchNoCase moreoverbot/5.0 bad_bot
BrowserMatchNoCase Jakarta Commons-HttpClient/3.0 bad_bot
BrowserMatchNoCase SocialSpider-Finder/0.2 bad_bot
BrowserMatchNoCase wordpress bad_bot
BrowserMatchNoCase istellabot bad_bot
BrowserMatchNoCase SeznamBot bad_bot
BrowserMatchNoCase Cliqzbot bad_bot
BrowserMatchNoCase SocialRankIOBot bad_bot
BrowserMatchNoCase Mail.RU_Bot bad_bot
BrowserMatchNoCase Clickag Bot bad_bot
BrowserMatchNoCase Mediatoolkitbot bad_bot
BrowserMatchNoCase SemrushBot bad_bot
BrowserMatchNoCase DotBot/1.1 bad_bot
BrowserMatchNoCase DataForSeoBot bad_bot
BrowserMatchNoCase www.timpi.io bad_bot
BrowserMatchNoCase DotBot bad_bot
BrowserMatchNoCase trendictionbot bad_bot
BrowserMatchNoCase BLEXBot/1.0 bad_bot
BrowserMatchNoCase SeekportBot bad_bot
BrowserMatchNoCase Turnitin bad_bot
BrowserMatchNoCase omgili/0.5 bad_bot
BrowserMatchNoCase CheckHost bad_bot
BrowserMatchNoCase Amazonbot bad_bot
BrowserMatchNoCase SEOkicks bad_bot
<RequireAll>
Require all granted
<RequireNone>
Require env bad_bot
</RequireNone>
</RequireAll>

</Limit>
</Location>


[shawnb61]

There has been some changes lately, e.g., the new recommended syntax is 'require', like @vbgamer45 uses.

I still use the old syntax.  So I took your list, added a few agents that have been bugging me lately (including GoogleOther and claudebot), and utilized the old syntax.  I also enclosed bots with embedded spaces in single quotes.

Try this:

Code Select Expand

# Begin Bad Bot Blocking
BrowserMatchNoCase OmniExplorer_Bot/6.11.1 bad_bot
BrowserMatchNoCase omniexplorer_bot bad_bot
BrowserMatchNoCase Baiduspider bad_bot
BrowserMatchNoCase Baiduspider/2.0 bad_bot
BrowserMatchNoCase yandex bad_bot
BrowserMatchNoCase yandeximages bad_bot
BrowserMatchNoCase Spinn3r bad_bot
BrowserMatchNoCase sogou bad_bot
BrowserMatchNoCase Sogouwebspider/3.0 bad_bot
BrowserMatchNoCase Sogouwebspider/4.0 bad_bot
BrowserMatchNoCase sosospider+ bad_bot
BrowserMatchNoCase jikespider bad_bot
BrowserMatchNoCase ia_archiver bad_bot
BrowserMatchNoCase PaperLiBot bad_bot
BrowserMatchNoCase ahrefsbot bad_bot
BrowserMatchNoCase ahrefsbot/1.0 bad_bot
BrowserMatchNoCase SiteBot/0.1 bad_bot
BrowserMatchNoCase DNS-Digger/1.0 bad_bot
BrowserMatchNoCase DNS-Digger-Explorer/1.0 bad_bot
BrowserMatchNoCase boardreader bad_bot
BrowserMatchNoCase radian6 bad_bot
BrowserMatchNoCase R6_FeedFetcher bad_bot
BrowserMatchNoCase R6_CommentReader bad_bot
BrowserMatchNoCase ScoutJet bad_bot
BrowserMatchNoCase ezooms bad_bot
BrowserMatchNoCase CC-rget/5.818 bad_bot
BrowserMatchNoCase libwww-perl/5.813 bad_bot
BrowserMatchNoCase 'magpie-crawler 1.1' bad_bot
BrowserMatchNoCase jakarta bad_bot
BrowserMatchNoCase discobot/1.0 bad_bot
BrowserMatchNoCase MJ12bot bad_bot
BrowserMatchNoCase MJ12bot/v1.2.0 bad_bot
BrowserMatchNoCase MJ12bot/v1.2.5 bad_bot
BrowserMatchNoCase SemrushBot/0.9 bad_bot
BrowserMatchNoCase MLBot bad_bot
BrowserMatchNoCase butterfly bad_bot
BrowserMatchNoCase SeznamBot/3.0 bad_bot
BrowserMatchNoCase HuaweiSymantecSpider bad_bot
BrowserMatchNoCase Exabot/2.0 bad_bot
BrowserMatchNoCase netseer/0.1 bad_bot
BrowserMatchNoCase 'NetSeer crawler/2.0' bad_bot
BrowserMatchNoCase NetSeer/Nutch-0.9 bad_bot
BrowserMatchNoCase psbot/0.1 bad_bot
BrowserMatchNoCase moreoverbot/5.0 bad_bot
BrowserMatchNoCase 'Jakarta Commons-HttpClient/3.0' bad_bot
BrowserMatchNoCase SocialSpider-Finder/0.2 bad_bot
BrowserMatchNoCase wordpress bad_bot
BrowserMatchNoCase istellabot bad_bot
BrowserMatchNoCase SeznamBot bad_bot
BrowserMatchNoCase Cliqzbot bad_bot
BrowserMatchNoCase SocialRankIOBot bad_bot
BrowserMatchNoCase Mail.RU_Bot bad_bot
BrowserMatchNoCase 'Clickag Bot' bad_bot
BrowserMatchNoCase Mediatoolkitbot bad_bot
BrowserMatchNoCase SemrushBot bad_bot
BrowserMatchNoCase DotBot/1.1 bad_bot
BrowserMatchNoCase DataForSeoBot bad_bot
BrowserMatchNoCase www.timpi.io bad_bot
BrowserMatchNoCase DotBot bad_bot
BrowserMatchNoCase trendictionbot bad_bot
BrowserMatchNoCase BLEXBot/1.0 bad_bot
BrowserMatchNoCase SeekportBot bad_bot
BrowserMatchNoCase Turnitin bad_bot
BrowserMatchNoCase omgili/0.5 bad_bot
BrowserMatchNoCase CheckHost bad_bot
BrowserMatchNoCase Amazonbot bad_bot
BrowserMatchNoCase SEOkicks bad_bot
BrowserMatchNoCase Claudebot bad_bot
BrowserMatchNoCase bomborabot bad_bot
BrowserMatchNoCase commoncrawl bad_bot
BrowserMatchNoCase dataforseo-bot bad_bot
BrowserMatchNoCase GoogleOther bad_bot
BrowserMatchNoCase keys-so-bot bad_bot
BrowserMatchNoCase MojeekBot bad_bot
<Limit GET POST HEAD>
Order Allow,Deny
Allow from all
Deny from env=bad_bot
</Limit>

[Rhindeer]

There has been some changes lately, e.g., the new recommended syntax is 'require', like @vbgamer45 uses.

I still use the old syntax.  So I took your list, added a few agents that have been bugging me lately (including GoogleOther and claudebot), and utilized the old syntax.  I also enclosed bots with embedded spaces in single quotes.

Try this:

Code Select Expand

# Begin Bad Bot Blocking
BrowserMatchNoCase OmniExplorer_Bot/6.11.1 bad_bot
BrowserMatchNoCase omniexplorer_bot bad_bot
BrowserMatchNoCase Baiduspider bad_bot
BrowserMatchNoCase Baiduspider/2.0 bad_bot
BrowserMatchNoCase yandex bad_bot
BrowserMatchNoCase yandeximages bad_bot
BrowserMatchNoCase Spinn3r bad_bot
BrowserMatchNoCase sogou bad_bot
BrowserMatchNoCase Sogouwebspider/3.0 bad_bot
BrowserMatchNoCase Sogouwebspider/4.0 bad_bot
BrowserMatchNoCase sosospider+ bad_bot
BrowserMatchNoCase jikespider bad_bot
BrowserMatchNoCase ia_archiver bad_bot
BrowserMatchNoCase PaperLiBot bad_bot
BrowserMatchNoCase ahrefsbot bad_bot
BrowserMatchNoCase ahrefsbot/1.0 bad_bot
BrowserMatchNoCase SiteBot/0.1 bad_bot
BrowserMatchNoCase DNS-Digger/1.0 bad_bot
BrowserMatchNoCase DNS-Digger-Explorer/1.0 bad_bot
BrowserMatchNoCase boardreader bad_bot
BrowserMatchNoCase radian6 bad_bot
BrowserMatchNoCase R6_FeedFetcher bad_bot
BrowserMatchNoCase R6_CommentReader bad_bot
BrowserMatchNoCase ScoutJet bad_bot
BrowserMatchNoCase ezooms bad_bot
BrowserMatchNoCase CC-rget/5.818 bad_bot
BrowserMatchNoCase libwww-perl/5.813 bad_bot
BrowserMatchNoCase 'magpie-crawler 1.1' bad_bot
BrowserMatchNoCase jakarta bad_bot
BrowserMatchNoCase discobot/1.0 bad_bot
BrowserMatchNoCase MJ12bot bad_bot
BrowserMatchNoCase MJ12bot/v1.2.0 bad_bot
BrowserMatchNoCase MJ12bot/v1.2.5 bad_bot
BrowserMatchNoCase SemrushBot/0.9 bad_bot
BrowserMatchNoCase MLBot bad_bot
BrowserMatchNoCase butterfly bad_bot
BrowserMatchNoCase SeznamBot/3.0 bad_bot
BrowserMatchNoCase HuaweiSymantecSpider bad_bot
BrowserMatchNoCase Exabot/2.0 bad_bot
BrowserMatchNoCase netseer/0.1 bad_bot
BrowserMatchNoCase 'NetSeer crawler/2.0' bad_bot
BrowserMatchNoCase NetSeer/Nutch-0.9 bad_bot
BrowserMatchNoCase psbot/0.1 bad_bot
BrowserMatchNoCase moreoverbot/5.0 bad_bot
BrowserMatchNoCase 'Jakarta Commons-HttpClient/3.0' bad_bot
BrowserMatchNoCase SocialSpider-Finder/0.2 bad_bot
BrowserMatchNoCase wordpress bad_bot
BrowserMatchNoCase istellabot bad_bot
BrowserMatchNoCase SeznamBot bad_bot
BrowserMatchNoCase Cliqzbot bad_bot
BrowserMatchNoCase SocialRankIOBot bad_bot
BrowserMatchNoCase Mail.RU_Bot bad_bot
BrowserMatchNoCase 'Clickag Bot' bad_bot
BrowserMatchNoCase Mediatoolkitbot bad_bot
BrowserMatchNoCase SemrushBot bad_bot
BrowserMatchNoCase DotBot/1.1 bad_bot
BrowserMatchNoCase DataForSeoBot bad_bot
BrowserMatchNoCase www.timpi.io bad_bot
BrowserMatchNoCase DotBot bad_bot
BrowserMatchNoCase trendictionbot bad_bot
BrowserMatchNoCase BLEXBot/1.0 bad_bot
BrowserMatchNoCase SeekportBot bad_bot
BrowserMatchNoCase Turnitin bad_bot
BrowserMatchNoCase omgili/0.5 bad_bot
BrowserMatchNoCase CheckHost bad_bot
BrowserMatchNoCase Amazonbot bad_bot
BrowserMatchNoCase SEOkicks bad_bot
BrowserMatchNoCase Claudebot bad_bot
BrowserMatchNoCase bomborabot bad_bot
BrowserMatchNoCase commoncrawl bad_bot
BrowserMatchNoCase dataforseo-bot bad_bot
BrowserMatchNoCase GoogleOther bad_bot
BrowserMatchNoCase keys-so-bot bad_bot
BrowserMatchNoCase MojeekBot bad_bot
<Limit GET POST HEAD>
Order Allow,Deny
Allow from all
Deny from env=bad_bot
</Limit>

Thank you SO MUCH! The site didn't break and now gonna test everything in Chrome! I'll update ya! I really appreciate you!

[shawnb61]

Note that some in the above list are valid search engines. 

But most of us are running on some form of shared host with limited resources.  We just can't let everyone crawl...  You have to think of it as a budget.  Who can you afford to let in?

I have a crawl delay in my robots.txt.  I try to allow all legit search engine crawlers that honor the crawl delay to crawl.  I want folks to find my content - even if they are in Asia or Russia...

So for now I have allowed even yandex, baidu, sogou, bingbot, mj12bot, mail_ru.  I block all 'market research', 'seo research' and AI bots.  Literally anything I cannot find a search engine for.

I have a love/hate relationship with yandex & bingbot...  At various times I have had them blocked, because both can go off & crawl far too aggressively.  But they both appear to be following crawl-delay at the moment. 

Note also that Googlebot does NOT honor crawl-delay.  You need to use their search console to limit the rate.  On occasion, even Googlebot can hit you pretty hard, if they get it in their head you need a complete recrawl...

Just things to be aware of.  For now, I suggest you stay as restrictive as possible until you are stable. 

[Rhindeer]

@shawnb61 IT WORKED! Tested in Chrome and it was perfect, and also got to watch my guest list drop from 710 users to 17. Thank you so much! Myself and my community really appreciate you!

[Rhindeer]

Note that some in the above list are valid search engines. 

But most of us are running on some form of shared host with limited resources.  We just can't let everyone crawl...  You have to think of it as a budget.  Who can you afford to let in?

I have a crawl delay in my robots.txt.  I try to allow all legit search engine crawlers that honor the crawl delay to crawl.  I want folks to find my content - even if they are in Asia or Russia...

So for now I have allowed even yandex, baidu, sogou, bingbot, mj12bot, mail_ru. 

I have a love/hate relationship with yandex & bingbot...  At various times I have had them blocked, because both can go off & crawl far too aggressively.  But they both appear to be following crawl-delay at the moment. 

Note also that Googlebot does NOT honor crawl-delay.  You need to use their search console to limit the rate.  On occasion, even Googlebot can hit you pretty hard, if they get it in their head you need a complete recrawl...

Just things to be aware of.  For now, I suggest you stay as restrictive as possible until you are stable. 

Thank you for this, that makes total sense. I MAY in the future test run with allowing Google, Yandex, and Bing back since they've never given us issues (when they popped in we might get 20 extra guests, but never hundreds!) but for now I'm happy to keep everyone blocked. xD The Amazon bot was the worst one this round! I've never seen anything like this before.

[shawnb61]

To be clear - the above blocks this new bot called "GoogleOther", but this is not the normal search engine googlebot.  So Google search is allowed to crawl. 

GoogleOther is supposedly their research bot??? 

It does not honor crawl delay or any other robots.txt directive.  At all.  It has absolutely smashed my site. 

And get this - it was doing hundreds of 'action=verificationcode' calls per hour on my site...  From a valid Google IP. 

Blocked.

[Rhindeer]

To be clear - the above blocks this new bot called "GoogleOther", but this is not the normal search engine googlebot.  So Google search is allowed to crawl. 

GoogleOther is supposedly their research bot??? 

It does not honor crawl delay or any other robots.txt directive.  At all.  It has absolutely smashed my site. 

And get this - it was doing hundreds of 'action=verificationcode' calls per hour on my site...  From a valid Google IP. 

Blocked.

Ewwww wtf???

Yeah, no thanks, that one'll stay blocked!

[Steve]

It appears the OP's problem has been resolved. Marking solved.

[shawnb61]

I have a love/hate relationship with yandex & bingbot...  At various times I have had them blocked, because both can go off & crawl far too aggressively.  But they both appear to be following crawl-delay at the moment. 

For the record, yandex is not honoring crawl-delay anymore, so I blocked them again.

Oddly, they DO read robots.txt.  If they are disallowed there, they will honor that.  But not crawl-delay.

Their support site says they do not honor crawl-delay, and suggests you create an account with them to control crawls (like Google...)

[Rhindeer]

Hate to resurrect an old topic, but my forum's back to getting swarmed by 600+ guests, haha.

Any new bots I should know of to add to my list of bots to block? I can access the IP info, but not quite sure what name I should use to block the bots. For example, Alibaba is a bot that is wrecking us and their WHOis info is:

Code Select Expand

NetRange:       47.74.0.0 - 47.87.255.255
CIDR:           47.76.0.0/14, 47.80.0.0/13, 47.74.0.0/15
NetName:        AL-3
NetHandle:      NET-47-74-0-0-1
Parent:         NET47 (NET-47-0-0-0-0)
NetType:        Direct Allocation
OriginAS:       
Organization:   Alibaba Cloud LLC (AL-3)
RegDate:        2016-03-17
Updated:        2017-04-26
Ref:            https://rdap.arin.net/registry/ip/47.74.0.0

OrgName:        Alibaba Cloud LLC
OrgId:          AL-3
Address:        400 S El Camino Real, Suite 400
City:           San Mateo
StateProv:      CA
PostalCode:     94402
Country:        US
RegDate:        2010-10-29
Updated:        2024-11-25
Comment:        1.For AliCloud IPR Infringement and Abuse Claim, please use below link with browser to report: https://intl.aliyun.com/report
Comment:       
Comment:        2.For Alibaba.com and Aliexpress.com's IPR Infringement , please use below link with browser to report: https://ipp.alibabagroup.com
Comment:       
Comment:        3.For Alibaba.com and Aliexpress.com's Abuse, please send email to those two mail lists to report: [email protected] and [email protected]
Comment:       
Comment:        4. For network issue, please send email to this mail list: [email protected]
Ref:            https://rdap.arin.net/registry/entity/AL-3

OrgNOCHandle: ALIBA-ARIN
OrgNOCName:   Alibaba NOC
OrgNOCPhone:  +1-408-748-1200
OrgNOCEmail:  [email protected]
OrgNOCRef:    https://rdap.arin.net/registry/entity/ALIBA-ARIN

OrgAbuseHandle: NETWO4028-ARIN
OrgAbuseName:   Network Abuse
OrgAbusePhone:  +1-408-785-5580
OrgAbuseEmail:  [email protected]
OrgAbuseRef:    https://rdap.arin.net/registry/entity/NETWO4028-ARIN

OrgTechHandle: ALIBA-ARIN
OrgTechName:   Alibaba NOC
OrgTechPhone:  +1-408-748-1200
OrgTechEmail:  [email protected]
OrgTechRef:    https://rdap.arin.net/registry/entity/ALIBA-ARIN

#
# ARIN WHOIS data and services are subject to the Terms of Use
# available at: https://www.arin.net/resources/registry/whois/tou/
#
# If you see inaccuracies in the results, please report at
# https://www.arin.net/resources/registry/whois/inaccuracy_reporting/
#
# Copyright 1997-2025, American Registry for Internet Numbers, Ltd.
Which ID or handle would I use to block them in my htaccess?

[shawnb61]

Hate to resurrect an old topic, but my forum's back to getting swarmed by 600+ guests, haha.

Any new bots I should know of to add to my list of bots to block? I can access the IP info, but not quite sure what name I should use to block the bots. For example, Alibaba is a bot that is wrecking us and their WHOis info is:

Which ID or handle would I use to block them in my htaccess?

I've been sharing my updated list up here:
https://github.com/sbulen/SMF-bot-hygiene

I now include a BUNCH of IP ranges...

And yeah, those IPs got me as well, so they're in the list.

Good luck!

[Rhindeer]

Hate to resurrect an old topic, but my forum's back to getting swarmed by 600+ guests, haha.

Any new bots I should know of to add to my list of bots to block? I can access the IP info, but not quite sure what name I should use to block the bots. For example, Alibaba is a bot that is wrecking us and their WHOis info is:

Which ID or handle would I use to block them in my htaccess?

I've been sharing my updated list up here:
https://github.com/sbulen/SMF-bot-hygiene

And yeah, those IPs got me as well, so they're in the list.

Good luck!

You're amazing! Thank you so much!

[a10]

What about this one :O) (see attachment)
Was a zillion different ip's mainly from South America, impossible to block, no ranges, single ip's looking like a botnet using hacked private computers. Went away after a few days. btw no slowdown nor any other problem.

[shawnb61]

Usually you'll find a common ASN when they're coordinated like that.

Tools for identifying & blocking by ASN are shared here:
https://www.simplemachines.org/community/index.php?msg=4187833

[Rhindeer]

Currently a bunch of IPs from Huawei are going to town. xD Jeez, it's like you take care of one issue and a million others crop up! We got down to 100 guests yesterday after using the list, now it's back into the 500s. Ahhh!

[shawnb61]

Yep.  I analyze the logs about once a week...

[lather]

Thanks for this. I edited my htaccess file and guests dropped from 490 to 40.

[HITG]

Just for everyone interested a premium dns and crawlers delay code did the job. Also DDoS protection might be good for this but so far they are slowly coming in.

[Rhindeer]

It ramped up to the 1700s now. Lots of bots from Vietnam? I'm trying to figure out how to use the tools to squash these. Can't get Cloudflare, but may look into DNS? But dang they got aggressive.

[a10]

9337 guests  approx 90% South America + Vietnam and a few more asians. Individual ip's, no ranges to be spotted. Looks like hacked personal \ business computers \ devices. And among all that, ranges from amazon, huawei, alibaba etc.

No impact on general forum behaviour but have set it to login only for a while. Expecting things will calm down in a few days.

It's the new reality, ai \ scrapers. In earlier times, the plague was bot's registration attempts, they have basically disappeared on my forum.

[sudoku]

If nothing else, it makes your forum appear "busy" and lively. 
My forum was near 5000 just recently. It used to only have 100-150 guests at any given time - for years...

[Rhindeer]

9337 guests  approx 90% South America + Vietnam and a few more asians. Individual ip's, no ranges to be spotted. Looks like hacked personal \ business computers \ devices. And among all that, ranges from amazon, huawei, alibaba etc.

No impact on general forum behaviour but have set it to login only for a while. Expecting things will calm down in a few days.

It's the new reality, ai \ scrapers. In earlier times, the plague was bot's registration attempts, they have basically disappeared on my forum.

I set mine to login only and the numbers keep increasing still. D8 Maybe I did it wrong, what was your process?

They keep timing out my forum.

[a10]

I set mine to login only and the numbers keep increasing still. D8 Maybe I did it wrong, what was your process?

They keep timing out my forum.

Bots will still be there and trying but won't access anything except the welcome page. Stopping guests should\will\may hopefully restrict extreme use of bandwidth and server resources. Am on a shared server so host would maybe react, but no problems yet.

[shawnb61]

The problem is that if it hits your forum at all there is still a lot of work to do: connect to db & load settings; look for login cookie/session, maybe create a session; load the theme, maybe update stats, etc.

Under most circumstances, not a problem.  When you're flooded with activity, it's a problem.

That's why you have to filter them out BEFORE they even get to load a page, via .htaccess.

I haven't seen these guys (yet), so they're not added to my .htaccess starter pack:
https://github.com/sbulen/SMF-bot-hygiene

I would see which IPs/CIDRs are hitting you the most & add to .htaccess.

[a10]

If of interest, attached a sample from some minutes of activity this evening. This is very few of the different ip's in use.

[shawnb61]

If of interest, attached a sample from some minutes of activity this evening. This is very few of the different ip's in use.

I've looked up about a dozen of those, & it's a mix of:
 - AWS AS16509, AS14618
 - Indonesia AS138064, AS63859
 - Indonesia - some folks I couldn't find the ASN
 - Vietnam - AS45899

Of these, I think the Vietnam ASN covers the most of the IPs.

So...  One way to get a bunch of the CIDRs is to go to this site:
https://hackertarget.com/as-ip-lookup/

And enter the ASN: AS45899

The list you'll get doing so covers a lot of the IPs in the list provided.

The question is whether you have users there.  When I have one or two IPs, I just simply use the ACP admin user search & look up users by the first couple of nodes of an offending IP, e.g., "113.172" and see if I get any hits.

For a much bigger task like this, I actually download a copy of my smf_members into Excel & do a VLookup of member IPs against these ranges. 

If I find no matches, it's a no brainer.  Block 'em all.  Most of these attacks seem to come from commercial datacenters, not end-user ISPs.  Just block 'em.   

If I find matches, so blocking them may cut off existing users, it's a much harder decision...

(I'm on the road so can't do so or update my shared version of .htaccess for another week or so...)

[shawnb61]

I don't have all my utilities at my disposal (out of town), but if you're basically down, it might be worth trying this.  It has not been cleaned & de-duped & checked against users, etc.  This ASN appears to cover a big hunk of Vietnam. 

AS45899:

Code Select Expand

Deny from 103.10.44.0/22
Deny from 103.107.180.0/22
Deny from 103.120.242.0/23
Deny from 103.131.68.0/22
Deny from 103.148.56.0/23
Deny from 103.15.90.0/23
Deny from 103.154.244.0/23
Deny from 103.171.92.0/24
Deny from 103.197.184.0/23
Deny from 103.237.96.0/22
Deny from 103.238.214.0/24
Deny from 103.238.215.0/24
Deny from 103.255.239.0/24
Deny from 103.42.56.0/24
Deny from 103.42.57.0/24
Deny from 103.42.58.0/24
Deny from 103.42.59.0/24
Deny from 103.48.191.0/24
Deny from 103.53.168.0/24
Deny from 103.53.169.0/24
Deny from 103.56.156.0/22
Deny from 103.68.252.0/22
Deny from 103.69.192.0/23
Deny from 103.89.92.0/22
Deny from 103.9.76.0/22
Deny from 103.9.76.0/24
Deny from 103.9.77.0/24
Deny from 103.9.78.0/24
Deny from 103.9.79.0/24
Deny from 113.160.0.0/16
Deny from 113.161.0.0/16
Deny from 113.162.0.0/16
Deny from 113.163.0.0/16
Deny from 113.164.0.0/16
Deny from 113.165.0.0/16
Deny from 113.166.0.0/16
Deny from 113.167.0.0/16
Deny from 113.168.0.0/16
Deny from 113.169.0.0/16
Deny from 113.170.0.0/16
Deny from 113.171.0.0/16
Deny from 113.172.0.0/16
Deny from 113.173.0.0/16
Deny from 113.174.0.0/16
Deny from 113.175.0.0/16
Deny from 113.176.0.0/16
Deny from 113.177.0.0/16
Deny from 113.178.0.0/16
Deny from 113.179.0.0/16
Deny from 113.180.0.0/16
Deny from 113.181.0.0/16
Deny from 113.182.0.0/16
Deny from 113.183.0.0/16
Deny from 113.184.0.0/16
Deny from 113.185.0.0/16
Deny from 113.186.0.0/16
Deny from 113.187.0.0/16
Deny from 113.188.0.0/16
Deny from 113.189.0.0/16
Deny from 113.190.0.0/16
Deny from 113.191.0.0/16
Deny from 123.16.0.0/16
Deny from 123.17.0.0/16
Deny from 123.18.0.0/16
Deny from 123.19.0.0/16
Deny from 123.20.0.0/16
Deny from 123.21.0.0/16
Deny from 123.22.0.0/16
Deny from 123.23.0.0/16
Deny from 123.24.0.0/16
Deny from 123.25.0.0/16
Deny from 123.26.0.0/16
Deny from 123.27.0.0/16
Deny from 123.28.0.0/16
Deny from 123.29.0.0/16
Deny from 123.30.0.0/18
Deny from 123.30.0.0/20
Deny from 123.30.0.0/21
Deny from 123.30.0.0/23
Deny from 123.30.10.0/23
Deny from 123.30.10.0/24
Deny from 123.30.100.0/23
Deny from 123.30.102.0/23
Deny from 123.30.106.0/23
Deny from 123.30.106.0/24
Deny from 123.30.109.0/24
Deny from 123.30.112.0/20
Deny from 123.30.112.0/21
Deny from 123.30.112.0/23
Deny from 123.30.114.0/23
Deny from 123.30.116.0/23
Deny from 123.30.118.0/23
Deny from 123.30.12.0/23
Deny from 123.30.120.0/22
Deny from 123.30.120.0/23
Deny from 123.30.124.0/23
Deny from 123.30.128.0/18
Deny from 123.30.128.0/20
Deny from 123.30.130.0/23
Deny from 123.30.14.0/23
Deny from 123.30.141.0/24
Deny from 123.30.144.0/20
Deny from 123.30.144.0/22
Deny from 123.30.144.0/23
Deny from 123.30.144.0/24
Deny from 123.30.145.0/24
Deny from 123.30.146.0/23
Deny from 123.30.146.0/24
Deny from 123.30.147.0/24
Deny from 123.30.148.0/23
Deny from 123.30.148.0/24
Deny from 123.30.151.0/24
Deny from 123.30.152.0/23
Deny from 123.30.152.0/24
Deny from 123.30.153.0/24
Deny from 123.30.16.0/20
Deny from 123.30.16.0/21
Deny from 123.30.16.0/23
Deny from 123.30.160.0/20
Deny from 123.30.162.0/24
Deny from 123.30.166.0/23
Deny from 123.30.168.0/21
Deny from 123.30.168.0/23
Deny from 123.30.168.0/24
Deny from 123.30.169.0/24
Deny from 123.30.170.0/23
Deny from 123.30.170.0/24
Deny from 123.30.171.0/24
Deny from 123.30.172.0/23
Deny from 123.30.172.0/24
Deny from 123.30.173.0/24
Deny from 123.30.174.0/23
Deny from 123.30.174.0/24
Deny from 123.30.175.0/24
Deny from 123.30.176.0/20
Deny from 123.30.176.0/21
Deny from 123.30.176.0/23
Deny from 123.30.176.0/24
Deny from 123.30.177.0/24
Deny from 123.30.178.0/23
Deny from 123.30.178.0/24
Deny from 123.30.179.0/24
Deny from 123.30.18.0/23
Deny from 123.30.180.0/23
Deny from 123.30.180.0/24
Deny from 123.30.181.0/24
Deny from 123.30.182.0/23
Deny from 123.30.182.0/24
Deny from 123.30.183.0/24
Deny from 123.30.184.0/21
Deny from 123.30.184.0/23
Deny from 123.30.184.0/24
Deny from 123.30.185.0/24
Deny from 123.30.186.0/23
Deny from 123.30.186.0/24
Deny from 123.30.187.0/24
Deny from 123.30.188.0/23
Deny from 123.30.188.0/24
Deny from 123.30.189.0/24
Deny from 123.30.190.0/23
Deny from 123.30.190.0/24
Deny from 123.30.191.0/24
Deny from 123.30.192.0/18
Deny from 123.30.192.0/20
Deny from 123.30.192.0/21
Deny from 123.30.192.0/23
Deny from 123.30.2.0/23
Deny from 123.30.20.0/23
Deny from 123.30.202.0/23
Deny from 123.30.204.0/22
Deny from 123.30.208.0/20
Deny from 123.30.208.0/21
Deny from 123.30.208.0/23
Deny from 123.30.208.0/24
Deny from 123.30.209.0/24
Deny from 123.30.210.0/23
Deny from 123.30.210.0/24
Deny from 123.30.211.0/24
Deny from 123.30.212.0/23
Deny from 123.30.212.0/24
Deny from 123.30.214.0/23
Deny from 123.30.214.0/24
Deny from 123.30.215.0/24
Deny from 123.30.216.0/21
Deny from 123.30.217.0/24
Deny from 123.30.22.0/23
Deny from 123.30.223.0/24
Deny from 123.30.224.0/20
Deny from 123.30.224.0/23
Deny from 123.30.226.0/23
Deny from 123.30.228.0/23
Deny from 123.30.232.0/21
Deny from 123.30.232.0/23
Deny from 123.30.232.0/24
Deny from 123.30.233.0/24
Deny from 123.30.234.0/23
Deny from 123.30.234.0/24
Deny from 123.30.235.0/24
Deny from 123.30.236.0/23
Deny from 123.30.236.0/24
Deny from 123.30.237.0/24
Deny from 123.30.238.0/23
Deny from 123.30.238.0/24
Deny from 123.30.239.0/24
Deny from 123.30.24.0/21
Deny from 123.30.24.0/23
Deny from 123.30.240.0/20
Deny from 123.30.240.0/23
Deny from 123.30.240.0/24
Deny from 123.30.241.0/24
Deny from 123.30.245.0/24
Deny from 123.30.246.0/23
Deny from 123.30.246.0/24
Deny from 123.30.247.0/24
Deny from 123.30.248.0/23
Deny from 123.30.249.0/24
Deny from 123.30.251.0/24
Deny from 123.30.26.0/23
Deny from 123.30.28.0/23
Deny from 123.30.29.0/24
Deny from 123.30.30.0/23
Deny from 123.30.32.0/20
Deny from 123.30.32.0/21
Deny from 123.30.32.0/23
Deny from 123.30.34.0/23
Deny from 123.30.34.0/24
Deny from 123.30.36.0/23
Deny from 123.30.37.0/24
Deny from 123.30.38.0/23
Deny from 123.30.4.0/23
Deny from 123.30.40.0/21
Deny from 123.30.40.0/23
Deny from 123.30.40.0/24
Deny from 123.30.41.0/24
Deny from 123.30.42.0/23
Deny from 123.30.42.0/24
Deny from 123.30.48.0/20
Deny from 123.30.48.0/21
Deny from 123.30.48.0/23
Deny from 123.30.48.0/24
Deny from 123.30.49.0/24
Deny from 123.30.50.0/23
Deny from 123.30.50.0/24
Deny from 123.30.51.0/24
Deny from 123.30.52.0/23
Deny from 123.30.53.0/24
Deny from 123.30.54.0/23
Deny from 123.30.56.0/21
Deny from 123.30.56.0/23
Deny from 123.30.57.0/24
Deny from 123.30.58.0/23
Deny from 123.30.58.0/24
Deny from 123.30.59.0/24
Deny from 123.30.6.0/23
Deny from 123.30.60.0/23
Deny from 123.30.60.0/24
Deny from 123.30.61.0/24
Deny from 123.30.62.0/23
Deny from 123.30.64.0/18
Deny from 123.30.64.0/20
Deny from 123.30.64.0/23
Deny from 123.30.66.0/23
Deny from 123.30.66.0/24
Deny from 123.30.68.0/23
Deny from 123.30.74.0/23
Deny from 123.30.74.0/24
Deny from 123.30.75.0/24
Deny from 123.30.76.0/23
Deny from 123.30.78.0/23
Deny from 123.30.8.0/21
Deny from 123.30.8.0/23
Deny from 123.30.8.0/24
Deny from 123.30.80.0/20
Deny from 123.30.80.0/21
Deny from 123.30.82.0/23
Deny from 123.30.86.0/23
Deny from 123.30.88.0/23
Deny from 123.30.9.0/24
Deny from 123.30.90.0/23
Deny from 123.30.92.0/23
Deny from 123.30.96.0/20
Deny from 123.30.96.0/21
Deny from 123.30.96.0/23
Deny from 123.30.96.0/24
Deny from 123.30.97.0/24
Deny from 123.30.98.0/23
Deny from 123.31.0.0/18
Deny from 123.31.0.0/21
Deny from 123.31.110.0/24
Deny from 123.31.111.0/24
Deny from 123.31.112.0/20
Deny from 123.31.113.0/24
Deny from 123.31.128.0/18
Deny from 123.31.128.0/20
Deny from 123.31.14.0/23
Deny from 123.31.144.0/20
Deny from 123.31.15.0/24
Deny from 123.31.156.0/24
Deny from 123.31.16.0/21
Deny from 123.31.16.0/24
Deny from 123.31.160.0/20
Deny from 123.31.17.0/24
Deny from 123.31.176.0/20
Deny from 123.31.18.0/24
Deny from 123.31.184.0/23
Deny from 123.31.186.0/23
Deny from 123.31.19.0/24
Deny from 123.31.192.0/18
Deny from 123.31.192.0/20
Deny from 123.31.20.0/24
Deny from 123.31.208.0/20
Deny from 123.31.21.0/24
Deny from 123.31.22.0/24
Deny from 123.31.224.0/20
Deny from 123.31.23.0/24
Deny from 123.31.240.0/20
Deny from 123.31.25.0/24
Deny from 123.31.26.0/24
Deny from 123.31.32.0/19
Deny from 123.31.48.0/20
Deny from 123.31.64.0/18
Deny from 123.31.64.0/20
Deny from 123.31.64.0/23
Deny from 123.31.66.0/23
Deny from 123.31.8.0/24
Deny from 123.31.80.0/20
Deny from 123.31.9.0/24
Deny from 123.31.96.0/20
Deny from 123.31.96.0/23
Deny from 123.31.98.0/23
Deny from 14.160.0.0/16
Deny from 14.161.0.0/16
Deny from 14.162.0.0/16
Deny from 14.163.0.0/16
Deny from 14.164.0.0/16
Deny from 14.165.0.0/16
Deny from 14.166.0.0/16
Deny from 14.167.0.0/16
Deny from 14.168.0.0/16
Deny from 14.169.0.0/16
Deny from 14.170.0.0/16
Deny from 14.171.0.0/16
Deny from 14.172.0.0/16
Deny from 14.173.0.0/16
Deny from 14.174.0.0/16
Deny from 14.175.0.0/16
Deny from 14.176.0.0/16
Deny from 14.177.0.0/16
Deny from 14.178.0.0/16
Deny from 14.179.0.0/16
Deny from 14.180.0.0/16
Deny from 14.181.0.0/16
Deny from 14.182.0.0/16
Deny from 14.183.0.0/16
Deny from 14.184.0.0/16
Deny from 14.185.0.0/16
Deny from 14.186.0.0/16
Deny from 14.187.0.0/16
Deny from 14.188.0.0/16
Deny from 14.189.0.0/16
Deny from 14.190.0.0/16
Deny from 14.191.0.0/16
Deny from 14.224.0.0/16
Deny from 14.226.0.0/16
Deny from 14.227.0.0/16
Deny from 14.228.0.0/16
Deny from 14.229.0.0/16
Deny from 14.230.0.0/16
Deny from 14.231.0.0/16
Deny from 14.232.0.0/16
Deny from 14.233.0.0/16
Deny from 14.234.0.0/16
Deny from 14.235.0.0/16
Deny from 14.236.0.0/16
Deny from 14.237.0.0/16
Deny from 14.238.0.0/16
Deny from 14.239.0.0/16
Deny from 14.240.0.0/16
Deny from 14.241.0.0/16
Deny from 14.242.0.0/16
Deny from 14.243.0.0/16
Deny from 14.244.0.0/16
Deny from 14.245.0.0/16
Deny from 14.246.0.0/16
Deny from 14.247.0.0/16
Deny from 14.248.0.0/16
Deny from 14.249.0.0/16
Deny from 14.250.0.0/16
Deny from 14.251.0.0/16
Deny from 14.252.0.0/16
Deny from 14.253.0.0/16
Deny from 14.254.0.0/16
Deny from 14.255.0.0/16
Deny from 146.196.64.0/24
Deny from 146.196.65.0/24
Deny from 146.196.66.0/24
Deny from 146.196.67.0/24
Deny from 157.119.246.0/23
Deny from 157.119.246.0/24
Deny from 157.119.247.0/24
Deny from 157.66.13.0/24
Deny from 160.19.78.0/23
Deny from 160.191.52.0/23
Deny from 160.22.16.0/23
Deny from 160.22.2.0/24
Deny from 160.22.3.0/24
Deny from 160.25.74.0/23
Deny from 160.250.216.0/23
Deny from 160.30.54.0/23
Deny from 161.248.30.0/23
Deny from 163.61.74.0/23
Deny from 165.99.58.0/24
Deny from 165.99.59.0/24
Deny from 203.160.0.0/23
Deny from 203.160.132.0/22
Deny from 203.162.0.0/16
Deny from 203.210.128.0/17
Deny from 203.210.128.0/18
Deny from 203.210.128.0/20
Deny from 203.210.136.0/22
Deny from 203.210.144.0/20
Deny from 203.210.158.0/24
Deny from 203.210.160.0/20
Deny from 203.210.176.0/20
Deny from 203.210.192.0/18
Deny from 203.210.192.0/20
Deny from 203.210.192.0/24
Deny from 203.210.208.0/20
Deny from 203.210.224.0/20
Deny from 203.210.237.0/24
Deny from 203.210.240.0/20
Deny from 221.132.0.0/20
Deny from 221.132.16.0/24
Deny from 221.132.17.0/24
Deny from 221.132.18.0/24
Deny from 221.132.19.0/24
Deny from 221.132.20.0/24
Deny from 221.132.21.0/24
Deny from 221.132.22.0/24
Deny from 221.132.23.0/24
Deny from 221.132.24.0/24
Deny from 221.132.25.0/24
Deny from 221.132.26.0/24
Deny from 221.132.27.0/24
Deny from 221.132.28.0/24
Deny from 221.132.29.0/24
Deny from 221.132.30.0/23
Deny from 221.132.30.0/24
Deny from 221.132.31.0/24
Deny from 221.132.32.0/21
Deny from 221.132.32.0/22
Deny from 221.132.37.0/24
Deny from 221.132.38.0/24
Deny from 221.132.52.0/23
Deny from 221.132.54.0/23
Deny from 222.252.0.0/16
Deny from 222.253.0.0/16
Deny from 222.254.0.0/16
Deny from 222.255.0.0/16
Deny from 23.32.249.0/24
Deny from 23.48.52.0/22
Deny from 23.48.56.0/22
Deny from 23.53.208.0/22
Deny from 23.53.212.0/22
Deny from 43.239.220.0/23
Deny from 45.124.92.0/22
Deny from 45.254.32.0/22


[Rhindeer]

I don't have all my utilities at my disposal (out of town), but if you're basically down, it might be worth trying this.  It has not been cleaned & de-duped & checked against users, etc.  This ASN appears to cover a big hunk of Vietnam. 

Thank yooou! My forum was getting hammered by this same ASN so adding this list to htaccess has made my current guest list drop from 3000+ to 800 in just a couple minutes. (And it's still rapidly dropping.)

[a10]

I see tons in the 160~,170~,180~ range, South America, Brazil.
And those from Vietnam.

[shawnb61]

I see tons in the 160~,170~,180~ range, South America, Brazil.
And those from Vietnam.

If it's affecting your site, it's Pareto analysis time... 

Find the ASN that seems to be causing the biggest grief.  Do some checks to confirm you're not cutting off your valid users.  Add to .htaccess.

It's a never-ending game of whack-a-mole. 

Cleaning the lists...  I did perform some rudimentary cleaning of the list above...  I sorted it, removed the ipv6 entries.  Performed a quick manual de-dupe by looking for CIDRs ending in /16, then deleting everything with the same first two nodes.  E.g., in this example:
123.456.0.0/16
123.456.56.0/24
123.456.57.0/24

...it's only necessary to keep the first entry, all other 123.456.*.* entries can be deleted since the /16 entry includes all those.  A quick pass doing so greatly reduces the size of the list. 

[shawnb61]

One other item to consider...  I've made the following change to my forum and it has basically eliminated the weird spikes in guest counts.  Effectively giving them a buzz cut.  Same with the MySQL CPU usage spikes that can accompany them. 

I believe those come from crawlers that disallow cookies.  SMF, not finding a cookie, can't find a session, so creates a new session - which looks like a new visitor.  But with cookies disabled, that crawler will never find their prior session, i.e., it'll never be used.  So the next page load creates yet another session.  Rinse & repeat...  So...  Don't write them.  Although writing a session is a trivial & fast I/O, if you pile on a tremendous number of them, it bottlenecks and can cause issues.

This helps with crawlers that disallow cookies that are loading pages - i.e., even when not blocked by .htaccess.  Not a lot of bots behave like this (or you'd see these spikes every day), but the ones that do can cause problems.

In Sources/Sessions.php, near the top of the sessionWrite() function, add this:

Code Select Expand

// Don't bother writing the session if cookies are disabled; no way to retrieve it later
   if (empty($_COOKIE))
      return true;
It is one part of this PR: https://github.com/SimpleMachines/SMF/pull/8394

You will need to undo this change if that PR is ever merged & included in a future patch.

[Rhindeer]

@ShawnB ahhhh excellent! I'm going to add that code in when I get home and see how it goes! Thank you so much!

[a10]

...and suddenly they all left :O) Remains Microsoft and other more or less gentle bots.

[vbgamer45]

@shawnb61  this is great thanks for the tip.

[bayonetbrant]

Current .htaccess file looks like it was already modified in an attempt to clear out some bots.

@shawnb61 would the code from your GitHub acc't replace this?  or add to it?

Thanks!

Code Select Expand


#Simple .htaccess Bot Blocker Mod
# Rate limit search engine crawlers
<IfModule mod_ratelimit.c>
  SetEnvIfNoCase User-Agent "^Baiduspider" crawler_bot
  SetEnvIfNoCase User-Agent "^AhrefsBot" crawler_bot
  SetEnvIfNoCase User-Agent "^Ezooms" crawler_bot
  SetEnvIfNoCase User-Agent "^Googlebot" crawler_bot
  SetEnvIfNoCase User-Agent "^Slurp" crawler_bot
  SetEnvIfNoCase User-Agent "^msnbot" crawler_bot
  SetEnvIfNoCase User-Agent "^Yandex" crawler_bot

  RLimitCPU crawler_bot 1 1
  RLimitIO crawler_bot 512 512
  RLimitMEM crawler_bot 128 128
</IfModule>

<IfModule mod_rewrite.c>
RewriteEngine on
RewriteBase /
RewriteCond %{HTTP_USER_AGENT} ADSARobot|ah-ha|almaden|aktuelles|Anarchie|amzn_assoc|ASPSeek|ASSORT|ATHENS|Atomz|attach|attache|autoemailspider|BackWeb|Bandit|BatchFTP|bdfetch|big.brother|BlackWidow|bmclient|Boston\ Project|BravoBrian\ SpiderEngine\ MarcoPolo|Bot\ mailto:[email protected]|Buddy|Bullseye|bumblebee|capture|CherryPicker|ChinaClaw|CICC|clipping|Collector|Copier|Crescent|Crescent\ Internet\ ToolPak|Custo|cyberalert|DA$|Deweb|diagem|Digger|Digimarc|DIIbot|DISCo|DISCo\ Pump|DISCoFinder|Download\ Demon|Download\ Wonder|Downloader|Drip|DSurf15a|DTS.Agent|EasyDL|eCatch|ecollector|efp@gmx\.net|Email\ Extractor|EirGrabber|email|EmailCollector|EmailSiphon|EmailWolf|Express\ WebPictures|ExtractorPro|EyeNetIE|FavOrg|fastlwspider|Favorites\ Sweeper|Fetch|FEZhead|FileHound|FlashGet\ WebWasher|FlickBot|fluffy|FrontPage|GalaxyBot|Generic|Getleft|GetRight|GetSmart|GetWeb!|GetWebPage|gigabaz|Girafabot|Go\!Zilla|Go!Zilla|Go-Ahead-Got-It|GornKer|gotit|Grabber|GrabNet|Grafula|Green\ Research|grub-client|Harvest|hhjhj@yahoo|hloader|HMView|HomePageSearch|http\ generic|HTTrack|httpdown|httrack|ia_archiver|IBM_Planetwide|Image\ Stripper|Image\ Sucker|imagefetch|IncyWincy|Indy*Library|Indy\ Library|informant|Ingelin|InterGET|Internet\ Ninja|InternetLinkagent|Internet\ Ninja|InternetSeer\.com|Iria|Irvine|JBH*agent|JetCar|JOC|JOC\ Web\ Spider|JustView|KWebGet|Lachesis|larbin|LeechFTP|LexiBot|lftp|libwww|likse|Link|Link*Sleuth|LINKS\ ARoMATIZED|LinkWalker|LWP|lwp-trivial|Mag-Net|Magnet|Mac\ Finder|Mag-Net|Mass\ Downloader|MCspider|Memo|Microsoft.URL|MIDown\ tool|Mirror|Missigua\ Locator|Mister\ PiX|MMMtoCrawl\/UrlDispatcherLLL|^Mozilla$|Mozilla.*Indy|Mozilla.*NEWT|Mozilla*MSIECrawler|MS\ FrontPage*|MSFrontPage|MSIECrawler|MSProxy|multithreaddb|nationaldirectory|Navroad|NearSite|NetAnts|NetCarta|NetMechanic|netprospector|NetResearchServer|NetSpider|Net\ Vampire|NetZIP|NetZip\ Downloader|NetZippy|NEWT|NICErsPRO|Ninja|NPBot|Octopus|Offline\ Explorer|Offline\ Navigator|OpaL|Openfind|OpenTextSiteCrawler|OrangeBot|PageGrabber|Papa\ Foto|PackRat|pavuk|pcBrowser|PersonaPilot|Ping|PingALink|Pockey|Proxy|psbot|PSurf|puf|Pump|PushSite|QRVA|RealDownload|Reaper|Recorder|ReGet|replacer|RepoMonkey|Robozilla|Rover|RPT-HTTPClient|Rsync|Scooter|SearchExpress|searchhippo|searchterms\.it|Second\ Street\ Research|Seeker|Shai|Siphon|sitecheck|sitecheck.internetseer.com|SiteSnagger|SlySearch|SmartDownload|snagger|Snake|SpaceBison|Spegla|SpiderBot|sproose|SqWorm|Stripper|Sucker|SuperBot|SuperHTTP|Surfbot|SurfWalker|Szukacz|tAkeOut|tarspider|Teleport\ Pro|Templeton|TrueRobot|TV33_Mercator|UIowaCrawler|UtilMind|URLSpiderPro|URL_Spider_Pro|Vacuum|vagabondo|vayala|visibilitygap|VoidEYE|vspider|Web\ Downloader|w3mir|Web\ Data\ Extractor|Web\ Image\ Collector|Web\ Sucker|Wweb|WebAuto|WebBandit|web\.by\.mail|Webclipping|webcollage|webcollector|WebCopier|webcraft@bea|webdevil|webdownloader|Webdup|WebEMailExtrac|WebFetch|WebGo\ IS|WebHook|Webinator|WebLeacher|WEBMASTERS|WebMiner|WebMirror|webmole|WebReaper|WebSauger|Website|Website\ eXtractor|Website\ Quester|WebSnake|Webster|WebStripper|websucker|webvac|webwalk|webweasel|WebWhacker|WebZIP|Wget|Whacker|whizbang|WhosTalking|Widow|WISEbot|WWWOFFLE|x-Tractor|^Xaldon\ WebSpider|WUMPUS|Xenu|XGET|Zeus.*Webster|Zeus [NC]
RewriteRule ^.* - [F,L]
</IfModule>
#Simple .htaccess Bot Blocker Mod

#Simple .htaccess Bot Blocker Mod
<IfModule mod_rewrite.c>
RewriteEngine on
RewriteBase /
RewriteCond %{HTTP_USER_AGENT} ADSARobot|ah-ha|almaden|aktuelles|Anarchie|amzn_assoc|ASPSeek|ASSORT|ATHENS|Atomz|attach|attache|autoemailspider|BackWeb|Bandit|BatchFTP|bdfetch|big.brother|BlackWidow|bmclient|Boston\ Project|BravoBrian\ SpiderEngine\ MarcoPolo|Bot\ mailto:[email protected]|Buddy|Bullseye|bumblebee|capture|CherryPicker|ChinaClaw|CICC|clipping|Collector|Copier|Crescent|Crescent\ Internet\ ToolPak|Custo|cyberalert|DA$|Deweb|diagem|Digger|Digimarc|DIIbot|DISCo|DISCo\ Pump|DISCoFinder|Download\ Demon|Download\ Wonder|Downloader|Drip|DSurf15a|DTS.Agent|EasyDL|eCatch|ecollector|efp@gmx\.net|Email\ Extractor|EirGrabber|email|EmailCollector|EmailSiphon|EmailWolf|Express\ WebPictures|ExtractorPro|EyeNetIE|FavOrg|fastlwspider|Favorites\ Sweeper|Fetch|FEZhead|FileHound|FlashGet\ WebWasher|FlickBot|fluffy|FrontPage|GalaxyBot|Generic|Getleft|GetRight|GetSmart|GetWeb!|GetWebPage|gigabaz|Girafabot|Go\!Zilla|Go!Zilla|Go-Ahead-Got-It|GornKer|gotit|Grabber|GrabNet|Grafula|Green\ Research|grub-client|Harvest|hhjhj@yahoo|hloader|HMView|HomePageSearch|http\ generic|HTTrack|httpdown|httrack|ia_archiver|IBM_Planetwide|Image\ Stripper|Image\ Sucker|imagefetch|IncyWincy|Indy*Library|Indy\ Library|informant|Ingelin|InterGET|Internet\ Ninja|InternetLinkagent|Internet\ Ninja|InternetSeer\.com|Iria|Irvine|JBH*agent|JetCar|JOC|JOC\ Web\ Spider|JustView|KWebGet|Lachesis|larbin|LeechFTP|LexiBot|lftp|libwww|likse|Link|Link*Sleuth|LINKS\ ARoMATIZED|LinkWalker|LWP|lwp-trivial|Mag-Net|Magnet|Mac\ Finder|Mag-Net|Mass\ Downloader|MCspider|Memo|Microsoft.URL|MIDown\ tool|Mirror|Missigua\ Locator|Mister\ PiX|MMMtoCrawl\/UrlDispatcherLLL|^Mozilla$|Mozilla.*Indy|Mozilla.*NEWT|Mozilla*MSIECrawler|MS\ FrontPage*|MSFrontPage|MSIECrawler|MSProxy|multithreaddb|nationaldirectory|Navroad|NearSite|NetAnts|NetCarta|NetMechanic|netprospector|NetResearchServer|NetSpider|Net\ Vampire|NetZIP|NetZip\ Downloader|NetZippy|NEWT|NICErsPRO|Ninja|NPBot|Octopus|Offline\ Explorer|Offline\ Navigator|OpaL|Openfind|OpenTextSiteCrawler|OrangeBot|PageGrabber|Papa\ Foto|PackRat|pavuk|pcBrowser|PersonaPilot|Ping|PingALink|Pockey|Proxy|psbot|PSurf|puf|Pump|PushSite|QRVA|RealDownload|Reaper|Recorder|ReGet|replacer|RepoMonkey|Robozilla|Rover|RPT-HTTPClient|Rsync|Scooter|SearchExpress|searchhippo|searchterms\.it|Second\ Street\ Research|Seeker|Shai|Siphon|sitecheck|sitecheck.internetseer.com|SiteSnagger|SlySearch|SmartDownload|snagger|Snake|SpaceBison|Spegla|SpiderBot|sproose|SqWorm|Stripper|Sucker|SuperBot|SuperHTTP|Surfbot|SurfWalker|Szukacz|tAkeOut|tarspider|Teleport\ Pro|Templeton|TrueRobot|TV33_Mercator|UIowaCrawler|UtilMind|URLSpiderPro|URL_Spider_Pro|Vacuum|vagabondo|vayala|visibilitygap|VoidEYE|vspider|Web\ Downloader|w3mir|Web\ Data\ Extractor|Web\ Image\ Collector|Web\ Sucker|Wweb|WebAuto|WebBandit|web\.by\.mail|Webclipping|webcollage|webcollector|WebCopier|webcraft@bea|webdevil|webdownloader|Webdup|WebEMailExtrac|WebFetch|WebGo\ IS|WebHook|Webinator|WebLeacher|WEBMASTERS|WebMiner|WebMirror|webmole|WebReaper|WebSauger|Website|Website\ eXtractor|Website\ Quester|WebSnake|Webster|WebStripper|websucker|webvac|webwalk|webweasel|WebWhacker|WebZIP|Wget|Whacker|whizbang|WhosTalking|Widow|WISEbot|WWWOFFLE|x-Tractor|^Xaldon\ WebSpider|WUMPUS|Xenu|XGET|Zeus.*Webster|Zeus [NC]
RewriteRule ^.* - [F,L]
</IfModule>
#Simple .htaccess Bot Blocker Mod

[shawnb61]

Current .htaccess file looks like it was already modified in an attempt to clear out some bots.

@shawnb61 would the code from your GitHub acc't replace this?  or add to it?

You have 3 blocks of code there...  The first uses mod_ratelimit.c which I'm not familiar with (though taken at face value, appears to be quite useful...)

The next two blocks use a pretty standard method for blocking via user agent, and appear to be exact clones.  One of those two is unnecessary.

A couple of thoughts:

If these are getting put in your .htaccess via a cPanel function, then they should be updated via cPanel.  Removing them manually won't work...

The bots that appear to be throttled in the first block are eliminated in my code, so that first block would be unnecessary.

The bots blocked in the next two overlap with my set, but only block by useragent & not by IP. 

So it comes down to how many of those useragents were put there deliberately due to specific issues with your forum.  It's up to you whether to merge with my list or not.

My initial thought is to simply replace it with mine.  If some of those guys unique to your old list come back, you can add them.  Keep a backup.

And read all my notes about my selection criteria over on GitHub.  Make sure you're comfy with the overall approach. 

[bayonetbrant]

My initial thought is to simply replace it with mine.  If some of those guys unique to your old list come back, you can add them.  Keep a backup.

And read all my notes about my selection criteria over on GitHub.  Make sure you're comfy with the overall approach. 

I will definitely keep a backup

Our .htaccess file was set up by one of members who does some of this stuff professionally, but who left the community a while back b/c of some personality issues, and not available.

I did ready your release notes (thank you for being comprehensive about them!) and I think the approach seems pretty sound.

[Rhindeer]

@shawnb61 I made that change to session.php and boom, that handled the issue! Our guests are now in the low double digits. Thank you so much!

[shawnb61]

@shawnb61 I made that change to session.php and boom, that handled the issue! Our guests are now in the low double digits. Thank you so much!

I'm glad it worked out.  I made that change back in January (https://github.com/SimpleMachines/SMF/pull/8394), & I haven't seen a spike in guests since.  More importantly, I haven't seen the corresponding spikes in MySQL CPU either.  Both charts got a serious buzz cut...

[a10]

So, the South America \ Vietnam \ Indonesia rats back again, was 17.000+ guests in 720 minutes. A zillion unique ip's. Normal guests \ bots 500 to 750. Pageviews for the last few days, see attachment, normal around 5.000.

Set forum to login only, and testing some country blocks, brazil, vietnam, indonesia (10.500 lines of ranges in htaccess!). Helped, but always the risk of chaos and blocking legit members\guests. Still lots of hits, colombia, chili, ecuador, argentina etc.

Total madness, attempting to scrape everything, even trying member's profiles, probably script counting up from forum/index.php?action=profile;u=1

But amazingly zero disruption to normal forum behaviour, speed etc.

[HITG]

Did you guys know a premium dns can solve this without touching the htaccess.

[Kindred]

did you know that we can do it via htacess without paying for a "premium" dns?

[HITG]

did you know that we can do it via htacess without paying for a "premium" dns?

The only thing is htaccess is meant to be really untouched file when it comes down to big forum engines.

[vbgamer45]

Cloudflare on the free plan working well for me can block by asn as well. I have an article on country blocking at https://www.simplemachines.org/community/index.php?topic=591920.msg4190552#new

So far on free plan handled 8.7 million requests and 731k unique visitors in three days.

[HITG]

Cloudflare on the free plan working well for me can block by asn as well. I have an article on country blocking at https://www.simplemachines.org/community/index.php?topic=591920.msg4190552#new

So far on free plan handled 8.7 million requests and 731k unique visitors in three days.

If they ever start charging you just use a premium dns can stop spam too. Cheap for year and you can do route of dns to stop it.

[shawnb61]

Very tempting.  My host, ICDSoft, even provides instructions how to tie into the Cloudflare free plan:
https://www.icdsoft.com/en/kb/view/2077_how_to_enable_ddos_protection_and_cdn_with_cloudflare_on_your_website

[Tripredacus]

I am going to have to really look into this before I actually open my forum to the public. I already know this is a huge issue since I admin on another forum and these traffic swarms often knock the site offline for a short period.

[vii]

So, the South America \ Vietnam \ Indonesia rats back again, was 17.000+ guests in 720 minutes. A zillion unique ip's. Normal guests \ bots 500 to 750. Pageviews for the last few days, see attachment, normal around 5.000.

Set forum to login only, and testing some country blocks, brazil, vietnam, indonesia (10.500 lines of ranges in htaccess!). Helped, but always the risk of chaos and blocking legit members\guests. Still lots of hits, colombia, chili, ecuador, argentina etc.

Total madness, attempting to scrape everything, even trying member's profiles, probably script counting up from forum/index.php?action=profile;u=1

But amazingly zero disruption to normal forum behaviour, speed etc.

I can't help but be curious about who exactly is doing this. My bot traffic includes lots of residential IPs, so pretty stereotypical of botnet traffic. The regions you just listed are exactly who is hitting my forum as well. I've long had other mitigations in place, so it never got as bad as 17,000, but I did top out in the 7000-range. It's quite irritating either way.

Also, thanks @shawnb61 for the session tip 🫡

[Kindred]

Very tempting.  My host, ICDSoft, even provides instructions how to tie into the Cloudflare free plan:
https://www.icdsoft.com/en/kb/view/2077_how_to_enable_ddos_protection_and_cdn_with_cloudflare_on_your_website

Yeah,  my forum got swarmed the other day.  Icdsoft support analyzed and added a block to my htaccess for the new agent.
I love that host!

[a10]

Yeah,  my forum got swarmed the other day.  Icdsoft support analyzed and added a block to my htaccess for the new agent.
I love that host!

Looks good, possible to share that agent's htaccess setting ?
Also looking for something to block Amazon.
Thanks