Help ... getting 508 Exceeded Resources Error

[woolly bugger]

I've been hosting my SMF forum with inmotionhosting (powerPlan) for decades. Recently my users and I have been virtually unable to access the forum.

Resource Limit Is Reached
The website is temporarily unable to service your request as it exceeded resource limit. Please try again later.

Tech support chats and phone calls revealed a large number of guest users, and bots. T

I've put the board in maintenance mode but the site continues to be hit with bots / spiders crawling thousands of odd random pages, profiles, logon attempts ets.

This has been going on for about a week now and i don't know where to turn.

The Tech recommended using Cloudflair to mitigate the problem. Is this a viable solution? or would the Botbanish mod be a better solution.


Would moving to a new host have a positive effect long term?

Top user agents------------------------------------------------------------------------------------
807  "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; ClaudeBot/1.0; +claudebot@anthr
184  "Mozilla/5.0 (compatible; DotBot/1.2; +https://opensiteexplorer.org/dotbot; [email protected])"
21  "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/78.0.3
18  "Mozilla/5.0 (Linux; Android 7.0;) AppleWebKit/537.36 (KHTML, like Gecko) Mobile Safari/537.36
15  "Friendly_Crawler/Nutch-1.20-SNAPSHOT"
12  "COMODO DCV"
12  "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/117.0.
  9  "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; bingbot/2.0; +http://www.bing.c
  8  "Sogou web spider/4.0(+http://www.sogou.com/docs/help/webmasters.htm#07)"
  6  "magpie-crawler/1.1 (robots-txt-checker; +http://www.brandwatch.net)"

In maintenance mode this is what I'm seeing in a 15 minute snapshot


You cannot view this attachment.



Please advise....

[shawnb61]

You are going to have to block a few of the bad ones. 

We were just discussing claudebot over in the Open Journal thread:
https://www.simplemachines.org/community/index.php?msg=4173320

Have you been blocking bad bots via .htaccess?  If not, you should start doing so.  If so, add claudebot to the list.

Are you attempting to regulate good bots via robots.txt?  If not, you should consider doing so.  Most bots do in fact follow it, and you can tell them to slow down if they are hitting you too fast.

Vbgamer included his .htaccess here, it's a very helpful list if you don't have one (I would add claudebot to this list):

My current badbot useragent list

Code Select Expand

<Location />
<Limit GET POST PUT>

# Begin Bad Bot Blocking
BrowserMatchNoCase OmniExplorer_Bot/6.11.1 bad_bot
BrowserMatchNoCase omniexplorer_bot bad_bot
BrowserMatchNoCase Baiduspider bad_bot
BrowserMatchNoCase Baiduspider/2.0 bad_bot
BrowserMatchNoCase yandex bad_bot
BrowserMatchNoCase yandeximages bad_bot
BrowserMatchNoCase Spinn3r bad_bot
BrowserMatchNoCase sogou bad_bot
BrowserMatchNoCase Sogouwebspider/3.0 bad_bot
BrowserMatchNoCase Sogouwebspider/4.0 bad_bot
BrowserMatchNoCase sosospider+ bad_bot
BrowserMatchNoCase jikespider bad_bot
BrowserMatchNoCase ia_archiver bad_bot
BrowserMatchNoCase PaperLiBot bad_bot
BrowserMatchNoCase ahrefsbot bad_bot
BrowserMatchNoCase ahrefsbot/1.0 bad_bot
BrowserMatchNoCase SiteBot/0.1 bad_bot
BrowserMatchNoCase DNS-Digger/1.0 bad_bot
BrowserMatchNoCase DNS-Digger-Explorer/1.0 bad_bot
BrowserMatchNoCase boardreader bad_bot
BrowserMatchNoCase radian6 bad_bot
BrowserMatchNoCase R6_FeedFetcher bad_bot
BrowserMatchNoCase R6_CommentReader bad_bot
BrowserMatchNoCase ScoutJet bad_bot
BrowserMatchNoCase ezooms bad_bot
BrowserMatchNoCase CC-rget/5.818 bad_bot
BrowserMatchNoCase libwww-perl/5.813 bad_bot
BrowserMatchNoCase magpie-crawler 1.1 bad_bot
BrowserMatchNoCase jakarta bad_bot
BrowserMatchNoCase discobot/1.0 bad_bot
BrowserMatchNoCase MJ12bot bad_bot
BrowserMatchNoCase MJ12bot/v1.2.0 bad_bot
BrowserMatchNoCase MJ12bot/v1.2.5 bad_bot
BrowserMatchNoCase SemrushBot/0.9 bad_bot
BrowserMatchNoCase MLBot bad_bot
BrowserMatchNoCase butterfly bad_bot
BrowserMatchNoCase SeznamBot/3.0 bad_bot
BrowserMatchNoCase HuaweiSymantecSpider bad_bot
BrowserMatchNoCase Exabot/2.0 bad_bot
BrowserMatchNoCase netseer/0.1 bad_bot
BrowserMatchNoCase NetSeer crawler/2.0 bad_bot
BrowserMatchNoCase NetSeer/Nutch-0.9 bad_bot
BrowserMatchNoCase psbot/0.1 bad_bot
BrowserMatchNoCase moreoverbot/5.0 bad_bot
BrowserMatchNoCase Jakarta Commons-HttpClient/3.0 bad_bot
BrowserMatchNoCase SocialSpider-Finder/0.2 bad_bot
BrowserMatchNoCase wordpress bad_bot
BrowserMatchNoCase istellabot bad_bot
BrowserMatchNoCase SeznamBot bad_bot
BrowserMatchNoCase Cliqzbot bad_bot
BrowserMatchNoCase SocialRankIOBot bad_bot
BrowserMatchNoCase Mail.RU_Bot bad_bot
BrowserMatchNoCase Clickag Bot bad_bot
BrowserMatchNoCase Mediatoolkitbot bad_bot
BrowserMatchNoCase SemrushBot bad_bot
BrowserMatchNoCase DotBot/1.1 bad_bot
BrowserMatchNoCase DataForSeoBot bad_bot
BrowserMatchNoCase www.timpi.io bad_bot
BrowserMatchNoCase DotBot bad_bot
BrowserMatchNoCase trendictionbot bad_bot
BrowserMatchNoCase BLEXBot/1.0 bad_bot
BrowserMatchNoCase SeekportBot bad_bot
BrowserMatchNoCase Turnitin bad_bot
BrowserMatchNoCase omgili/0.5 bad_bot
BrowserMatchNoCase CheckHost bad_bot
BrowserMatchNoCase Amazonbot bad_bot
BrowserMatchNoCase SEOkicks bad_bot
<RequireAll>
Require all granted
<RequireNone>
Require env bad_bot
</RequireNone>
</RequireAll>

</Limit>
</Location>


Some other reading materials:
https://github.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/blob/master/_generator_lists/bad-user-agents.list
https://www.imperva.com/blog/most-active-good-bots/
https://radar.cloudflare.com/traffic/verified-bots

[woolly bugger]

ok, my .htaccess file is written by cpanel and if I add to it I get 500 server errors.

here is the current file.

Code Select Expand

# php -- BEGIN cPanel-generated handler, do not edit
# Set the "ea-php81" package as the default "PHP" programming language.
<IfModule mime_module>
  AddHandler application/x-httpd-ea-php81 .php .php8 .phtml
</IfModule>
# php -- END cPanel-generated handler, do not edit

<Files 403.shtml>
order allow,deny
allow from all
</Files>

deny from 124.220.160.30
deny from 153.149.181.144
deny from 85.208.48.215
deny from 185.125.219.13
deny from 194.67.207.94
deny from 185.5.249.185
deny from 194.67.207.9
deny from 193.124.190.18
deny from 193.124.188.83
deny from 194.67.210.77
deny from 94.142.141.230
deny from 193.124.191.92
deny from 194.113.106.9
deny from 213.219.247.144
deny from 213.219.247.125
deny from 89.22.235.136
deny from 216.244.66.240

# BEGIN cPanel-generated php ini directives, do not edit
# Manual editing of this file may result in unexpected behavior.
# To make changes to this file, use the cPanel MultiPHP INI Editor (Home >> Software >> MultiPHP INI Editor)
# For more information, read our documentation (https://go.cpanel.net/EA4ModifyINI)
<IfModule php8_module>
   php_flag engine On
   php_flag short_open_tag On
   php_value precision 12
   php_value output_buffering Off
   php_flag zlib.output_compression Off
   php_flag implicit_flush Off
   php_value serialize_precision 100
   php_value max_execution_time 300
   php_value max_input_time 300
   php_value max_input_vars 10000
   php_value memory_limit 2048M
   php_value error_reporting E_ALL & ~E_NOTICE
   php_flag display_errors On
   php_flag display_startup_errors Off
   php_flag log_errors On
   php_value log_errors_max_len 1024
   php_flag ignore_repeated_errors Off
   php_flag ignore_repeated_source Off
   php_flag report_memleaks On
   php_value error_log "error_log"
   php_value variables_order "EGPCS"
   php_flag register_argc_argv On
   php_value post_max_size 768M
   php_flag magic_quotes_sybase Off
   php_value default_mimetype "text/html"
   php_value include_path ".:/usr/lib/php:/usr/local/lib/php"
   php_value upload_max_filesize 128M
   php_value default_socket_timeout 60
   php_value smtp_port 25
   php_value odbc.defaultlrl 4096
   php_value odbc.defaultbinmode 1
   php_value mysql.connect_timeout 60
   php_flag mysql.trace_mode Off
   php_flag msql.allow_persistent On
   php_value msql.max_persistent -1
   php_value msql.max_links -1
   php_value pgsql.ignore_notice 0
   php_value pgsql.log_notice 0
   php_flag sybase.allow_persistent On
   php_value sybase.max_persistent -1
   php_value sybase.max_links -1
   php_value sybase.min_error_severity 10
   php_value sybase.min_message_severity 10
   php_flag sybase.compatability_mode Off
   php_value bcmath.scale 0
   php_value session.save_handler "files"
   php_value session.save_path "/tmp"
   php_flag session.use_cookies On
   php_value session.name "PHPSESSID"
   php_flag session.auto_start Off
   php_value session.cookie_lifetime 0
   php_value session.cookie_path "/"
   php_value session.serialize_handler "php"
   php_value session.gc_probability 1
   php_value session.gc_divisor 100
   php_value session.gc_maxlifetime 1440
   php_value session.entropy_length 0
   php_value session.cache_limiter "nocache"
   php_value session.cache_expire 180
   php_flag session.use_trans_sid Off
   php_value url_rewriter.tags "a=href,area=href,frame=src,input=src,form=,fieldset="
   php_value mssql.min_error_severity 10
   php_value mssql.min_message_severity 10
   php_flag mssql.compatability_mode Off
</IfModule>
<IfModule lsapi_module>
   php_flag engine On
   php_flag short_open_tag On
   php_value precision 12
   php_value output_buffering Off
   php_flag zlib.output_compression Off
   php_flag implicit_flush Off
   php_value serialize_precision 100
   php_value max_execution_time 300
   php_value max_input_time 300
   php_value max_input_vars 10000
   php_value memory_limit 2048M
   php_value error_reporting E_ALL & ~E_NOTICE
   php_flag display_errors On
   php_flag display_startup_errors Off
   php_flag log_errors On
   php_value log_errors_max_len 1024
   php_flag ignore_repeated_errors Off
   php_flag ignore_repeated_source Off
   php_flag report_memleaks On
   php_value error_log "error_log"
   php_value variables_order "EGPCS"
   php_flag register_argc_argv On
   php_value post_max_size 768M
   php_flag magic_quotes_sybase Off
   php_value default_mimetype "text/html"
   php_value include_path ".:/usr/lib/php:/usr/local/lib/php"
   php_value upload_max_filesize 128M
   php_value default_socket_timeout 60
   php_value smtp_port 25
   php_value odbc.defaultlrl 4096
   php_value odbc.defaultbinmode 1
   php_value mysql.connect_timeout 60
   php_flag mysql.trace_mode Off
   php_flag msql.allow_persistent On
   php_value msql.max_persistent -1
   php_value msql.max_links -1
   php_value pgsql.ignore_notice 0
   php_value pgsql.log_notice 0
   php_flag sybase.allow_persistent On
   php_value sybase.max_persistent -1
   php_value sybase.max_links -1
   php_value sybase.min_error_severity 10
   php_value sybase.min_message_severity 10
   php_flag sybase.compatability_mode Off
   php_value bcmath.scale 0
   php_value session.save_handler "files"
   php_value session.save_path "/tmp"
   php_flag session.use_cookies On
   php_value session.name "PHPSESSID"
   php_flag session.auto_start Off
   php_value session.cookie_lifetime 0
   php_value session.cookie_path "/"
   php_value session.serialize_handler "php"
   php_value session.gc_probability 1
   php_value session.gc_divisor 100
   php_value session.gc_maxlifetime 1440
   php_value session.entropy_length 0
   php_value session.cache_limiter "nocache"
   php_value session.cache_expire 180
   php_flag session.use_trans_sid Off
   php_value url_rewriter.tags "a=href,area=href,frame=src,input=src,form=,fieldset="
   php_value mssql.min_error_severity 10
   php_value mssql.min_message_severity 10
   php_flag mssql.compatability_mode Off
</IfModule>
# END cPanel-generated php ini directives, do not edit

Support says to generate or make changes in .htaccess use: multiPHP INI Editor -- but I'm pretty clueless in this dept.

[Aleksi "Lex" Kilpinen]

For that your host would be best to help you, they should know their own product best.

I do have to say though, it is also completely possible that you have simply reached the limits of your hosting plan and trying to limit bots is just a temporary measure to allow you to get back online. So now that you have seen this issue once, it's not a bad idea to look around for a hosting plan that might allow you more leeway. Especially since InMotion limits are slightly vague to begin with.

[Kindred]

Well, claudebot is responsible for triggering resource limits on Teo of my forums...

Everything points to things being fine after blocking it

[shawnb61]

I am not very familiar with MultiPHP INI...  I thought that was for updating php.ini, not .htaccess.

It is fair to ask your support for a specific example of using it to update .htaccess to block one bot.  A method that won't give you http 500 errors.  Start with claudebot...

[Arantor]

Well, claudebot is responsible for triggering resource limits on Teo of my forums...

Everything points to things being fine after blocking it

It also has plenty of evidence to support it not respecting robots.txt. Block it and move on.

[woolly bugger]

Well, claudebot is responsible for triggering resource limits on Teo of my forums...

Everything points to things being fine after blocking it

finally shut the bots down and everything is running smoothly again!