Problems/ High server load using SMF forum feature "image proxy"

Started by Wellwisher, February 17, 2018, 09:59:51 PM

Previous topic - Next topic

Wellwisher

I am having problems/ High server load using SMF forum feature "image proxy". It's causing stability issues on our VPS.

I spoke to our VPS technician who advised:

Quote

Looking over the access logs, it looks like your site is getting hit by a large number
of bots to your proxy.php file on your forum.


As a result, I disabled the SMF "image proxy feature" and cleared the 30GB+ of cached images (which this SMF 'image proxy' feature saved) and everything was running fine. Now the issue is my site is not completely secure (SSL). Any help or advice?

drewactual

if you're NOT running OPCache, install and do so... it will reduce the server load tremendously as it recycles and stores a scripts execution in RAM to be used over and over until you clear that cache or invalidate that file.  and a side effect of this is the speed of your site increase... like, silly speed increase.

note:  you can't just install OPCache, you have to configure and start it.   to save you lots of research time, these settings in your httpd, your php.ini, your user.ini, or your htaccess can be set like this for great performance (also note if you use your htaccess it has to be entered properly)

zend_extension=opcache.so
opcache.memory_consumption=128
opcache.max_accelerated_files=4000
opcache.revalidate_freq=60
opcache.fast_shutdown=1
opcache.enable_cli=1
opcache.use_cwd=1
opcache.load_comments=1
opcache.save_comments=1
opcache.revalidate_path=1
opcache.validate_timestamps=1

drewactual

also.. of course your site "isn't secure" after you cleared the proxy cache...

what it does is takes images that aren't secure your users post on your site, and makes a local copy of them which is secure- so you get your green lock. 

turn the proxy back on and install OPCache (if it is already installed, configure as provided above).

shawnb61

Yes, I experienced the same thing. 

(1) Regarding the CPU, there is a fix in the works in 2.1 that I am pretty sure will be ported back to 2.0.  You can find it here.  This resulted in a substantial improvement in resources used by the image proxy by bypassing the proxy for bots:
  https://github.com/SimpleMachines/SMF2.1/pull/4503

If you are brave enough to make similar changes to 2.0.15, this will definitely help; I have made the changes to my 2.0.15 forum.  Bear in mind that if that fix is baked into 2.0, you will need to back your changes out prior to applying the SMF update. 

(2) Even with the above fix, and verifying that bots no longer invoke the proxy, I still have issues with CPU when bots crawl.  I now think that the CPU issues are not 100% due to the proxy.  I have an additional theory/fix that I am looking into. 

(3) There will be cache maintenance in 2.1.  I am not sure whether this will be ported back to 2.0 or not.  I use a simple cron job to prune the cache nightly for my 2.0 forum.  Feel free to use it:
  https://github.com/sbulen/sjrbTools/blob/master/proxy-maint-cron.php

Crawls happen all the time...  Google seems to love my site & recrawls portions on a regular basis...  I never had an issue up thru 2.0.13.  The above changes help, but there is still some work to do. 
Address the process rather than the outcome.  Then, the outcome becomes more likely.   - Fripp

Wellwisher

@drewactual this sounds really good, I will try this out for sure. Thanks for saving me the time bud.  :D
@shawnb61 I am glad I am not the only one going through this issue. Seems like I have to take advance measures to help alleviate the problem.

Guys one thing I noticed is that large image hosts sometimes don't delivery "https" secure thumbnails for example:

Non secure address example:
http://www.imagehost.com/img.jpeg

But if we remove: "http://www."

Replace with: "https://"

So it looks like this example:
https://test.com/img.jpeg

Then most of the time the image host will deliver secure thumbnail image. One side effect is that those images that aren't https secure, won't display.  :P

Anyhoo, thanks for all the kind tips and tricks lads.   8)

albertlast

Ther is also a new pr for the proxy,
which should reduce the load of the proxy by providing cache information in the http header:
https://github.com/SimpleMachines/SMF2.1/pull/4550
Orginal came the code from here:
https://www.simplemachines.org/community/index.php?topic=558395.0

MobileCS

Nginx users - I've created a guide on how to Proxy Images without Proxy.php that will solve this issue.

https://www.simplemachines.org/community/index.php?topic=558946.0

Advertisement: