Advertisement:

Author Topic: Excessive CPU usage  (Read 2434 times)

Offline shnazzle

  • Semi-Newbie
  • *
  • Posts: 27
Excessive CPU usage
« on: June 15, 2019, 12:08:39 PM »
Has anybody experienced any extreme cpu usage since installing RC2?

HostGator shut us down last night for having 3 stints of 90+ seconds in the "Extreme CPU Usage" bounds.

I've shut down shoutbox, which is the only plugin I've got installed. But it never did this on 2.0.15.

What activities could cause a lot of cpu processing in 2.1 RC2?

Offline shawnb61

  • Developer
  • SMF Hero
  • *
  • Posts: 1,523
    • sbulen on GitHub
Re: Excessive CPU usage
« Reply #1 on: June 15, 2019, 12:24:55 PM »
The short answer is no, nothing specific about RC2 at this time. 

It would help to know a bit more about your environment - # of users, posts, # of active users you tend to see, PHP version.

Anything in the error logs?

Is there anything in your web access log that provides a clue?  (My first suspicion when I see this is always significant crawling activity...) 

Are you using https?  Are you using the image proxy?  How many files in the cache?

Do you have a lot of records in log_topics?  (This can get out of hand even on mid-sized forums: https://www.simplemachines.org/community/index.php?topic=212330.msg1667071#msg1667071)

Address the process rather than the outcome.  Then, the outcome becomes more likely.   - Fripp

Offline shnazzle

  • Semi-Newbie
  • *
  • Posts: 27
Re: Excessive CPU usage
« Reply #2 on: June 15, 2019, 12:45:11 PM »
- https
- no image proxy
- 522k posts
- 39k topics
- 60ish concurrent users max, usually more around 30
- Php 5.5, although I just changed it to 5.6
- Mysql Server version: 5.6.41-84.1 - Percona Server (GPL), Release 84.1, Revision b308619
- nothing in cache dir
- 181k rows in log_topics
- error logs have a few errors, nothing massive.
- there is a bot/apider/crawler called "link" with 85MB activity and 9,800+10 requests

Any help?

Offline shawnb61

  • Developer
  • SMF Hero
  • *
  • Posts: 1,523
    • sbulen on GitHub
Re: Excessive CPU usage
« Reply #3 on: June 15, 2019, 12:52:57 PM »
Wait...  How many requests from "link"?   Is that in scientific notation???
Address the process rather than the outcome.  Then, the outcome becomes more likely.   - Fripp

Offline albertlast

  • Development Contributor
  • Full Member
  • *
  • Posts: 589
Re: Excessive CPU usage
« Reply #4 on: June 15, 2019, 12:54:59 PM »
the stats side could create a big cpu usage on database side when "like" is enabled,
maybe this could be a reason,
the fix for this got into rc3.

Offline shnazzle

  • Semi-Newbie
  • *
  • Posts: 27
Re: Excessive CPU usage
« Reply #5 on: June 15, 2019, 02:42:32 PM »
Wait...  How many requests from "link"?   Is that in scientific notation???

Yes indeed it is, so, it's a lot.

the stats side could create a big cpu usage on database side when "like" is enabled,
maybe this could be a reason,
the fix for this got into rc3.
What's that now? Likes are causing issues at the minute?


Offline albertlast

  • Development Contributor
  • Full Member
  • *
  • Posts: 589
Re: Excessive CPU usage
« Reply #6 on: June 15, 2019, 03:08:45 PM »
the graphic didn't help,
since it didn't tell from what it show" is that php?" "is that mysql?" "something different?"

Offline shnazzle

  • Semi-Newbie
  • *
  • Posts: 27
Re: Excessive CPU usage
« Reply #7 on: June 15, 2019, 03:21:39 PM »
Sorry it wasn't really meant as a diagnosis thing.
The trough is where we were shut down. Then went back up in the wee hours and climbed progressively into Very High with onlh very little forum activity.
So it's just a general picture of; it's using a lot of CPU for little practical reason. At the time near the end of thr graph, I think 10 people were active.

I've asked HostGator for better logs of what exactly was taking up so much CPU in the times when we peaked

Offline Illori

  • Project Manager
  • SMF Legend
  • *
  • Posts: 51,431
Re: Excessive CPU usage
« Reply #8 on: June 15, 2019, 03:31:31 PM »
if they will not provide details which i doubt they will, i would start looking for a new host as they are oversold and most likely just trying to push you away anyway.

Offline shnazzle

  • Semi-Newbie
  • *
  • Posts: 27
Re: Excessive CPU usage
« Reply #9 on: June 15, 2019, 03:55:40 PM »
if they will not provide details which i doubt they will, i would start looking for a new host as they are oversold and most likely just trying to push you away anyway.

Way ahead of you.
I've found a few options. Unfortunately the best one is Windows-based, which I really don't like working on  :(

Offline shawnb61

  • Developer
  • SMF Hero
  • *
  • Posts: 1,523
    • sbulen on GitHub
Re: Excessive CPU usage
« Reply #10 on: June 15, 2019, 06:03:38 PM »
Wait...  How many requests from "link"?   Is that in scientific notation???
Yes indeed it is, so, it's a lot.

If you are getting so many hits from a bot that the # of hits must be listed in scientific notation, that's likely your problem... 

I use Hostgator.   All your logs are available on cPanel.  Even up-to-the-minute. 

I would download the current log & see if I could identify:
 - the IP range of the culprit
 - the activities the culprit is doing

When doing this, I sometimes load the logs directly into Excel for analysis.

And block the culprit, either by IP range or by user-agent if you can find a good identifier in there. 

Over the years, I have had occasional issues with bots & crawls.  I have a few agents blocked in my .htaccess file. 

EDIT:  Your logs are found in cPanel, under Metrics, Raw Access.  Awstats helps if the culprit has been around long enough to be included in those stats. 
Address the process rather than the outcome.  Then, the outcome becomes more likely.   - Fripp

Offline Sir Osis of Liver

  • SMF Hero
  • ******
  • Posts: 9,362
Re: Excessive CPU usage
« Reply #11 on: June 15, 2019, 06:12:50 PM »
i would start looking for a new host

Hostgator has been having problems lately.  Moved a large forum off there about a month ago, it was trashed and offline for a while, support was no help.  Take Illori's advice.



Offline shnazzle

  • Semi-Newbie
  • *
  • Posts: 27
Re: Excessive CPU usage
« Reply #12 on: June 15, 2019, 06:35:17 PM »
I've blocked a few of the heavy hitters.
After reading closely, the +10 is the number of robots.txt hits.
Not sure how that works.9800 made it through but 10 got caught by robots.txt?

Definitely moving hosting asap

Offline GigaWatt

  • The Smiley Guy
  • Support Specialist
  • SMF Hero
  • *
  • Posts: 2,187
  • Gender: Male
    • Macedonian electronics forum
Re: Excessive CPU usage
« Reply #13 on: June 17, 2019, 08:01:14 AM »
Not sure how that works.9800 made it through but 10 got caught by robots.txt?

That's scientific notation. It means 9800 * 10^10 (multiplied by 10 to the 10th power). It means you've got this many hits on robots.txt: 98.000.000.000.000.
"This is really a generic concept about human thinking - when faced with large tasks we're naturally inclined to try to break them down into a bunch of smaller tasks that together make up the whole."

"A 500 error loosely translates to the webserver saying, "WTF?"..."

Offline drewactual

  • Full Member
  • ***
  • Posts: 699
    • College Football Fan Site CFB51
Re: Excessive CPU usage
« Reply #14 on: June 17, 2019, 12:28:02 PM »
make sure you have a good cache set up, both for scripts and for static. 

a static cache will eliminate the processor punches on a txt file.  a good script cache like OPCache will eliminate the processor power needed to run a script every time it's called.  you said you upgraded to 5.6 from 5.5, and i wager you left OPCache behind. 

as an example, i dropped from redlining CPU regularly to less than 10% average CPU usage after implementing OPCache and with the same traffic.  it makes THAT much difference.  HostGator has it available if you ask for it- or- you may be able to turn it on and enter it's parameters/settings in your htaccess or local ini file.

i had something that went truly 'viral' once and it racked up over 30k users concurrently.  i sat watching it real time and recall thinking to myself "that God I set the cache up right, else that would have started a fire and burned the data center down the CPU would have been so hot!!!"...

Offline shnazzle

  • Semi-Newbie
  • *
  • Posts: 27
Re: Excessive CPU usage
« Reply #15 on: June 25, 2019, 01:28:30 PM »
make sure you have a good cache set up, both for scripts and for static. 

a static cache will eliminate the processor punches on a txt file.  a good script cache like OPCache will eliminate the processor power needed to run a script every time it's called.  you said you upgraded to 5.6 from 5.5, and i wager you left OPCache behind. 

as an example, i dropped from redlining CPU regularly to less than 10% average CPU usage after implementing OPCache and with the same traffic.  it makes THAT much difference.  HostGator has it available if you ask for it- or- you may be able to turn it on and enter it's parameters/settings in your htaccess or local ini file.

i had something that went truly 'viral' once and it racked up over 30k users concurrently.  i sat watching it real time and recall thinking to myself "that God I set the cache up right, else that would have started a fire and burned the data center down the CPU would have been so hot!!!"...

Unfortunately OPCache is not availble to us :( 
Interested now though!

Offline Aleksi "Lex" Kilpinen

  • A Peculiar Finn
  • Lead Support Specialist
  • SMF Super Hero
  • *
  • Posts: 18,696
  • Gender: Male
  • Don't worry, I'm n00b friendly
    • Aleksi.Kilpinen on Facebook
    • LexArma on GitHub
    • aleksi-kilpinen on LinkedIn
    • There's No Place Like 127.0.0.1
Re: Excessive CPU usage
« Reply #16 on: June 25, 2019, 01:38:15 PM »
Hostgator is owned by Endurance International Group (EIG) - Google it, learn of it, jump ship.

A Finnish Support Specialist
 Happily running multiple SMF 2.0 installations.
  Fooling around with an i7 990X @ 3,47Ghz / 12Gb / Win 10 x64 / 3840x2160


How you can help SMF

"Before you allow people access to your forum, especially in an administrative position, you must be aware that that person can seriously damage your forum.
 Therefore, you should only allow people that you trust, implicitly, to have such access." -Douglas

Offline drewactual

  • Full Member
  • ***
  • Posts: 699
    • College Football Fan Site CFB51
Re: Excessive CPU usage
« Reply #17 on: June 25, 2019, 07:32:07 PM »
Unfortunately OPCache is not availble to us :( 
Interested now though!

that's unfortunate.  and also something i would put my foot down about- as it is used wide, far, and deep across the interwebz.  the only reasonable excuse for not having it present already is if they're using one of the install flavors (fpm; mpm-worker; fastcgi ect) that conflict with zendOPCache..... and even then, there are alternatives.

Offline shawnb61

  • Developer
  • SMF Hero
  • *
  • Posts: 1,523
    • sbulen on GitHub
Re: Excessive CPU usage
« Reply #18 on: June 25, 2019, 10:21:33 PM »
Do you have SMF caching turned on?  Even if you don't have some of the more advanced caching like OpCache available, SMF's file-based caching produces very good results. 

Look under Admin | Configuration | Server Settings | Caching and set it to "Level 1 Caching" if it's not already. 

Another thing to try - if you are on https, and have the Image Proxy enabled, I would disable it until this is sorted out.  Crawls & the Image Proxy in 2.0.15 are bad news; they don't play nice. 

I use HostGator.  In general, I've been happy with them.  Never had a problem with their support.  I have had similar CPU issues during aggressive crawls.  BUT...  Only since 2.0.14...  Now, did 2.0.14 do something?  Or did Hostgator do something coincidentally at the time I upgraded to 2.0.14? 

I've taken a few steps to minimize the CPU issues (upgrading PHP helped; disabling the image proxy; enabling persistent DB connections).   
Address the process rather than the outcome.  Then, the outcome becomes more likely.   - Fripp

Offline Aleksi "Lex" Kilpinen

  • A Peculiar Finn
  • Lead Support Specialist
  • SMF Super Hero
  • *
  • Posts: 18,696
  • Gender: Male
  • Don't worry, I'm n00b friendly
    • Aleksi.Kilpinen on Facebook
    • LexArma on GitHub
    • aleksi-kilpinen on LinkedIn
    • There's No Place Like 127.0.0.1
Re: Excessive CPU usage
« Reply #19 on: June 26, 2019, 12:29:02 AM »
The thing about EIG is that it shops around for known shared hosts, keeps the branding, then “optimizes” cost structure of the hosts it buys out, fires great (expensive) support staff and migrate clients to a worse hardware infrastructure. Most often causing a slow degradation of service quality. EIG owns something like 60 different hosting brands.
A Finnish Support Specialist
 Happily running multiple SMF 2.0 installations.
  Fooling around with an i7 990X @ 3,47Ghz / 12Gb / Win 10 x64 / 3840x2160


How you can help SMF

"Before you allow people access to your forum, especially in an administrative position, you must be aware that that person can seriously damage your forum.
 Therefore, you should only allow people that you trust, implicitly, to have such access." -Douglas