SMF overloading my server's CPU

[gasman]

I have received a warning from my hosting company (lunarpages) that a php script is overloading the server's CPU. I am on a shared hosting plan, and the php script is using huge amounts of the server's CPU at its peak.

Your account is utilizing excessive resources, causing a significant
degradation of services on the server. This is a shared environment and we
can not allow one user to utilize the majority of the resources on a server
as it affects all users adversely.

Load Averages: 6.71 1.25 0.8

Top Process %CPU 79.0 /usr/bin/php
Top Process %CPU 64.0 /usr/bin/php
Top Process %CPU 47.0 /usr/bin/php

This has been going on for well over a week. I have been warned that unless I modify or disable the script, my account will be suspended. They can't tell me exactly which script is causing the problem, and have suggested I disable scripts one by one and see if it makes a difference to CPU usage.

This seems to have happened after I upgraded to SMS1.1 RC1.

I have no other php scripts on my site, apart from the forum. I have a cgi script that I disabled but it made no differrence to my server's CPU usage.

I have a small-medium forum, 1000+ members, up to 60 or 70 online at peak times.

I have been moved to a lower-spec server until the situation is resolved. My server's specs: PHP version 4.4.1, Apache version 1.3.34 (Unix), MySQL version 4.0.25-standard

Status.php is here: http://www.good-music-guide.com/status.php (it changes rapidly, so try refreshing)


Googlebot has been active recently, indexing pages from the forum.

My questions are:

• does SMF use this much CPU power? (Surely not!)
• Why only since the upgrade to 1.1
• and what can I do to lower it?
• Would disabling the search function help? Or creating a fulltext index?

   

cheers
Rob Lang
www.good-music-guide.com/forum

[Compuart]

does SMF use this much CPU power? (Surely not!)

No, it shouldn't and in general doesn't.

Why only since the upgrade to 1.1

Not sure, but things should have to get faster, especially if you've enabled caching.

and what can I do to lower it?

Check README: Checklist for performance problems

Would disabling the search function help? Or creating a fulltext index?

Most definitely! Searching without a fulltext index can be hurting your server.

[gasman]

Thanks for your reply.

I am having continuing server CPU load problems.

My host asked me to disable the forum for a 24 hour period to see if it made a difference to CPU loads, and it seems to have made a big difference:

Hello,
Here are a set of updated statistics for your site. It looks like CPU usage dropped dramatically during this period. Please let us know if you have any questions.

Nov 28th 2005
good-music-guide.com 8.40 1.69 0.8

Nov 27th 2005
good-music-guide.com 0.53 0.32 0.1

Nov 26th 2005
good-music-guide.com 3.05 0.84 0.2

What I have done so far:
1. disabled guest access
2. Robots.txt to disallow googlebot
3. Disabled search (and created fulltext file for when I re-enable)

My status.php still shows numbers that are way out of range. Unfortunately, I have no idea what they mean or how to optimise them.

How do I enable caching?
What (and how) can I do to optimise my database? (I have very limited MySQL abilities).
Why on earth is the forum using >8% average CPU load?

I will be suspended from my account if I can't get this resolved.

Any advise MUCH appreciated.

cheers
Rob

[Ben_S]

Move host, your cu rrent host is rubbish would be my advice.

Nothing you have posted shows that it is your account that caused the high load.

Your forum is not causing any harm to their server, ask the to show you proper figures that show it is, I'll bet they can't, what you have posted thus far does not show that your forum is causing any harm.

Looks like an oversold host to me.... Which is a shame since they have a good reputation. Their MySQL is also very poorly configured.

Why not invite them here to respond themselves?

[gasman]

Thanks for the advice, Ben.

I agree, that Lunarpages are not providing enough specific data to enable me to identify which particular php script within SMF is causing the problem. However, it does seem that the forum is the problem.  Our test of 24 hours downtime reduced CPU load to acceptable amounts, and now that the forum is up again, CPU load is sky high again.

My concern is this: either it is a weird setting in Lunarpages version of php or MySQL, in which case I can't do anything from my side, or it is a funny setting on the forum. I am hoping somebody can identify a simple setting which eases the load on the CPU.

If all else fails, I may have to change hosts. Big hassle.

Is anyone else having this problem? In particular, is anyone else hosted with Lunarpages on a shared plan, with a SMF forum on their site?

cheers
Rob Lang

[gasman]

As an additional note, I received a email from Lunarpages saying that we should be careful using any formmail scripts, as they can be hijacked and used to send SPAM. I have been getting a huge amount of Non-Delivery Report emails lately from somebody promoting their Ultimate Online Pharmacy. I didn't think much of it - just thought it was email spoofing.

Is it possible someone could hack into the mail system of the forum and use it to send spam?

[Ben_S]

Unlikly, there are no known security issues.

Do you have any mod installed?

[Angie]

There is no known issue to use the smf´s mailfunction for DoS or DDoS attacks.
Btw: most DDos or DoS attacks make use of the insecure formular (GET instead POST and inadequate filtering classes) not of the mailfunction itself.

the problem "high load": ask your hosting company for detailed informations.

We have on our smallest server 3 big smf, 1 smf is highly modded (using ajax features).
And we use 2 big java chat servers (both with own httpd) on this server.
Additionaly we produce on this server "tickers".
Tickers are autogenerated pictures (approx. 70.000 a day) ...

Here our Server load:

Processor: AMD Sempron(tm) 2600+ (1833.218MHz)
Load averages: 0.08, 0.09, 0.08
Current processes: 124 (120 sleeping, 3 running, 1 zombie)
Processes by CPU: httpd2- (41) 2.4%, (other) (27) 0.1% 
Memory usage: 87.394% (436412k / 499360k)

as you can see, smf is not producing high server load.

lg angie

[gasman]

Thanks Angie and Ben

The only package I have installed is the MySQL 5.0.12 Compatibility Update, ironic as I have now been shifted to a lower spec server with MySQL v.4.

Good to see what CPU loads other SMF packages produce.

I'm SURE that my little forum couldn't be causing the high CPU load alone!

I have examined one of the email headers from the Non-Delivery Report emails. I'm getting hundreds of these each day in my catch-all (default) account. I'm no expert in email headers, but the first "Received:" header (with the earliest timestamp) seems to originate from good-music-guide.com - i.e. my website. I believe this part of the header is the most difficult to forge. Does this mean the SPAM is originating from my mailserver? I.e. someone has hacked into my account to send SPAM? Or am I being naiive?

Here is one of the hundreds of emails I'm getting each day, with headers. Naturally, the Return-Path ([email protected]) is not a valid email address of mine (though good-music-guide.com is my domain), so I simply assumed I was a victim of email spoofing, and these undeliverable emails bounce back to my catch-all.

However, good-music-guide.com appears as the very first Received From header (highlighted):

Return-Path: <[email protected]>
Received: (qmail 3350 invoked from network); 26 Nov 2005 05:27:15 -0000
Received: from sgrelayg1.core.theplanet.net (195.92.195.145)
  by thallium.smartgroups.com with SMTP; 26 Nov 2005 05:27:15 -0000
Received: from [211.162.131.165] (helo=143184544)
   by sgrelayg1.core.theplanet.net with smtp (Exim 3.22 #1)
   id 1Efrn4-0005cR-00
   for [email protected]; Sat, 26 Nov 2005 04:36:00 +0000
Received: from good-music-guide.com (144286640 [143387216])
   by glomedical.com (Qmailv1) with ESMTP id 8A93C69BCB
   for <[email protected]>; Fri, 25 Nov 2005 23:35:17 -0500
Date: Fri, 25 Nov 2005 23:35:17 -0500
From: Doctor <[email protected]>
X-Mailer: The Bat! (v2.00.2) Personal
X-Priority: 3
Message-ID: <[email protected]>
To: Mc <[email protected]>
Subject: The Ultimate Online Pharmaceutical
MIME-Version: 1.0
Content-Type: multipart/alternative;
   boundary="----------EFA2D01C28DB155"
X-Virus-Scanned: by Ameriserv.net Anti-Virus E-Gateway

This is a multi-part message in MIME format.

------------EFA2D01C28DB155
Content-Type: text/plain
Content-Transfer-Encoding: 7bit

Vlabgra $3.3
Leviltra $3.3
Ciaalis $3.7
Imityrex $16.4
Flomyax $2.2
Ultrlam $0.78
Vionxx $4.75
Ambfien $2.2
Valtium $0.97
Xanahx $1.09
Sogma $3
Merimdia $2.2 


visit our website
http://nergron.com/?YLRYEXXFEdR1RfVkd0RlxTQUBSQ11GREYfUVxZ

___
Best regards,
Online Pharmaceuticals

qwefgfgjs XFEdR1RfVkd0RlxTQUBSQ11GREYfUVxZ

------------EFA2D01C28DB155
Content-Type: text/html
Content-Transfer-Encoding: 7bit

<html>
<body>
<b>Vlagrra - $3.3 <br>
Lebvitra - $3.3<br>
Cialiqs - $3.7<br>
Imlitrex - $16.4<br>
Flmomax - $2.2<br>
Ulxtram - $0.78<br>
Viwoxx - $4.75 <br>
Amcbien - $2.2<br>
Vaglium - $0.97 <br>
Xaenax - $1.09<br>
Suoma - $3 <br>
Mseridia - $2.2</b><br>
<br>
<br>
<a href="http://nergron.com/?YLRYEXXFEdR1RfVkd0RlxTQUBSQ11GREYfUVxZ"><strong>visit our website</strong></a><br>

[Angie]

when you have doubts if you are a victim of DoS or not, ask your host to show you the /var/log/messages

lg angie

[MAZZA]

I think you have a received from header there because it's a bouncemail from your own mailserver. Not the original mail.

[gasman]

Thanks Angie and MAZZA

I have been assured by Lunarpages that the emails are not originating from my site, so its not a DoS attack.

They also agree that SMF usually does not use such high server resources. They have suggested that there may be some broken tables in the database.

Well, this is where I reach my limits of my expertise. I have used the forum's own "optimise tables" and the "find and repair errors" functions in the "forum maintenance" section of admin, but I suspect I need to investigate using phpMyAdmin. Could anyone lead me by the hand and show me how to check for broken tables using phpMyAdmin?

My status.php is still showing way high readings espescially in the  Load averages, and the opened vs open tables:

Basic Information
November 29, 2005, 12:08:45 PM
Operating System: CentOS release 3.6 (Final) 
Processor: Intel® Xeon(TM) CPU 2.80GHz (2791.087MHz)
Load averages: 5.20, 5.51, 5.66
Current processes: 204 (196 sleeping, 7 running, 1 zombie)
Processes by CPU: php ( 26.0%, x (2) 22.8%, spamd (3) 2.3%, httpd (85) 1.2%, (other) (53) 0.4% 
Memory usage: 97.843% (3774072k / 3857288k)
Swap: 0.698% (58544k / 8385920k) 

MySQL Statistics
MySQL 4.0.x
Connections per second:  5.6368 
Kilobytes received per second:  4.8524 
Kilobytes sent per second:  7.7874 
Queries per second:  85.7308 
Percentage of slow queries:  0 
Opened vs. Open tables:
(table_cache)  1527.6719 (should be <= 80) 
Table cache usage:
(table_cache)  1 (should be >= 0.5 and <= 0.9) 

Any ideas?

cheers
Rob

[Trekkie101]

Is this shared account?


Just, "Total Posts: 176590" would explain the trouble if this is a shared account.

[Angie]

Thanks Angie and MAZZA

I have been assured by Lunarpages that the emails are not originating from my site, so its not a DoS attack.

They also agree that SMF usually does not use such high server resources. They have suggested that there may be some broken tables in the database.

Well, this is where I reach my limits of my expertise. I have used the forum's own "optimise tables" and the "find and repair errors" functions in the "forum maintenance" section of admin, but I suspect I need to investigate using phpMyAdmin. Could anyone lead me by the hand and show me how to check for broken tables using phpMyAdmin?

My status.php is still showing way high readings espescially in the  Load averages, and the opened vs open tables:

Basic Information
November 29, 2005, 12:08:45 PM
Operating System: CentOS release 3.6 (Final) 
Processor: Intel® Xeon(TM) CPU 2.80GHz (2791.087MHz)
Load averages: 5.20, 5.51, 5.66
Current processes: 204 (196 sleeping, 7 running, 1 zombie)
Processes by CPU: php ( 26.0%, x (2) 22.8%, spamd (3) 2.3%, httpd (85) 1.2%, (other) (53) 0.4% 
Memory usage: 97.843% (3774072k / 3857288k)
Swap: 0.698% (58544k / 8385920k) 

MySQL Statistics
MySQL 4.0.x
Connections per second:  5.6368 
Kilobytes received per second:  4.8524 
Kilobytes sent per second:  7.7874 
Queries per second:  85.7308 
Percentage of slow queries:  0 
Opened vs. Open tables:
(table_cache)  1527.6719 (should be <= 80) 
Table cache usage:
(table_cache)  1 (should be >= 0.5 and <= 0.9) 

Any ideas?

cheers
Rob

even such many posts cant produce so high server loads!
check with phpMyAdmin or similiar tool your tables

Lg Angie

[Trekkie101]

even such many posts cant produce so high server loads!
check with phpMyAdmin or similiar tool your tables

Lg Angie

Your missing the point, shared server, lots of people, crammed on one server, each making loads of resource drains and this BIG site suddenly drives it up the wall.

This very community is not even twice as big and already was too big for a lesser powered dedicated server. Its on its own server just now.

[Ben_S]

How active is his forum though, number of posts aren't really that critical, it's how many people are active.

Also the celeron this site was on was certaily not a server

[gasman]

Yes, it is on a shared hosting plan - always has been, and never had this problem before.

It has a total of about 1300 members, but only about 50-60 online at once at peak times. Most of the members have not visited in weeks, or months. It has a total of 176000 posts. It gets about 500-700 new posts per day. There are no mods installed, ecept the MySQL v.5 compatibility mod. No funny stuff allowed - only txt and jpg attachments, no flash, simple gif/jpg avatars. No file sharing.

The vast bulk of the forum is simple php which outputs mainly text pages. Currently searching is disabled. It is on a Celeron sever now, only because they took it off the main servers pending review of CPU load problems.

The status.php output continues to show high CPU loads, even when there are only 5 or 6 members online. It also shows what I think are problems with the tables and table cache usage in the MySQL database, but I am not expert enough to interpret these.

cheers
Rob Lang

[gasman]

Out of interest, compare my forum (www.good-music-guide.com/forum) with this forum (http://www.simplemachines.org/community/index.php)

This forum has 3 times as many active users at any one time, about the same number of posts, and similar numbers of new posts per day.

yet, when we compare the corresponding status.php outputs, there is a huge difference between both CPU loads, as well as the database parameters.

www.good-music-guide.com/status.php  vs  www.simplemachines.org/community/status.php

What I am really after is for somebody to help me interpret the status.php output, and let me know what settings or changes I can make to optimise it, especially with regards to the tables and table cache numbers, which seem to be way off.

Please don't give up on me!

cheers
Rob Lang

[Joshua Dickerson]

Generally, MySQL doesn't make your CPU load that high. It usually eats memory faster than CPU. Some things that might eat your CPU are avatar resizing (quite possibly the culprit) and extreme output compression level.

What happens if you downgrade to 1.0? If you think it is MySQL, try creating a backup of the tables, dropping them, then recreating them. Maybe dropping the entire database and recreating it. I doubt it is MySQL because you said your host moved you to another server.

Might want to tell your host to use the thread cache. It is a pretty good feature. Why is InnoDB disabled? Ask your host to see the slow query log.

[gasman]

Thanks groundup!

I'll disable user-selected avatars for a while and see if it helps.

Can we degrade to an older version of SMF? I imagine only by deleted current files and re-installing an older version. Would that be compatible with the existing database?

Yes, the website (?including database) was changed to a new server by the host, so I doubt that I can fix it by simply re-installing the database from a backup.

I have had output compression off the entire time.

cheers
Rob