Advertisement:

Author Topic: Forum using up all system resources?  (Read 53283 times)

Offline Goosemoose

  • SMF Friend
  • SMF Hero
  • *
  • Posts: 1,400
  • Gender: Male
    • Goosemoose.com
Forum using up all system resources?
« on: September 12, 2004, 05:13:37 PM »
hi guys,
 my host just shut down my site because the forum started to hog all their system resources "This was shut down due to the fact that your site has a heavy enough mysql load to shut down the physical server."
  They made a backup of the database but won't turn it back on until I fix the problem. But since its not running I don't know how to go about this. If you have any ideas let me know.

Offline [Unknown]

  • SMF Friend
  • SMF Master
  • *
  • Posts: 36,102
  • Gender: Male
Re: Forum using up all system resources?
« Reply #1 on: September 12, 2004, 05:37:21 PM »
Do you have the facility to check the database?

If you do, can you see if any of the topics have a "numViews" higher than logically possible?  Meaning, more than say 10000 but with few replies?

Was your host using MySQL 3 or MySQL 4?

-[Unknown]

Offline Goosemoose

  • SMF Friend
  • SMF Hero
  • *
  • Posts: 1,400
  • Gender: Male
    • Goosemoose.com
Re: Forum using up all system resources?
« Reply #2 on: September 12, 2004, 05:40:43 PM »
I'm downloading the database right now. I'll look at it and let you know what I find.

Offline Goosemoose

  • SMF Friend
  • SMF Hero
  • *
  • Posts: 1,400
  • Gender: Male
    • Goosemoose.com
Re: Forum using up all system resources?
« Reply #3 on: September 12, 2004, 05:47:02 PM »
The backup I downloaded that they created for me is full of MYD and MYI files of each table. When I try to view them in notepad I see only junk. What should I use to view this and what table should I be looking for. Usually I would just look around but since there are hundred of files now it's harder to find.

Offline [Unknown]

  • SMF Friend
  • SMF Master
  • *
  • Posts: 36,102
  • Gender: Male
Re: Forum using up all system resources?
« Reply #4 on: September 12, 2004, 05:50:52 PM »
You might be able to view those if you install MySQL on your own server, create a database, and copy these files into mysql\data\databasename.  Those are the raw MySQL database files....

-[Unknown]

Offline Goosemoose

  • SMF Friend
  • SMF Hero
  • *
  • Posts: 1,400
  • Gender: Male
    • Goosemoose.com
Re: Forum using up all system resources?
« Reply #5 on: September 12, 2004, 06:10:16 PM »
Okay, I am installing in on my machine right now. They were running mysql 3.0 as it is illegal for them to install 4.0 due to legal reasons.

Offline Ben_S

  • SMF Friend
  • SMF Super Hero
  • *
  • Posts: 11,725
  • xxx
Re: Forum using up all system resources?
« Reply #6 on: September 12, 2004, 06:39:15 PM »
If they think it's illegal to install MySQL 4, then they are talking rubbish and I highly doubt that they have a clue what they are talking about when saying your forum was a resource hog.
Liverpool FC Forum with 14 million+ posts.

Offline Goosemoose

  • SMF Friend
  • SMF Hero
  • *
  • Posts: 1,400
  • Gender: Male
    • Goosemoose.com
Re: Forum using up all system resources?
« Reply #7 on: September 12, 2004, 06:47:46 PM »
Here is what they told me (webintellects.net):
"We are running MySQl 3. Version 4 is restricted by its license not to be commercially distributed so our software provider could not upgrade if they wanted to. "

Offline kegobeer

  • SMF Friend
  • SMF Hero
  • *
  • Posts: 3,994
  • Gender: Male
  • I am ninja!
    • The Kazebeer Family Website
Re: Forum using up all system resources?
« Reply #8 on: September 12, 2004, 06:52:52 PM »
[offtopic]
Time for you to switch hosts, I think.  ;D

If they are using MySQL Pro then they probably can't just upgrade - but as a host I know I would stay on the edge to keep my customers happy.

There are plenty of hosts that stay at the forefront!
[/offtopic]
"The truth of the matter is that you always know the right thing to do. The hard part is doing it." - Norman Schwarzkopf
Posting and you (Click "WATCH THIS MOVIE")

Offline [Unknown]

  • SMF Friend
  • SMF Master
  • *
  • Posts: 36,102
  • Gender: Male
Re: Forum using up all system resources?
« Reply #9 on: September 12, 2004, 07:00:52 PM »
Sigh.  Stupid hosts.

MySQL 3 was licensed under the lesser GPL.  MySQL 4 is licensed under the GPL.  Yes, there's a HUGE difference there, but they've completely and utterly missed it.

You see, MySQL 4 just cannot be sold.  Period.  End of story.  However, open source would not exist if people could not profit or at least break even off of it.  Don't think we're communists - only people who don't understand do.

You can sell SERVICES for MySQL, such as hosting, without problem.  This is the INTENT.  You're SUPPOSED to sell services.  That's how it WORKS.

http://www.mysql.com/company/legal/licensing/opensource-license.html

Even if it weren't, it's $595.  Compare that with Microsoft SQL.  So even then, how cheap is your host?

GPL here:
http://www.gnu.org/copyleft/gpl.html

LGPL here:
http://www.gnu.org/copyleft/lesser.html

-[Unknown]

Offline Ben_S

  • SMF Friend
  • SMF Super Hero
  • *
  • Posts: 11,725
  • xxx
Re: Forum using up all system resources?
« Reply #10 on: September 12, 2004, 07:03:35 PM »
I imagine the problem is they use something crappy like Ensim which comes with MySQL 3x and they for whatever reason cant/wont upgrade to MySQL 4x themselves.
Liverpool FC Forum with 14 million+ posts.

Offline Goosemoose

  • SMF Friend
  • SMF Hero
  • *
  • Posts: 1,400
  • Gender: Male
    • Goosemoose.com
Re: Forum using up all system resources?
« Reply #11 on: September 12, 2004, 07:32:24 PM »
But everything has been running fine for months. I haven't changed a thing since I installed smf months ago. They are using control maestro,  WEBppliance Pro 3.5.21-10
 
 

Offline [Unknown]

  • SMF Friend
  • SMF Master
  • *
  • Posts: 36,102
  • Gender: Male
Re: Forum using up all system resources?
« Reply #12 on: September 12, 2004, 07:48:39 PM »
SMF was designed with MySQL 4 in mind.  The problem is, MySQL 3 and 4 act very differently in some situations, and what is slow on one may be fast on the other.  In fact, there have been situations where we have had to pick between two methods - one fast on 4, another fast on 3.  Obviously, the descision could only logically be made to go with the current version (4) not the old and no longer available 3.

It still should not be slow, persay, but I can understand it performing worse.  Additionally, MySQL 4 is simply faster overall, and when you account for query caching it is faster indeed.  If your host is really concerned about their server's performance, they should be willing to upgrade MySQL - otherwise, they really are not worth your money.

MySQL 4 has been available for quite a while now.  Your host probably just over-reacted to a peak period.

-[Unknown]

Offline Goosemoose

  • SMF Friend
  • SMF Hero
  • *
  • Posts: 1,400
  • Gender: Male
    • Goosemoose.com
Re: Forum using up all system resources?
« Reply #13 on: September 12, 2004, 07:51:07 PM »
They took the entire site down. Then after I got a hold of them they put it back up and supposedly the resources shot up immediately so they took it down again. Right now they are waiting on me to fix the problem which I can't really do.

Offline [Unknown]

  • SMF Friend
  • SMF Master
  • *
  • Posts: 36,102
  • Gender: Male
Re: Forum using up all system resources?
« Reply #14 on: September 12, 2004, 07:52:23 PM »
Edit Settings.php and put it in maintenance mode.  See if it "shoots up again" then too.

Because that would obviously be a crock :P.

-[Unknown]

Offline Goosemoose

  • SMF Friend
  • SMF Hero
  • *
  • Posts: 1,400
  • Gender: Male
    • Goosemoose.com
Re: Forum using up all system resources?
« Reply #15 on: September 12, 2004, 08:04:44 PM »
I have put it in maitenance mode and asked them to turn it back on. Hopefully they will do it soon.

Offline Amacythe

  • SMF Friend
  • SMF Super Hero
  • *
  • Posts: 19,060
  • Gender: Female
Re: Forum using up all system resources?
« Reply #16 on: September 12, 2004, 08:38:20 PM »
Why go through all this trouble... the easiest solution would be to find a host that doesn't have their collective heads up their... ahem... find a host that knows what they are doing.

Offline Goosemoose

  • SMF Friend
  • SMF Hero
  • *
  • Posts: 1,400
  • Gender: Male
    • Goosemoose.com
Re: Forum using up all system resources?
« Reply #17 on: September 12, 2004, 10:52:04 PM »
Why go through all this trouble... the easiest solution would be to find a host that doesn't have their collective heads up their... ahem... find a host that knows what they are doing.

I've been using this host for years and haven't had a problem. I think the person who got stuck as tech support on a sunday just isn't very knowledgeable. I usually get much better answers and quick response. I'll find out when I call my usual contact tommorrow. I have hundreds of thousands of messages and over 4,000 users so it's not a small board. Hopefully I can get it resolved soon, otherwise I will definately think of changing.


Offline Goosemoose

  • SMF Friend
  • SMF Hero
  • *
  • Posts: 1,400
  • Gender: Male
    • Goosemoose.com
Re: Forum using up all system resources?
« Reply #18 on: September 13, 2004, 01:09:14 AM »
Do you have the facility to check the database?

If you do, can you see if any of the topics have a "numViews" higher than logically possible?  Meaning, more than say 10000 but with few replies?

Was your host using MySQL 3 or MySQL 4?

-[Unknown]

Okay, I finally got a server running on my local machine and checked out the database. The highest sumViews was 5600 and there were 400 replies to it so I don't think there were any rogue threads. Any other ideas?

Offline [Unknown]

  • SMF Friend
  • SMF Master
  • *
  • Posts: 36,102
  • Gender: Male
Re: Forum using up all system resources?
« Reply #19 on: September 13, 2004, 01:45:25 AM »
Hmm.. at the time it was dropped, how many people were listed in the smf_log_online table?

-[Unknown]

Offline Goosemoose

  • SMF Friend
  • SMF Hero
  • *
  • Posts: 1,400
  • Gender: Male
    • Goosemoose.com
Re: Forum using up all system resources?
« Reply #20 on: September 13, 2004, 01:56:09 AM »
It looks like there were only 22 on at the time they took it down. The top for the day was 48. Nothing abnormal.

Offline [Unknown]

  • SMF Friend
  • SMF Master
  • *
  • Posts: 36,102
  • Gender: Male
Re: Forum using up all system resources?
« Reply #21 on: September 13, 2004, 02:00:40 AM »
I don't suppose you could get them to look at the access log, load averages, or anything else from that time?  I can't imagine how else it could be suddenly using up enough resources to physically damage the server.  Unless the tech on duty was wasted or something.

-[Unknown]

Offline Goosemoose

  • SMF Friend
  • SMF Hero
  • *
  • Posts: 1,400
  • Gender: Male
    • Goosemoose.com
Re: Forum using up all system resources?
« Reply #22 on: September 13, 2004, 02:03:06 AM »
I could sure ask, what exactly do you want me to ask them to look for?

Offline [Unknown]

  • SMF Friend
  • SMF Master
  • *
  • Posts: 36,102
  • Gender: Male
Re: Forum using up all system resources?
« Reply #23 on: September 13, 2004, 02:11:18 AM »
I could sure ask, what exactly do you want me to ask them to look for?

Well, one possibility might be someone contacting the forum many times per second over a period of more than a few minutes.  I was consider that almost a DoS attack, and that's why I asked you to check numViews - this has been done before by people with malicious intents to other forums, and can cause high load.

I'm also wondering how bad the load averages really were.  On this server, they're currently.. "load average: 0.34, 0.71, 0.66" which, while not totally ideal of course (they used to be lower, strangely...) are perfectly fine.  Your first post, at least, made it sound like your account was disabled for fear of physical damage to the server.  For this, I might expect the load averages to be higher than 20 or so - although, I am not by any means a linux expert.  Being able to see the load average might give me an idea of how bad the issue is.

You might also want to ask if there is ANYTHING your host change on your server in the last week.  It could be that they even changed some minor setting in php.ini - obviously, I don't think such could have caused high-load problems, but it's possible and moreover, it's a start.

Right now, because you say your forum was performing okay in the past and you were not shut down by your host before now, I have nothing else to really go on.  It would be considerably different if you knew your host had changed something, or if you had had some problems before... but sudden problems are a lot harder to track down.

Of course, there is another question too (but I assume you've already asked yourself this..) have *you* changed anything in SMF's configuration or an .htaccess file in the past while yourself?  I doubt that's a problem either... but again, it may at least be something to go off of.

-[Unknown]

Offline A.M.A

  • SMF Friend
  • SMF Hero
  • *
  • Posts: 2,133
  • Gender: Male
  • ..
Re: Forum using up all system resources?
« Reply #24 on: September 13, 2004, 04:56:14 AM »
would not 'persistent connection' when activated on a non DS causes a high load!
Really sorry .. real life is demanding my full attention .. will be back soon hopefully :)

Offline Goosemoose

  • SMF Friend
  • SMF Hero
  • *
  • Posts: 1,400
  • Gender: Male
    • Goosemoose.com
Re: Forum using up all system resources?
« Reply #25 on: September 13, 2004, 01:46:40 PM »
Okay the forum is in maintenance mode and they have turned mysql back on. According to the tech there have been no changes over the past weel that he is "aware of". It's at http://www.goosemoose.com/rfc if you want to take a look at it. I am still not taking it out of maintenance mode yet though.

Offline Goosemoose

  • SMF Friend
  • SMF Hero
  • *
  • Posts: 1,400
  • Gender: Male
    • Goosemoose.com
Re: Forum using up all system resources?
« Reply #26 on: September 13, 2004, 02:21:29 PM »
Okay, I took it out of maintenance mode and all looks ok. Note that I still made no changes to the site or db.

Offline Goosemoose

  • SMF Friend
  • SMF Hero
  • *
  • Posts: 1,400
  • Gender: Male
    • Goosemoose.com
Re: Forum using up all system resources?
« Reply #27 on: September 13, 2004, 02:24:14 PM »
Strike that, they took it down within 5 minutes of it being out of mainentance mode, it started doing the same thing.

Offline [Unknown]

  • SMF Friend
  • SMF Master
  • *
  • Posts: 36,102
  • Gender: Male
Re: Forum using up all system resources?
« Reply #28 on: September 13, 2004, 09:13:28 PM »
Have you considered upgrading to RC1?

-[Unknown]

Offline Goosemoose

  • SMF Friend
  • SMF Hero
  • *
  • Posts: 1,400
  • Gender: Male
    • Goosemoose.com
Re: Forum using up all system resources?
« Reply #29 on: September 13, 2004, 09:44:55 PM »
I'm in the process of doing it right now. I had uploaded my beta 4 files on accident so now I'm going back up then to RC1. Do you think that will fix the problem? Has anyone else had anything similar?

Offline [Unknown]

  • SMF Friend
  • SMF Master
  • *
  • Posts: 36,102
  • Gender: Male
Re: Forum using up all system resources?
« Reply #30 on: September 13, 2004, 11:26:07 PM »
Well, the search functionality was a bit slower than desirable in Beta 5.  Still, if you had no problems until now it doesn't make any sense.

-[Unknown]

Offline Goosemoose

  • SMF Friend
  • SMF Hero
  • *
  • Posts: 1,400
  • Gender: Male
    • Goosemoose.com
Re: Forum using up all system resources?
« Reply #31 on: September 14, 2004, 04:10:15 PM »
Well I upgraded to RC1 and turned off maintenance mode and the forum crashed the server as soon as it was taken out of maintenance mode. I saw it myself as I could load the first few pages then it started going slower and slower until smf told me the db connection was lost. The host claims nothing has changed recently and I know I haven't changed any files for months. Does anybody have any other ideas? They are telling me now that they probably won't turn it back on.

Offline Grudge

  • SMF Friend
  • SMF Super Hero
  • *
  • Posts: 10,742
  • Gender: Male
  • Unofficial nuisance
Re: Forum using up all system resources?
« Reply #32 on: September 14, 2004, 04:16:01 PM »
You 100% sure that persistant connections isn't somehow on?
I'm only a half geek really...

Offline Goosemoose

  • SMF Friend
  • SMF Hero
  • *
  • Posts: 1,400
  • Gender: Male
    • Goosemoose.com
Re: Forum using up all system resources?
« Reply #33 on: September 14, 2004, 04:45:08 PM »
I just checked and smf has persistent connection unchecked

Offline Tony Reid

  • SMF Friend
  • SMF Hero
  • *
  • Posts: 4,151
  • Gender: Male
    • @AbsoluteBreeze on Twitter
    • www.fertilityfriends.co.uk
Re: Forum using up all system resources?
« Reply #34 on: September 14, 2004, 05:17:17 PM »
It sounds like an issue with the server to me.

I'd move hosts.
Tony Reid


My Big Board
www.FertilityFriends.co.uk/forum - An SMF powered forum with over 5 million posts

Offline [Unknown]

  • SMF Friend
  • SMF Master
  • *
  • Posts: 36,102
  • Gender: Male
Re: Forum using up all system resources?
« Reply #35 on: September 14, 2004, 07:02:30 PM »
If you want to give me FTP access I'll try to figure out what's eating load...?

-[Unknown]

Offline Goosemoose

  • SMF Friend
  • SMF Hero
  • *
  • Posts: 1,400
  • Gender: Male
    • Goosemoose.com
Re: Forum using up all system resources?
« Reply #36 on: September 15, 2004, 12:02:25 AM »
I imagine the problem is they use something crappy like Ensim which comes with MySQL 3x and they for whatever reason cant/wont upgrade to MySQL 4x themselves.
Yup , they were using Ensim!

Offline Amacythe

  • SMF Friend
  • SMF Super Hero
  • *
  • Posts: 19,060
  • Gender: Female
Re: Forum using up all system resources?
« Reply #37 on: September 15, 2004, 01:07:04 AM »
You say they were using Ensim, does this mean they have upgraded or are going to?

Offline Goosemoose

  • SMF Friend
  • SMF Hero
  • *
  • Posts: 1,400
  • Gender: Male
    • Goosemoose.com
Re: Forum using up all system resources?
« Reply #38 on: September 16, 2004, 01:33:54 AM »
well they still are using it but I moved mine to another server running plesk where i have my own virutual server, where everything seems to be running okay, though a bit slow.

Offline [Unknown]

  • SMF Friend
  • SMF Master
  • *
  • Posts: 36,102
  • Gender: Male
Re: Forum using up all system resources?
« Reply #39 on: September 16, 2004, 01:42:15 AM »
well they still are using it but I moved mine to another server running plesk where i have my own virutual server, where everything seems to be running okay, though a bit slow.

Can I see a phpinfo?  MySQL version information (not the stuff on the phpinfo!)?

-[Unknown]

Offline Goosemoose

  • SMF Friend
  • SMF Hero
  • *
  • Posts: 1,400
  • Gender: Male
    • Goosemoose.com
Re: Forum using up all system resources?
« Reply #40 on: September 16, 2004, 02:03:44 AM »
http://209.216.209.81/phpinfo.php
mysql is still version 3.

Offline [Unknown]

  • SMF Friend
  • SMF Master
  • *
  • Posts: 36,102
  • Gender: Male
Re: Forum using up all system resources?
« Reply #41 on: September 16, 2004, 03:13:10 AM »
http://209.216.209.81/phpinfo.php
mysql is still version 3.

Egad!  They have so many things compiled in.. ugh.

What do you/are you going to use the server for?  Mainly SMF and static content?  If so, you might consider upgrading to PHP 5.  Even if not, you should upgrade to 4.3.8 (and soon a 4.3.9) which includes some security fixes ;).

The configure line also includes a lot of things it might not need.  I'd take out --enable-mbrege, --enable-dio (never heard of that?), --enable-trans-sid, and --with-ncurses at least.

It's also nice to note that MySQL 4 is significantly faster, and is the current stable release.  MySQL 3 is now an "archived" download.

http://downloads.mysql.com/archives.php?p=mysql-3.23

MySQL 4 supports the all-powerful-and-beloved feature we call "query caching", which is just generally awesome.

You seem to be on Apache 2, which is good.

Also, since you're on a dedicated server, you might try using a persistant connection.  This should be faster too ;).

-[Unknown]

Offline Goosemoose

  • SMF Friend
  • SMF Hero
  • *
  • Posts: 1,400
  • Gender: Male
    • Goosemoose.com
Re: Forum using up all system resources?
« Reply #42 on: September 17, 2004, 03:40:55 AM »
Yeah i'm just using smf and static content. I'll look into the upgrades. Check out the forum and tell me what you think about the speed.