Optimizing board for minimum server impact.

Started by lyrae, April 21, 2005, 02:23:48 PM

Previous topic - Next topic

lyrae

Hi there,

I have an SMF board that I just set up for a site that launched today. The site consists only of the message board. We expect to have about 7000 members within a few weeks. Already today our host has expressed (in the form of account suspension) extreme objection to the amount of server load the board is creating. At the time they suspended the account the board had about 200 members online. We don't expect this amount to be a normal thing (maybe 100 at higher points of the day), but we should account for more rather than less.

We are not on a dedicated plan because the site is run by a non-profit collective and has no ads but rather is supported by donation -- we're not sure we can afford a dedicated server. However we have a supposedly good plan with 600GB transfer and 30GB of space, so we thought we would be okay. However I suppose server load is a bit of a subjective thing, and they don't seem happy.

My question is, how can we optimize the board settings for better performance? What settings and features should we consider disabling to avoid resource suck? How should I set the software to optimize tables, and what difference does this make? I guess ultimately the question is, how can I make this board's footprint on the server smaller?

[Unknown]

Quote from: lyrae on April 21, 2005, 02:23:48 PM
I have an SMF board that I just set up for a site that launched today. The site consists only of the message board. We expect to have about 7000 members within a few weeks. Already today our host has expressed (in the form of account suspension) extreme objection to the amount of server load the board is creating.

Interesting.  200 shouldn't be that bad, for a newish forum.  Would you be willing to post a link, so I can look at your statistics?

Can you post the version information in Admin -> Support and Credits, and some specs about your server?

QuoteWe are not on a dedicated plan because the site is run by a non-profit collective and has no ads but rather is supported by donation -- we're not sure we can afford a dedicated server. However we have a supposedly good plan with 600GB transfer and 30GB of space, so we thought we would be okay. However I suppose server load is a bit of a subjective thing, and they don't seem happy.

Do you know how many other people are on your same server?  Can you ask them what the server load was, exactly (I'm asking for load averages)?

QuoteMy question is, how can we optimize the board settings for better performance? What settings and features should we consider disabling to avoid resource suck? How should I set the software to optimize tables, and what difference does this make? I guess ultimately the question is, how can I make this board's footprint on the server smaller?

There are many things you can change, but what you should do greatly depends on MySQL version and the number of posts and members you currently have.

-[Unknown]

lyrae

#2
Thanks so much for replying -- I will get some server specs and get back to you this afternoon or evening. I definitely don't know how many sites are hosted on the same server, but it will be interesting to hear. I'm at work, which makes dealing with the host while receiving confused emails from users a little stressful. I had to dry out the armpits of my shirt in the office bathroom's  hand dryer. HA

The URL is http://www.stillepost.ca/ and here's the support & credits info.

Version Information:
Forum version: SMF 1.0.3 (more detailed)
Current SMF version: SMF 1.0.3
PHP version: 4.3.8
MySQL version: 4.0.24-standard
Server version: Apache/2.0.50 (Fedora)
GD version: bundled (2.0.23 compatible)

[Unknown]

Your forum shouldn't have had a negative effect on your server with those statistics.  Strange.  In any case, you might try running the following queries in phpMyAdmin.  What is phpMyAdmin?

ALTER TABLE smf_log_search
TYPE=InnoDB;
ALTER TABLE smf_log_online
TYPE=InnoDB;
ALTER TABLE smf_log_topics
TYPE=InnoDB;
ALTER TABLE smf_log_floodcontrol
TYPE=InnoDB;

See if that makes you see improved performance.  You may also wish to turn on "Display time taken to create every page" under "Edit Features and Option" to see what that shows.

Also, make sure you have "Check the avatar size every time it is displayed" under Edit Features and Options off.

-[Unknown]

[Unknown]


lyrae

OK. Things have been crazy -- 1000 members in two days -- and I've been laying low with the host because I'm hoping they'll be watching for traffic to level out after launching, so I haven't wanted to ask them about the load balance. I just don't want their response to be, "Now that you mention it, your site really does need an upgrade that will cost three times as much per month." I have the avatar size option off (I think it must have been off by default?) and I have just enabled the option for displaying time to build pages. I'll let you know what it says.

Can you explain those SQL queries a little more? I totally do trust you -- you're the man -- but I'd like to know what I'm doing before I execute them.

Thanks SO MUCH for your responses. I am moving from adminning PHPBB on another site and there have been many moments in the last two days when I've just had to reflect on how primitive that software is and what a relief SMF is.

I do have a couple of other questions (I get vertical scrollbars on posts longer than 3 lines over here, not on my site but others have on mine; sometimes settings just don't save, like stickying posts and changing 'news' text, but no errors are returned) -- should I start separate threads?

[Unknown]

Quote from: lyrae on April 24, 2005, 01:13:55 PM
OK. Things have been crazy -- 1000 members in two days

Congratulations.  This is a rate many other administrators would kill for.

QuoteI haven't wanted to ask them about the load balance.

Well, if you're on a shared server the load balance is going to be affected by everyone on the server.  They really couldn't say that without a good amount of evidence, which farnkly they can't have quite yet, but far too many hosts do.  Should they, there are those in Hosts and Hosting who know that SMF isn't usually to blame directly, without bizarre settings.

You may be able to find your load average yourself.  To try, create a file on your server with a .php extension and the following contents:

<?php

require_once('/path/to/SSI.php');

header('Content-Type: text/plain');

// Display load averages and server uptime.
system('uptime');

echo 
"\n";

// Memory usage.
system('free');

echo 
"\n";

// Show MySQL processes running for the forum.
$result db_query("
   SHOW PROCESSLIST"
__FILE____LINE__);
while (
$row mysql_fetch_assoc($result))
   
printf('%-8s%4s %-10s %s' "\n"$row['Command'], $row['Time'], $row['State'], $row['Info']);
mysql_free_result($result);

?>


Which will show other information as well, which would be very useful.  None of it should be sensitive, but if you prefer it's fine to send this as a link via pm, if you'd be willing.

QuoteI have the avatar size option off (I think it must have been off by default?) and I have just enabled the option for displaying time to build pages. I'll let you know what it says.

Yes, that's the default.  Your page load times are not horrible, but at the moment the load times here are about a third what you're getting, and we have just as many people online and more posts and members, currently.  While SMF 1.1 (used here) is more effecient than 1.0.3, this suggests to me that you are not the sole cause of any problems your host may blame you for.

After all, this server is dedicated, but it's only a Celeron 1700 with 512 megs of ram.  That's getting low end in today's market, I think.

QuoteCan you explain those SQL queries a little more?

The above queries will change the types of the named tables.  InnoDB is more efficient for tables that have data added to them more often, while MyISAM (the default) is faster for data that is relatively static.  MySQL 4.0.24 should support InnoDB fine, and the above changes should cause a small, but perhaps noticeable, improvement in performance.

QuoteI do have a couple of other questions (I get vertical scrollbars on posts longer than 3 lines over here, not on my site but others have on mine; sometimes settings just don't save, like stickying posts and changing 'news' text, but no errors are returned) -- should I start separate threads?

What browser are you using?  There is a known problem with Internet Explorer 5 and Safari, which I haven't had a chance to work out yet.  I'm guessing it's caused by the line-height, which is a shame.  If anyone is seeing scrollbars on your site where they shouldn't be (they are added when the post's content is too wide for the window) what browsers are they using?

Settings won't save how?  Does the resulting page just not load?  Timeout?  If it doesn't load, that would make it sound like Apache is segfaulting.  This is very bad and wouldn't be the fault of SMF or any other software you're running; if it's happening, your host should recompile Apache and PHP.

They should know how to find out if it's happening.  For one, it will show in the Apache error log.

If, on the other hand, the page loads fine but nothing happens - are you sure it isn't a collision?  Meaning, two people stickying the topic at the same time (which would reverse the first sticky to no net effect), and similar?

-[Unknown]

Ben_S

600GB transfer and 30GB, sounds like the server is horribly oversold, unless you are paying in the region of $50 - $60 / month.
Liverpool FC Forum with 14 million+ posts.

lyrae

Quote from: [Unknown] on April 24, 2005, 08:15:09 PM
You may be able to find your load average yourself.  To try, create a file on your server with a .php extension and the following contents:
[snip]
Which will show other information as well, which would be very useful.  None of it should be sensitive, but if you prefer it's fine to send this as a link via pm, if you'd be willing.

Okay -- I have done that and PMed you. I will also run those SQL queries tomorrow.

Quote
QuoteI do have a couple of other questions (I get vertical scrollbars on posts longer than 3 lines over here, not on my site but others have on mine; sometimes settings just don't save, like stickying posts and changing 'news' text, but no errors are returned) -- should I start separate threads?

What browser are you using?  There is a known problem with Internet Explorer 5 and Safari, which I haven't had a chance to work out yet.  I'm guessing it's caused by the line-height, which is a shame.  If anyone is seeing scrollbars on your site where they shouldn't be (they are added when the post's content is too wide for the window) what browsers are they using?

Aha -- I use Safari at home and I get the vertical scrollbars on this thread in particular. The person on my site who complained said he was using the most recent Netscape (it's been so long since I've heard someone say they used Netscape that my first reaction was to tell him to try another browser). I am using Firefox right now and am not having the problem. I enjoy Safari but maybe it's time to make a permanent transition.

QuoteSettings won't save how?  Does the resulting page just not load?  Timeout?  If it doesn't load, that would make it sound like Apache is segfaulting.  This is very bad and wouldn't be the fault of SMF or any other software you're running; if it's happening, your host should recompile Apache and PHP.

They should know how to find out if it's happening.  For one, it will show in the Apache error log.

If, on the other hand, the page loads fine but nothing happens - are you sure it isn't a collision?  Meaning, two people stickying the topic at the same time (which would reverse the first sticky to no net effect), and similar?

Well, I would say it's possible that collisions are taking place, but it's such an odd little problem and it's not limited to things like stickies. For example, once in a while the last news item we had up will come up visible again, even though we replaced it on Friday. It shouldn't be a browser-cache problem, I wouldn't think, because the page would be built dynamically. Right? So segfaults might be a consideration? Hm.

Another example -- and I can't be sure it's related, but it's another weird quirk, is that a theme that was created and is set both in Theme Settings and from what I can tell, in the DB (at smf_themes as image_url for that style) as using the default style's images. Yet when pages are being built, there are a tonne of errors going in my error log because the images directory under the new theme doesn't exist. When I look in the page source of referrers it appears to be requesting them from the default theme's directory as it should be. So why would there be requests of the new theme's directory which does not exist? I feel like my site's possessed! Based on what I can find in the DB (thanks for a DB structure that makes total intuitive sense, by the way) and in the admin panel settings, everything is OK -- these errors make no sense. I can't figure out where the requests are coming from. Am *I* even making sense?

I've put up a script to show the tail 30 of my error log, and I'll PM you the URL for that as well. I don't see anything that's obviously a segfault, but maybe you will? All of the current entries are requests for this new theme's images directory. Sigh.

I really appreciate all your help, by the way.

lyrae

So... my account has been suspended again and they won't restore it.

I have no idea what to do.

Your account was suspended becuase of overloading the server. Here is the log file

11:37:36  up 4 days,  8:45,  1 user,  load average: 9.95, 6.09, 4.29
380 processes: 343 sleeping, 27 running, 10 zombie, 0 stopped
CPU states:  cpu    user    nice  system    irq  softirq  iowait    idle
           total    0.2%   75.4%   24.0%   0.0%     0.0%    0.0%  100.0%
           cpu00    0.3%   75.4%   24.1%   0.0%     0.0%    0.0%    0.0%
           cpu01    0.0%    0.0%    0.0%   0.0%     0.0%    0.0%  100.0%
Mem:  2035800k av, 1987320k used,   48480k free,       0k shrd,  104020k buff
       568080k active,            1239000k inactive
Swap: 4192956k av,   58704k used, 4134252k free                  973676k cached

  PID USER     PRI  NI  SIZE  RSS SHARE STAT %CPU %MEM   TIME CPU COMMAND
4968 root      19   0  1496 1496   800 R    12.4  0.0   0:13   0 top
24612 root      26  10 44440  43M 44352 S N   8.5  2.1   1:02   0 /usr/sbin/httpd
-DSSL -DSSL.canttouchthis.net -DSSL.japlet.com -D
5879 admin91   31  10  7576 7568  4672 S N   8.5  0.3   0:00   0 /usr/bin/php
5876 admin91   26  10  7572 7564  4664 S N   7.8  0.3   0:00   0 /usr/bin/php
5881 admin91   31  10  7572 7564  4672 S N   7.4  0.3   0:00   0 /usr/bin/php
5888 admin91   31  10  5760 5752  4308 R N   6.4  0.2   0:00   0 /usr/bin/php
5883 admin91   30  10  5752 5744  4308 R N   5.6  0.2   0:00   0 /usr/bin/php
5886 admin91   31  10  5756 5748  4308 R N   5.6  0.2   0:00   0 /usr/bin/php
5891 admin91   29  10  5728 5720  4308 R N   5.6  0.2   0:00   0 /usr/bin/php
5895 admin91   30  10  5648 5640  4308 R N   4.6  0.2   0:00   0 /usr/bin/php
5893 admin91   31  10  5632 5624  4308 R N   3.9  0.2   0:00   0 /usr/bin/php
5897 admin91   30  10  4744 4736  3664 R N   2.1  0.2   0:00   0 /usr/bin/php
5915 admin91   30  10  4752 4744  3672 R N   2.1  0.2   0:00   0 /usr/bin/php
5896 admin91   28  10   952  952   836 S N   0.7  0.0   0:00   0 /bin/bash php-script
5920 admin91   26  10   952  952   836 S N   0.7  0.0   0:00   0 /bin/bash php-script
5921 admin91   29  10  1708 1696  1348 R N   0.7  0.0   0:00   0 /usr/bin/php
    7 root      15   0     0    0     0 SW    0.3  0.0   1:48   0 kswapd
2220 root      25  10  1808 1808   500 S N   0.3  0.0   0:00   0
/usr/local/bin/exclog /var/log/httpd/access_log
5878 admin91   26  10   948  948   836 S N   0.3  0.0   0:00   0 /bin/bash php-script
5880 admin91   27  10   952  952   836 S N   0.3  0.0   0:00   0 /bin/bash php-script
5882 admin91   27  10   948  948   836 S N   0.3  0.0   0:00   0 /bin/bash php-script
5890 admin91   26  10   948  948   836 S N   0.3  0.0   0:00   0 /bin/bash php-script
5892 admin91   26  10   956  956   836 S N   0.3  0.0   0:00   0 /bin/bash php-script
5894 admin91   26  10   948  948   836 S N   0.3  0.0   0:00   0 /bin/bash php-script
5898 admin91   28  10   952  952   836 S N   0.3  0.0   0:00   0 /bin/bash php-script
5899 admin91   29  10   160  160   132 R N   0.3  0.0   0:00   0 /usr/bin/php



[Unknown]

#10
They're running PHP as CGI and the expect efficiency?  How much were you paying for this hosting, again?

They've also got 10 zombie processes, which means bad things in a good (that is to sigh high, not positive) likelyhood.  It's normal, sometimes, to have a zombie every once and again, but not to have ten at once - even if the load is high.  Zombies are, being that they eat brains, bad.

Plus, that isn't a log file.  I hate it when hosts name things incorrectly.

I'm going to guess that the server was just not configured to handle so many users at once.  They should have had an accelerator installed (which can significant speed PHP requests up) and they shouldn't be using it as CGI.  It does look like memory was being used fairly efficiently, though, but that's not a huge surprise as the machine had a good amount of ram.

As for the script... I'm afraid your host didn't allow the free and uptime commands to be piped, so it didn't help.  But, the above contains that information.

The most recent version of Netscape basically *IS* Firefox.  But, the latest stable release of Netscape is... a fairly old version of Firebird, if I remember right.  As for your other problems, I might guess that they were the cause of this zombie problem your host was having.

I'm afraid I didn't see anything useful from the error log, and I suppose it's gone now.  I would have actually had to see the SERVER error log, not your account's error log, to see any segfaults (iirc.)  I had forgotten you weren't dedicated - sorry.

Anyway, it sounds like you're in the market for new hosting.  I assume your old host will still allow you to access the database and files so as to back them up, right?  What's your budget?  Do you have an idea of what bandwidth you were pulling? (we only pull 45 gigs here.)

-[Unknown]

lyrae

I have no idea what to do -- I wish you could talk to them instead of me. All I can do is demand they restore my account until there is a resolution. We just had two articles about this site come out today in large city papers and this happens. God. This is really terrible.

The hosting is about $30CAD/month plus tax. The dedicated hosting is $70USD/month... we are supported by donations so I don't know that we can do it, but I will consider it if you feel it would help. I can't imagine raising that much but maybe... however all things considered I don't want it to be with THIS host.

We were using about 150GB bandwidth with a PHPBB on another server. I haven't had a chance to see how SMF compares in that regard since I've only been up for a week.

The worst part about this is I've discovered that the host does no technical support by telephone. I can't even get them to let me talk to a manager on the phone. Can you believe that?!

[Unknown]

I'm willing to talk to them for you, but it often does little help.  I'm a decidedly blunt person, and when I tell a host "you've got zombie processes, PHP in CGI mode, and no accelerator" they usually get insulted or something, and then it's almost worse.

Hmmm... $30CAD is what, $22USD?  Sorry, I'm an American :P.  70 is definitely a big jump from that... I know of dedicated which is less, but nothing is jumping out at me just now that's less than $60.  I do suggest looking at Hosts and Hosting, a bit.  I'll see what I can come up with.

I would guess that SMF would pull, with the same usage, slightly less than phpBB... but it wouldn't be a lot less.  Possibly the same, possibly even a bit more...

I'm sorry to hear this happened on the worst day possible :(.  Would your host put you back up if you offered to pay a higher price for this month, or anything?

-[Unknown]

lyrae

Based on the specs they sent me, is there any reason why this site couldn't be run on that server?

I'm trying to get a response from the "Administrators", and haven't heard back yet. It's been over an hour. Maybe I'll see what they have to say. I don't think there's any harm in pointing out what's lacking in their service. It could even benefit their other customers if they saw fit to actually implement any of the necessary changes. But what are the chances of that? I can't even get one of them to speak to me on the telephone.

I'm really... I have no idea what to do right now.

lyrae

Update:
I ended up having to change hosts. I can't even explain what the HELL was wrong with my host, but they were so incompetent they were basically useless. We switched to a VPS and they misconfigured the DNS, I had to have a quick education on DNS (via Google) to get the site up and running, and they continued to have unexplained outages of hours at a time. It was EXCRUCIATING.

Fortunately a guy related to the community the board revolves around offered us a hosting deal with his own company, a small business, and everything has been WONDERFUL since then. I know this is a lot of capitalised words, but man, the last two weeks have been so painful and it's wonderful to be with a host who knows what he's doing.

I so appreciate all your efforts to help me through with my former host. I guess it was just a lost cause -- they had no idea what they're doing and I have no idea how they're still in business. Instead of Canaca I now only refer to them as Can'taca. I also found the Better Business Bureau report on them, and let me say it's not pretty. They've informed the BBB that they will no longer so much as acknowledge any complaints from them. I just hope I manage to get a refund per their 30-day money-back guarantee... this whole thing has just been a nightmare.

Again -- thank you. I hope to never have to post in this thread again! You've given me a great impression of SMF support (coming from a PHPBB, this is wonderful to see).

[Unknown]

Quote from: lyrae on May 04, 2005, 11:54:56 PM
Fortunately a guy related to the community the board revolves around offered us a hosting deal with his own company, a small business, and everything has been WONDERFUL since then. I know this is a lot of capitalised words, but man, the last two weeks have been so painful and it's wonderful to be with a host who knows what he's doing.

I'm very sorry to hear it was painful, and I wish there was more I could have done to help.  But, I'm very happy to hear things are going better now.

QuoteI so appreciate all your efforts to help me through with my former host. I guess it was just a lost cause -- they had no idea what they're doing and I have no idea how they're still in business. Instead of Canaca I now only refer to them as Can'taca. I also found the Better Business Bureau report on them, and let me say it's not pretty. They've informed the BBB that they will no longer so much as acknowledge any complaints from them. I just hope I manage to get a refund per their 30-day money-back guarantee... this whole thing has just been a nightmare.

Unfortunately, this is all to common.  I don't know if this will make you feel better (it wouldn't me), but there are a lot of hosts out there that really have no clue, and just do it as a way to get quick money or because they've grown to fat for their belts.  For example, one host a client of mine was with has been listed in the top 5 of a lot of supposedly objective reviews... let me just say, it was not in the top five.  I have never seen better service than from people like charlottezweb, Thunderace, Xrd.nl, Douglas, and other like people who post here from time to time and own their own small hosting company.

It's also just as easy to find a smaller hosting company that is very very bad to go with.  I had a personal friend who was really swindling some clients, and giving them relatively poor service.  Whenever he had a moderate or worse problem, he'd just call me and beg me to fix it for him.  He wasn't a bad guy, really, and I think he meant to do better; but hosting just wasn't his thing, and he was trying to do it anyway.

My point is... don't feel bad.  It happens sometimes, and often you can't tell until it's too late - just as in this case.  In the long run, it's alright now... and the fact that it really hurt the ads you put out is unavoidable.  There's really not much you could have done to have known, and what's done is done.

QuoteAgain -- thank you. I hope to never have to post in this thread again! You've given me a great impression of SMF support (coming from a PHPBB, this is wonderful to see).

Thanks... and to think, I felt relatively useless and powerless in this topic :(.  Give me a bug, and I can fix it so much easier ;) (yes, I realize it's a bad thing I like bugs.)

-[Unknown]

lyrae

Quote from: [Unknown] on May 05, 2005, 12:19:28 AM
Quote from: lyrae on May 04, 2005, 11:54:56 PM
Again -- thank you. I hope to never have to post in this thread again! You've given me a great impression of SMF support (coming from a PHPBB, this is wonderful to see).

Thanks... and to think, I felt relatively useless and powerless in this topic :(.  Give me a bug, and I can fix it so much easier ;) (yes, I realize it's a bad thing I like bugs.)

Ha. Well, at least that way you're fixing your own problems instead of trying to understand some company's massive fsckups.

I posted an extensive review of Can'taca in the Hosts & Hosting forum. http://www.simplemachines.org/community/index.php?topic=35057.0
I hope it helps someone out there!

jerm

#17
hey,
just to put in my few cents worth, not sure if you still visit this boards, but
i was with canaca, and i know exactly how you feel. i got my money back, although i had to fight for it.. i was just about to go to the 'better business bureau' of canada if they didn't.. took about 15 tickets before i got my money back.
also save the refund ticket incase they never actually send you the money.
on another note to show how incredibly unorgainzed this hosting is,
i cancled my account on dec 31, 2004,
i got an email from them about 5 days ago telling me that they upgraded a server program.. phpbb2.. i haven't had a server with them for 5 months..
i just sent them back a reply and laughed at them.

so ya.. there was no saving your server cause canaca doesn't know anything about everything.
sucks you had to go through the same thing so many ppl do with that company, they just keep getting ppl cause of their low prices and high bandwidth/hard drive
i can go on and on about this host, but i'll stop here..
lol maybe you should have a thread sticked about hosts not to go to

-edit-
and i just had to write this.. even though its offtopic cause now im fustrated again
if you go to www.phpbb.com, at the bottom you'll see a link
"phpBB Hosting at $7.95", i thought since it was in words, that php worked with that company cause that was the best place to host phpbb.. so i got it.
little did i know.. i even comlaiend on their forums about it, emailed the 'owner' (with no response) and my threads got locked.. so phpbb didn't care that they were advertising the worse hosting company
and thats why i went out looking for a new forum type.. and i find myself here lol
*trys to calm down again*
-end edit-

Advertisement: