Forum slows down

[[Unknown]]

Well... Zend Optimizer recompiles it from disk every page view (and has been shown in many cases and under slashdotting to be less effective than eAccelerator.)  On the other hand, eAccelerator caches and saves it - and accesses that, yes.  If you run PHP as mod_php or similar, you can turn off disk completely (shm_only, etc.)

Anyway, I'm glad they identified the major problem and improved things for you.

-[Unknown]

[blunt]

Well, I spoke too soon, it seems

We've been down again twice (that I know of) since I last posted.  What do you make of this?

Server Status

Warning: mysql_query(): Too many connections in /home2/blunted/public_html/forum/status.php on line 468

Warning: mysql_query(): A link to the server could not be established in /home2/blunted/public_html/forum/status.php on line 468

Warning: mysql_fetch_row(): supplied argument is not a valid MySQL result resource in /home2/blunted/public_html/forum/status.php on line 469

Warning: mysql_free_result(): supplied argument is not a valid MySQL result resource in /home2/blunted/public_html/forum/status.php on line 470

Warning: mysql_query(): Too many connections in /home2/blunted/public_html/forum/status.php on line 473

Warning: mysql_query(): A link to the server could not be established in /home2/blunted/public_html/forum/status.php on line 473

Warning: mysql_query(): Too many connections in /home2/blunted/public_html/forum/status.php on line 483

Warning: mysql_query(): A link to the server could not be established in /home2/blunted/public_html/forum/status.php on line 483

Warning: mysql_query(): Too many connections in /home2/blunted/public_html/forum/status.php on line 497

Warning: mysql_query(): A link to the server could not be established in /home2/blunted/public_html/forum/status.php on line 497

Basic Information
September 12, 2005, 04:10:51 AM
Operating System: Red Hat Enterprise Linux ES release 3 (Taroon Update 5) 
Processor: Intel® Xeon(TM) CPU 2.80GHz (2790.790MHz)
Load averages: 3.79, 3.30, 1.94
Current processes: 439 (436 sleeping, 2 running, 1 zombie)
Processes by CPU: httpd (38) 14.7%, mysqld (261) 5.1%, updatedb (1) 3.0%, [kscand] (1) 1.5%, [exim (1) 1.1%, spamd (6) 1.3% 
Memory usage: 95.535% (1963608k / 2055376k)
Swap: 6.128% (125016k / 2040212k) 

MySQL Statistics
MySQL
Query cache enabled:
(query_cache_type)  0

That's from this morning's blackout.  There was only me connected to the forum at the time.

Here's what I've been getting from their support people (from the last couple of days) -

Posted on 10 Sep 2005 11:57 AM

--------------------------------------------------------------------------------
As you can see, the CPU is being wiped out - httpd (11) 106.5% and there's a spike in the load averages, which looks like it's going to get worse.



Blunt
 
 
Posted on 10 Sep 2005 04:30 PM

--------------------------------------------------------------------------------
Is anybody there?


Victor T.

 
 
Posted on 11 Sep 2005 06:08 AM

--------------------------------------------------------------------------------
Hello,

We have been having people attack the server on occasion. The server is up and running fine and I have been monitoring it for the past hour or so and it has been running at a normal load of around 1.5.

Please feel free to contact our technical support staff if you need any assistance.

Regards,
Victor



Blunt
 
 
Posted on 11 Sep 2005 05:32 PM

--------------------------------------------------------------------------------
Hi Victor

If the server is under dos attack, is there any way you can filter it? Any idea why it's happening?

It was down again, just now -

Basic Information
September 11, 2005, 05:27:18 PM
Operating System: Red Hat Enterprise Linux ES release 3 (Taroon Update 5)
Processor: Intel® Xeon(TM) CPU 2.80GHz (2790.790MHz)
Load averages: 8.92, 2.42, 1.64
Current processes: 153 (149 sleeping, 4 running, 0 zombie)
Processes by CPU: httpd (14) 106.0%, mysqld (19) 6.6%, perl (3) 3.6%, (other) (52) 2.5%, [kscand] (1) 1.5%
Memory usage: 60.801% (1249680k / 2055376k)
Swap: 13.924% (284072k / 2040212k)


Josh B.

 
 
Posted on 11 Sep 2005 08:40 PM

--------------------------------------------------------------------------------
Hello,
We are still investigating this issue. I am also going to have our management review this. We will update you shortly.


Thanks,
Josh


Tim Dorr

 
 
Posted on 11 Sep 2005 10:08 PM

--------------------------------------------------------------------------------
Hi,

I'm not sure what Victor is saying about attacks on the server. It is under no such attack that I can see.

One user account is causing a high amount of load because of the script they are running on their site. I've modified it to remove this extra load and they should be dropping now. There was also a lot of stuck mail messages that can cause the load to go up slightly, and have cleared those out. You should see some improvment now.


-Tim Dorr
Owner/Manager
A Small Orange Software

Any idea what's going on?  I'm baffled.  (but it doesn't take much)

[[Unknown]]

I can tell you exactly what's going on.  First, I'll say that it was a convergence of likelyhoods....

So, the error message in question is here:
Warning: mysql_query(): Too many connections in /home2/blunted/public_html/forum/status.php on line 468

That means MySQL won't let SMF/status.php connect, because max_connections is set too low.  In other words, you've reached your max as far as concurrent users go.

This:
Processes by CPU: httpd (38) 14.7%, mysqld (261) 5.1%, updatedb (1) 3.0%, [kscand] (1) 1.5%, [exim (1) 1.1%, spamd (6) 1.3%

Means a few things.  One, exim seems to be a zombie.  Couldn't tell you why, but that's a bad thing.  Then, you have mysqld eating 261 processes.  That's also more-than-likely a bad thing.  You also have updatedb going, which makes it sound like the server was doing maintenance, etc. just now.  This may have been right after a server backup, etc.

(query_cache_type)  0

Means, obviously, that the query cache is disabled.  Couldn't tell you why, since the ram usage seems quite a lot better now.  Anyway, it seems enabled at the time of posting, although it's also full and dropping queries.  What's going on on that server?

-[Unknown]

[Bonk]

You seem to recommend running php as an apache module frequently. (I do, but don't like it).

I thought php was still not thread safe and therefore it should not be run as an apache module. (as php's documentation reccomends). I understood that the most stable way to run php is as a cgi on apache on a *nix. (my statically linked apache-php cgi on QNX at home was rock solid)

I had problems with our server hanging continually and had to tweak the apache mpm settings to stabilise apache somewhat. Now it hangs occasionally, but it is still an irritation.

# WinNT MPM
# ThreadsPerChild: constant number of worker threads in the server process
# MaxRequestsPerChild: maximum  number of requests a server process serves
<IfModule mpm_winnt.c>
Win32DisableAcceptEx
ThreadsPerChild 64
MaxRequestsPerChild  10000
</IfModule>

I did not choose the platform, but have to cope with it.

[[Unknown]]

If you run it with prefork/etc. it should be fine.  You'd be surprised just how many people run it as an Apache module - we do here.  It doesn't segfault.

There are still some issues, I understand, but I've seen it work well in threaded environments in production.

-[Unknown]

[Joshua Dickerson]

hmm.. I think it is recommended for certain version. Apache 2, and then some more recent 1 versions.

Those crazy numbers of mysqld processes might be caused by persistent connections that haven't died yet. Just a wild guess.