503 Service Unavailable - Too Many PHP Processes

Started by azureCorsair, March 06, 2014, 11:11:43 PM

Previous topic - Next topic

azureCorsair

On my forum, we have an app that is driven pretty much entirely by one file. (That might be the problem on its own?)

That file is modeled pretty closely after SMF's index.php, with the exception that all the functions called are in that same file instead of being spread out.

It pulls in the same set of files that index does, plus Subs-Auth and Subs-Post to use the findMembers and sendpm functions, loads the session, user settings, theme (to populate $context), and permissions, then checks the posted "action" variable, calls the appropriate function, then calls obExit.

The problem we're running into seems to be that we're now generating a bunch of PHP processes that are taking up about 100 MB of memory each and never terminating, which seems to be leading to a whole bunch of 503 errors for users.

I'm on SMF version 1.1.16.

I can't post external links, so I'm not sure how to link to my forum's status.php that one of the stickies asked me to upload, or I would.

Any insight or guidance is appreciated.

Edit: Sorry, forgot to mention that we're on shared hosting on a Deluxe GoDaddy account, Linux OS.

kat



Two problems seem evident.

SMF v1.1.16?!?! Bug fixes are released for very good reasons...

The other problem is "Godaddy". They're probably not the worst host in the world. But, they're sure a long way from being the best...

Can you see what those processes are?

BTW, if you need to post http://www.forum.com, you can put http://www(dot)forum(dot)com, to get around the anti-spam thingy about posting links.

azureCorsair

#2
Quote from: K@ on March 07, 2014, 08:06:22 AM
SMF v1.1.16?!?! Bug fixes are released for very good reasons...
We've made some changes to the source files that we're hesitant to lose, but I can see about upgrading. I'm not sure it's related, though, since this problem isn't happening with the forum proper. It started only when we rolled out this app. Edit: I'm not seeing how to upgrade without losing our mods. The update package says it doesn't work with mods (or I assume that's what "except when you have mods installed" means), and the upgrade package wants me to just replace all the sources and template files, which is no good. Am I missing something?

QuoteThe other problem is "Godaddy". They're probably not the worst host in the world. But, they're sure a long way from being the best...
Let's assume that changing hosts isn't really an option at this time.

QuoteCan you see what those processes are?
Apart from seeing that they're all PHP 5.3 processes, no, I don't think so. Status.php shows me running MySQL processes and what queries they're running, though.

QuoteBTW, if you need to post hxxp:www.forum.com [nonactive], you can put http://www(dot)forum(dot)com, to get around the anti-spam thingy about posting links.
Oh, right, I should have thought of that.

There's probably just some really obvious bug in the code that I'm not seeing because I'm not sure what to look for.

margarett

I can see one of two chances:
-you hire an experienced programmer (not saying you aren't, but someone better) for them to debug the code for possible hick ups
-you duplicate the setup in your local machine (so that you don't influence your live site) and try to reproduce that issue. If you can, then introduce some verification points throughout the functions to show memory available before and after each function. Hopefully this allows you to identify the culprit...
Se forem conduzir, não bebam. Se forem beber... CHAMEM-ME!!!! :D

QuoteOver 90% of all computer problems can be traced back to the interface between the keyboard and the chair

kat

If I have this right, you can find out what processes are running, if it'd help, by using the code, below. You'll need to change the username, &c. obviously, in the "user configuration" section, before running it.

<?php

######
### start of user configuration
######

## mysql username
$mysql_username "usergoeshere";
## mysql password
$mysql_password "passgoeshere";
## mysql host
$mysql_host "localhost";

## leave at 0 if you don't want it to refresh.. default is 20
$refreshTime 20;

######
### end of user configuration
######


if($refreshTime 0){
   print 
"<meta http-equiv='refresh' content='$refreshTime'>";
}

$link mysql_connect($mysql_host$mysql_username$mysql_password);
$result mysql_list_processes($link);

print 
"
<style>
body {
  background: #eaeaea;
}

td, th {
   background: #ffffff;
   font-family: arial;
   font-size: 11px;
}

.killed td {
   background: pink;
}

</style>
<table cellpadding='5' cellspacing='0' border='0'>
<tr>
<th>ID</th>
<th>HOST</th>
<th>USER</th>
<th>DB</th>
<th>COMMAND</th>
<th>STATE</th>
<th>TIME</th>
<th>INFO</th>
</tr>
"
;

while (
$row mysql_fetch_assoc($result)){

   if(
$row[Time] > 100){
      
$class " class='killed'";
   }else{
      
$class "";
   }
   print 
"
   <tr"
.$class.">
   <td>
$row[Id]</td>
   <td>
$row[Host]</td>
   <td>
$row[User]</td>
   <td>
$row[db]</td>
   <td>
$row[Command]</td>
   <td>
$row[State]</td>
   <td>
$row[Time]</td>
   <td>
$row[Info]</td>
   </tr>
   "
;
   
   foreach(
$row as $k=>$v){
      
$row[$k] = htmlentities($vENT_QUOTES);
   }

}

print 
"</table>";

?>

azureCorsair

That looks similar to the list that status.php provides, but it shows the sleeping and locked processes instead of just saying how many there are.

I'm not sure how I could replicate the issue on a local copy; it seems to occur only when there's a bunch of simultaneous users. The app was available to staff only for testing for a while before going live, and the issue never cropped up then. It was also pretty much fine for most of today until about an hour ago, I guess because there were less people online. So I don't know how I'd recreate that kind of situation.

青山 素子

Quote from: azureCorsair on March 07, 2014, 08:54:44 AM
We've made some changes to the source files that we're hesitant to lose, but I can see about upgrading. I'm not sure it's related, though, since this problem isn't happening with the forum proper. It started only when we rolled out this app. Edit: I'm not seeing how to upgrade without losing our mods. The update package says it doesn't work with mods (or I assume that's what "except when you have mods installed" means), and the upgrade package wants me to just replace all the sources and template files, which is no good. Am I missing something?

You can also use the package manager system to install the patch packages. They're linked right on the admin area, or you can download them from the customization site at http://custom.simplemachines.org/upgrades/


Quote from: azureCorsair on March 07, 2014, 08:54:44 AM
Let's assume that changing hosts isn't really an option at this time.

You'll probably continue to run into issues, just so you know. Optimization will only get you so far on an overloaded sub-par system.


Quote from: azureCorsair on March 07, 2014, 08:54:44 AM
There's probably just some really obvious bug in the code that I'm not seeing because I'm not sure what to look for.

Maybe. If the scripts aren't terminating, they must be waiting on something. Can you try making copies that do much less and then testing until you get to where the process sticks around?
Motoko-chan
Director, Simple Machines

Note: Unless otherwise stated, my posts are not representative of any official position or opinion of Simple Machines.


azureCorsair

I did some reading and switched the app's primary table from MyISAM to InnoDB, and that seems to have solved the issue. Whether that treated a symptom or the cause, I'm not sure.

Advertisement: