SMF crashed my site - what should I do to resolve this?

[robg2251]

Here's what my host tech support had to say: 

========================================
my questions: What would reviewing the crashed table tell us? Would it tell us why
it crashed? Can we expect a crash like this in the future?

Host response: The reason the table crashed is because /var reached 100% while trying to write to that table. That table was 6.5GB and is the SMF error log essentially. Once /var reached 100% and mysql continued trying to log errors to that table, the table crashed and brought down the entire server.

As for expecting problems like this in the future, it's not likely in the immediate future, as I have moved your mysql directory to /home where it will have more room to grow, however I just checked the size of the new smf error table, and it is already at 129MB which is a lot of errors to log in one day. You can safely assume that at that rate, it would reach 6.5GB within 2 months, though it would continue to grow until it filled up /home this time which has 165G free as of right now.

I would recommend looking through the SMF admin panel for something regarding an error log of sorts to see what messages are being added and why so many are being added, and I would recommend attempting to fix the problem(s) that are being logged the most to prevent this from continuing to be a problem. If you cannot find any type of error log, you may wish to contact SMF support for assistance in determining why the database is growing at such an exponential rate.
========================================

Looking at the errors, they are coming in pretty steady.   They are all from Guests, some of the same IP addresses, but they are staggered.  Should I block those IP addresses?   Should I remove all of the errors?  - right now it appears there are 60,1711 errors. wow.   

Any suggestions would be appreciated.   
http://www.monroetalks.com/forum/index.php

[robg2251]

That might have been a lot to read - here's the brief: 

For some reason, we're getting bombarded with error logs daily, hourly.  So many that it reaches our table max of 6.5 GB and crashes mysql.   The errors are coming from various IP's one right after another.  There are over 60,000 in there right now.   Any ideas what these are??

[Aleksi "Lex" Kilpinen]

So, have you checked the Admin CP error log, to see what it might tell you - and possibly your servers error log, to see what the errors really are?

[robg2251]

Here's the error from the server - these are coming in quite a bit, several per second:
[Sat Feb 28 13:53:50 2009] [error] [client 66.249.71.69] client denied by server configuration: /home/monroe/public_html/forum/index.php

[Aleksi "Lex" Kilpinen]

That IP is Googlebot, and the error is that it's access to your site has been disallowed by the server.

[robg2251]

Well I wouldn't mind a google bot crawling the site, what can I do to resolve this?    The errors are hitting like crazy.. from what the tech support said, that file is going to fill up and eventually crash again.. 

[robg2251]

Is there any other information that I could supply that might help us solve this issue?  We're crashing daily now, and our host is blaming it on SMF errors building up.. tons every second. 

[DragonballZ]

I dont think it is SMF, if it was, your host would have canceled you immediately, its surely due to some configuration problem on their end.

client denied by server configuration -
that is at the http conf or .htaccess level and would be a 403 error. unless you have SMF defined as your 403 error handler, then SMF wouldnt even be connected to the issue.

Anyway, the SMF error log is almost useless,especially when it fills up with rubbish, you may as well turn error logging off at least.

[JimM]

Do not turn error logging off!  That is poor advice.

Please post the exact text from the SMF error log and we will try to help you work through them.  Some errors are insignificant but others are not.  You're goal should be to eliminate every error you are getting.

[robg2251]

Here's a common one -

Unable to verify referring url. Please go back and try again.
http://monroetalks.com/forum/index.php?amp;action=quickmod2;topic=11907.0

[robg2251]

I think we've found some more info on this.   I'll post some more of the errors below this. 
Some of the processes are almost 40 minutes old and not dieing.  Processes from SMF are not dieing once a visitor leaves the site.   We're getting so many of these, they're causing our site to crash every few hours.  This wasn't happening with the 1.1.7, I realized we've been having this problem ever since we upgraded to 2.0.. Any advice??    I would greatly appreciate it if someone could make a suggestion for something to change, look into or update..  thanks in advance. 

25196 monroe    19   0 66304  19m  11m S  0.0  1.0   0:00.15 /usr/bin/php /home/monroe/public_html/forum/index.php
25199 monroe    18   0 68424  18m  11m S  0.0  0.9   0:00.15 /usr/bin/php /home/monroe/public_html/forum/index.php
24852 monroe    16   0 66860  18m  11m S  0.0  0.9   0:00.14 /usr/bin/php /home/monroe/public_html/forum/index.php
24865 monroe    19   0 66052  18m  11m S  0.0  0.9   0:00.14 /usr/bin/php /home/monroe/public_html/forum/index.php
25210 monroe    18   0 67428  18m  11m S  0.0  0.9   0:00.14 /usr/bin/php /home/monroe/public_html/forum/index.php
25183 monroe    18   0 67764  18m  11m S  0.0  0.9   0:00.14 /usr/bin/php /home/monroe/public_html/forum/index.php
24849 monroe    19   0 67516  18m  11m S  0.0  0.9   0:00.13 /usr/bin/php /home/monroe/public_html/forum/index.php
25215 monroe    18   0 65320  17m  10m S  0.0  0.9   0:00.12 /usr/bin/php /home/monroe/public_html/forum/index.php
24878 monroe    19   0 65912  17m  10m S  0.0  0.9   0:00.13 /usr/bin/php /home/monroe/public_html/forum/index.php
25045 monroe    19   0 65260  17m  10m S  0.0  0.9   0:00.12 /usr/bin/php /home/monroe/public_html/forum/index.php
25258 monroe    18   0 65112  17m  10m S  0.0  0.9   0:00.13 /usr/bin/php /home/monroe/public_html/forum/index.php
25193 monroe    18   0 65760  17m  10m S  0.0  0.9   0:00.11 /usr/bin/php /home/monroe/public_html/forum/index.php
25293 monroe    19   0 65036  17m  10m R  9.9  0.9   0:00.11 /usr/bin/php /home/monroe/public_html/forum/index.php
25212 monroe    17   0 66292  17m  10m S  0.0  0.9   0:00.12 /usr/bin/php /home/monroe/public_html/forum/index.php
25219 monroe    18   0 66300  17m  10m S  0.0  0.9   0:00.12 /usr/bin/php /home/monroe/public_html/forum/index.php
25205 monroe    18   0 66388  17m  10m S  0.0  0.9   0:00.12 /usr/bin/php /home/monroe/public_html/forum/index.php
24868 monroe    18   0 66652  17m  10m S  0.0  0.9   0:00.12 /usr/bin/php /home/monroe/public_html/forum/index.php
25287 monroe    18   0 66452  17m  10m S  0.0  0.9   0:00.11 /usr/bin/php /home/monroe/public_html/forum/index.php

[robg2251]

Thought I should add this note - I saw this thread:  http://www.simplemachines.org/community/index.php?topic=296088.0  - wondering if that has something to do with our errors.   Can anyone confirm?

We just installed this theme, figuring we were doing a good thing updating to a 2.0 theme:
http://custom.simplemachines.org/themes/index.php?lemma=1370

Any advice would be appreciated!!

[nsdcblue]

i went to look at your site, and couldnt see it...may i ask when you started seeing the laggs an white outs...was there any additions or code changes you can remember doing...like any mod adds or say code put in for a background image or anything??


checking editing

[robg2251]

i went to look at your site, and couldnt see it...may i ask when you started seeing the laggs an white outs...was there any additions or code changes you can remember doing...like any mod adds or say code put in for a background image or anything??

The sites down once again, waiting on the host to reboot the server.  We did massage the theme a bit, changing the colors up..  no mods have been added, since all of them error out when we tried installing them.  So basically we're just running the same theme as I linked above, with the colors changed up.    Check back in a few, the site should be back up and running.... for how long, we don't know.   A recommendation was to switch to a default theme. I'm going to wait for the site to come back and give that a try.. then I guess we can narrow down what the problem is. 

[nsdcblue]

i know i am not an smf whiz nor into alot of coding etc. but in looking at your forum theme and seeing it has a background image, it is really beginning to make me wonder. before using any of the smf 2.0's i have been able to add a background image to any of the themes as well as default and havent had any problems.
in the change over to 2.0 beta 4 and seeing the differences in how things were  rearanged in the coding and where things are, i came here to check on what code to use to add a background image, and used it. now i know some mods were installed etc and after 4 times of smf eating up the cpu was told by the server to do something or take down the site. not knowing what was the cause, we took down the smf2.0 beta 4 and put up smf 2.0 RC1 default...tried it....was wonderfully fast no lags no probs. added mods...no problems no lags. then we tried the green  template before with the background....problems  n lagging occurred...so we changed back to default ...no lagging no white pages. sooooo we added a background to default....left it over night...an this morning the lagging an white pages n the slowness occurred was unreal...so i removed the background...just changing the background color.  and since...we have had no problems whatsoever.  so is it all in my mind??    something is definately wrong if a person cant add a background to their forum without causen so much problem..

and also to mention...i have never seen so much down time n lagging with this site as well...an i see smf 2.0 beta4 is being used....and i see the template uses a background image...makes me go hmmmmmmmm

[nsdcblue]

omg  wat a time to post this....
another thing too.....i cannot get the edit at the right with the pencil to work....yet the modify button does...this was something else noted at our forum as a problem with a background image being  used.

[robg2251]

Moved to the default theme and all of the error logs went away. 
http://www.simplemachines.org/community/index.php?topic=296088.msg1958962#new

I don't want to stick with this theme - I wonder if SMF knows of this error, and if it's something that can be corrected..  should we post a bug report?  Do they have a bug report link on this site?

[DragonballZ]

this is some crazy  I dont know what kind of difference a background image could cause...it all seems way too bizarre..

are you all using output compression?? GZ? compression is done server side and uses up server cpu... maybe with output buffering on, and new stuff gets added to the output stream, it has to recompress the buffer?

[robg2251]

We installed our theme, removed the background image, and all is well.  No errors. 
http://monroetalks.com/forum/index.php

So that's definitely it - background images = BAD!!    Hopefully this gets resolved. 

[nsdcblue]

this is some crazy  I dont know what kind of difference a background image could cause...it all seems way too bizarre..

are you all using output compression?? GZ? compression is done server side and uses up server cpu... maybe with output buffering on, and new stuff gets added to the output stream, it has to recompress the buffer?

yes it is bizzare.  all i have to go on is what i know and what has happened as a result of, and based on my knowledge, as i had said, i am more on the graphics side and am just learning the ins and outs, and would love to know the root cause...its very disheartening not to be able to design your own background without problems occurring.

as to the answer to your question, i am not sure, all i know is what i am told is that the host emailed and told to change the problem or get rid of the software for it was eating up the servers cpu. which an inquirey was made here as to what to do, so we went with what we knew n seen. i can ask  and see about it. i appreciate in any input, for it is a problem and its frustrating when tryin to figure out what to do and solve it properly...getting to the real root of it.