Simple Machines Community Forum

SMF Support => SMF 2.0.x Support => Topic started by: greyknight17 on May 26, 2008, 05:56:04 PM

Title: Core dumps being created
Post by: greyknight17 on May 26, 2008, 05:56:04 PM
I tried to resolve this issue myself but couldn't figure out what's wrong. It looks like SMF keeps recreating core files (35MB+) every day. I have to constantly delete them daily. I tried a fresh SMF upload and it still doesn't help.

Using SMF 2.0 beta 3.1 default theme. No mods installed. I tried changing the permissions around as well so all standard files are writable.

Thanks.
Title: Re: Core dumps being created
Post by: Douglas on May 27, 2008, 01:05:20 AM
I'd recommend you take a peek at http://en.wikipedia.org/wiki/Core_dump and then look at Apache or PHP as the potential culprits.
Title: Re: Core dumps being created
Post by: 青山 素子 on May 27, 2008, 01:05:55 AM
SMF doesn't make those, it's one of the other apps (probably PHP) doing it while running a script in that directory. Core dumps usually indicate something very wrong with the application. How big is a dump? I might be able to look at it with gdb and see if I can pick out what is causing it.
Title: Re: Core dumps being created
Post by: SleePy on May 27, 2008, 01:08:21 AM
I had this for the longest while as well for my 1.1 forum.

I updated to Apache 2.2.8 and PHP 5.2.6 and it went away. I was using before that Apache 2.2.6 and PHP 5.2.5
Title: Re: Core dumps being created
Post by: greyknight17 on May 29, 2008, 12:01:28 PM
Was wondering if it could be a coincidence. It started to happen around the time I updated to the latest SMF and I thought it could be related ;)

I have the following for my webhost:

Apache version   1.3.41 (Unix)
PHP version   5.2.5

Will ask them if they could upgrade as I see the Apache version is definitely older than the one you listed there Sleepy.

Motoko-chan: The dumps vary. Something I have a lot that are 0 bytes and ranging from 3MB to 30MB+. I get a handful of these (about 7 or so) each day. I will contact the host first and see if they can do anything on their end for this.

Thanks guys.
Title: Re: Core dumps being created
Post by: 青山 素子 on May 29, 2008, 12:28:58 PM
Quote from: greyknight17 on May 29, 2008, 12:01:28 PM
Was wondering if it could be a coincidence. It started to happen around the time I updated to the latest SMF and I thought it could be related ;)

It's possible that the new code has exposed a problem with PHP or Apache. It isn't an uncommon thing for that to happen. (The best example I can think of right now is when a more efficient piece of software gets installed and exposes an underlying race condition because the code now runs faster.)
Title: Re: Core dumps being created
Post by: greyknight17 on May 30, 2008, 12:02:46 PM
Yep. Sounds like a very good possibility that it's the case, especially with the very outdated version they have. Still haven't gotten a response back from them yet, but will followup shortly.

Thanks :)
Title: Re: Core dumps being created
Post by: argimiro on June 06, 2008, 04:52:29 AM
files named core.(5 digits[0-9]) - all roughly 7mb in size - first 4 bytes always 7f 45 4c 46 (?ELF)

Is that correct?

I'm getting tens of these created in the forum root directory on 2.0 beta 3.1 - but not on a 1.1.5 installation on the same server

If these files are the same issue talked about in this thread I will ask my host about it too

okay here's their reply;

Quote
The core files in your SMF forum are the corpses of programs (compiled and linked binary executables) that have upped and died. Unix-family operating systems (such as Linux) save these core files when a program dies. They contain information about the state of the machine and the program, including a snapshot of the execution stack at the moment of death. A programmer familiar with the program (and having the source code to it) can play pathologist and examine the entrails to see what was the cause of death. For anyone else, core files are useless and just take up space. For a system such as SMF, it's most likely that PHP itself died, and a systems programmer might be interested in looking at the core file to see what PHP choked on.

Apart from deleting the files they had no other suggestion. We're on PHP Version 4.4.8

The innards of the corpses don't mean anything to me unfortunately. I've kept them all the same.
Title: Re: Core dumps being created
Post by: 青山 素子 on June 06, 2008, 11:26:54 AM
If you aren't going to use a debugger to investigate the cause of the crash, delete the files. They are worthless otherwise.

In general, you should not get core dumps. However, they do occasionally happen when there is a null pointer or similar error that indicates a problem in how the program was built (or one of the supporting libraries). An experienced systems manager will often go through these files to see if they can determine the cause of the issue (at least what caused it) so they can check their server configuration or with the provider of the program at fault.
Title: Re: Core dumps being created
Post by: argimiro on June 06, 2008, 07:56:30 PM
I've discovered I can make it core dump at will.

I'm using the Amber11final theme which doesn't look as if it natively supports 2.0 beta 3.1 - I originally installed the theme on 1.1.5, then ran the large upgrade package. That gave me a /forums/Themes/amber11final/ directory with all the language variables updated, which I subsequently saved and re-zipped for use on fresh installs of 2.0 beta 3.1

That wasn't quite sufficient though - it was giving '500 Internal Server plus a 404' errors on all topic display pages. After picking through the theme I isolated a tract of code down near the bottom of Display.template.php:


foreach ($category['boards'] as $board)
echo '
<option value="?board=',
                        $board['id'],
                        '.0"',
                        $board['is_current'] ? ' selected="selected"' : '',
                        '> ' . str_repeat('==', $board['child_level']) . '=> ' . $board['name'] . '</option>';


This code seems to have something to do with displaying a 'Jump To..' dropdown at the bottom of topic display pages.

I found that if I commented out that code, the theme ran on 2.0 almost perfectly - just a few variables not showing up here and there.

I then discovered that it was that small tract of code that was causing the core dumps - if I re-enable it, the 500 Internal Server & 404 errors come back, and so do the core dumps - one is generated per page load. If I comment out that code, the core dumps stop and the theme returns to near-normal operation.

If I try replacing the code with


foreach ($category['boards'] as $elephant97)
        {
        echo "core dump<br>";
        }


I get the 500/404's & the core dumps too, so I guess it's the foreach ($category['boards'] bit that's somehow responsible..? Maybe it's the same for everyone or maybe this is just the specific cause on my combination of theme / host

I had no problem with


foreach ($category['randomnonexistentindex'] as $cardboardbox)
        {
        echo "core dump<br>";
        }


though so it's as if there's something going on with the 'boards' index of $category. least in my case.

OP is using SMF 2.0 beta 3.1 default theme though so.. guess it's a different cause there
Title: Re: Core dumps being created
Post by: greyknight17 on June 07, 2008, 10:50:54 PM
Motoko-chan, still interested in taking a look? My host isn't budging on the Apache & PHP upgrade. Planning on switching soon, but want to confirm that it's not something else causing the problem.

Thanks.
Title: Re: Core dumps being created
Post by: 青山 素子 on June 08, 2008, 12:52:11 AM
I don't mind. Hopefully I could be of some help. I'm not all that experienced in the dark arts of poking through core dumps, but I can hopefully find the part that broke. I'll create an FTP account for you as soon as I can and PM the details.
Title: Re: Core dumps being created
Post by: Captain Kaos on June 08, 2008, 04:54:37 AM
I find same thing on my server , i have 3 forums
Only in SMF 1.1.5 versions   not in the latest SMF 2.0 Beta 3.1 Public
do i see the core.  files
Title: Re: Core dumps being created
Post by: 青山 素子 on June 08, 2008, 12:16:14 PM
I suppose I can do one more examination if you want. I'm not sure I have time beyond that at the moment.
Title: Re: Core dumps being created
Post by: 青山 素子 on June 08, 2008, 01:59:13 PM
Okay, I don't have the same layout on my box, so I can't adequately trace the fault.

Greyknight, it is PHP crashing, but I can't see the stack since my libraries are at different versions. Do you have shell access?
Title: Re: Core dumps being created
Post by: greyknight17 on June 19, 2008, 09:01:13 PM
Thanks Motoko-chan. The host actually upgraded PHP to 5.2.6....must be recently after my complaint.

PMed you back with cPanel login info. It does have SSH access listed there.
Title: Re: Core dumps being created
Post by: Captain Kaos on June 19, 2008, 10:20:31 PM
Hey guys all i did to fix the issues was go back to php v4.***
Never had it happen again
either some thing in PHPv5 that conflicts with SMF v1.***
did not happen on the latest beta on same servers

Host rolled every one on his server to V4 and fixed all other SMF forums on there too. I aint no programmer but this cant be that hard to work out ? Is it

http://www.simplemachines.org/community/index.php?topic=223829.msg1580747#msg1580747
Title: Re: Core dumps being created
Post by: 青山 素子 on June 19, 2008, 10:28:03 PM
Quote from: Captain Kaos on June 19, 2008, 10:20:31 PM
either some thing in PHPv5 that conflicts with SMF v1.***

It has to be some configuration issue. I've been running the 1.1 series including 1.1.5 under PHP5 (5.2.5 at the moment) for some time and no issues here.


Quote from: Captain Kaos on June 19, 2008, 10:20:31 PM
Host rolled every one on his server to V4 and fixed all other SMF forums on there too. I aint no programmer but this cant be that hard to work out ? Is it

Depends. If it can be reproduced, the cause can be determined. If not, it will be very difficult. I haven't had any issues, so I can't begin to try and replicate.
Title: Re: Core dumps being created
Post by: Captain Kaos on June 19, 2008, 10:31:44 PM
Wish you guys luck , But yes you right unless one of you that know
code can have it happen, You guys are stuck  would be good if some
of the others that had it went back to a different version to see
if that helped,very lucky down under here , we have good Hosts that
will work with you. Thanks guys
Title: Re: Core dumps being created
Post by: 青山 素子 on June 19, 2008, 11:39:15 PM
If the host will work with you, have them do backtraces using the core dumps so that it can be determined where the fault happened.
Title: Re: Core dumps being created
Post by: Captain Kaos on June 19, 2008, 11:40:36 PM
Ill ring them and ask  , Ill get back to you
Title: Re: Core dumps being created
Post by: greyknight17 on June 25, 2008, 12:25:34 AM
My host is not doing much here. I just want to get to the bottom of this and see what's causing it before I move over to another host.
Title: Re: Core dumps being created
Post by: Captain Kaos on June 25, 2008, 01:18:15 AM
Hey Guys
My ISP was not happy to help, I did not expect that from them, so one thing i did try and was to go back to PHP v5 and still i have not had any more core files, both forums i run are now preforming perfect.
soryy cant give you any more info than that.
I do know only change on server was made , they installed some SUPHP software ????
as the Core files did not have owner ship this allocated them to the hostee's account
so then they would stop being created when there limit was exceeded.

hope you can make sence of that
Title: Re: Core dumps being created
Post by: 青山 素子 on June 25, 2008, 02:30:04 AM
Unfortunately, without access to the binaries on the server, it is near impossible to tell where the failure occurred.

For example, this is all I can make of the core file I was sent by greyknight:

Core was generated by `/usr/bin/php /REDACTED/index.php'.
Program terminated with signal 11, Segmentation fault.
[New process 8451]
#0  0x08231b91 in ?? ()


Without the right php binary (and associated shared libraries), I can't tell anything about the stack when the crash occurred, nor what might have been going on.
Title: Re: Core dumps being created
Post by: xyxis_fahim on June 26, 2008, 11:39:24 AM
Ok my host said they will not deal with this because this was created by SMF script.
What can I do?
Title: Re: Core dumps being created
Post by: 青山 素子 on June 26, 2008, 12:01:47 PM
I don't think there is much you can do. There isn't really a way to prove to them it is not because it is not very possible to poke at the dump yourself.

I will say this, and you are welcome to quote this to your host:

Any PHP script should not cause a segmentation fault. It's rather silly to think that. If a script causes a seg fault, that means it has exposed a bug or problem in the program running it.

Why is this the case? Well, a segmentation fault is a memory violation. Specifically, when a program tries to access a non-accessible area of memory, or do something like write to a location that is "read-only". PHP and most other scripting languages can't possibly cause this because they have no control over how their parser allocates and uses memory. Rather, if this occurs while running a script, it means that particular code path the script is using has a big bug in it.

While it is not standard practice in most places to treat every core dump with a thorough investigation (at least for non-life-critical operations), recurring dumps indicate a problem that should be investigated. Given that most script parsers make use of shared libraries - particularly PHP - the fault might not be in the parser itself, but one of the libraries. This means that other applications that use this library are at risk of having problems too.

Since that many access attempts in the past involve stuff like stack smashing and crashing the application (the infamous Morris Worm crashed fingerd as an attack point), that small annoyance in the script parser might be a much larger underlying problem in the system waiting for someone to break in using it.

Remember, however, it isn't the script that is the problem, it's the actual thing crashing. Removing a script that causes segmentation violations doesn't remove the problem, it just hides the symptoms. In an analogy with the old coal-mine canary: if the canary keeps dying/passing out in the mine, you know there is a problem with the mine. The correct thing to do is to get out of there and figure how to fix the issue. You do not just get rid of the canary and venture in again.
Title: Re: Core dumps being created
Post by: greyknight17 on June 28, 2008, 11:56:28 AM
Thanks for all your help Motoko-chan. I guess I will be switching hosts then and hope for the best. The new host will have a more recent version of everything including Apache and PHP.
Title: Re: Core dumps being created
Post by: metallica48423 on June 28, 2008, 04:42:03 PM
QuoteI then discovered that it was that small tract of code that was causing the core dumps

What code, specifically, did you establish may be causing this before?

Let us know how it goes, greyknight
Title: Re: Core dumps being created
Post by: greyknight17 on July 05, 2008, 12:41:53 AM
The craziest thing happened these past few days. I have just been monitoring them to see if I was just lucky. They haven't been showing up anymore. As far as I know, all the versions of the software are still the same including the old Apache 1.3 version. I will ask my host to see what they changed.
Title: Re: Core dumps being created
Post by: MrPhil on July 13, 2008, 10:18:56 PM
Quote from: argimiro on June 06, 2008, 04:52:29 AM

If these files are the same issue talked about in this thread I will ask my host about it too

okay here's their reply;

Quote
The core files in your SMF forum are the corpses of programs (compiled and linked binary executables) that have upped and died. Unix-family operating systems (such as Linux) save these core files when a program dies. They contain information about the state of the machine and the program, including a snapshot of the execution stack at the moment of death. A programmer familiar with the program (and having the source code to it) can play pathologist and examine the entrails to see what was the cause of death. For anyone else, core files are useless and just take up space. For a system such as SMF, it's most likely that PHP itself died, and a systems programmer might be interested in looking at the core file to see what PHP choked on.

Cool! I'm being quoted as an authority by someone's hosting service! Or did you lift that from my posting on lunarforums.com?
Title: Re: Core dumps being created
Post by: MissRancher on July 17, 2008, 03:01:43 PM
I'm having this problem too. I ran the same version of SMF (1.1.4) on GoDaddy for a year with never a core file created. As soon as I installed on A2Hosting (through Fantastico), it's core galore.

I got this email from them:

Hello,

Your site and the other site are both dumping core with the same reason.

Core was generated by `host -W 1 99.150.196.229'.
Program terminated with signal 6, Aborted.
(gdb) backtrace
#0 0x00002aaaac432155 in raise () from /lib64/libc.so.6
#1 0x00002aaaac433bf0 in abort () from /lib64/libc.so.6
#2 0x00002aaaab61ca22 in isc_assertion_typetotext () from /usr/lib64/libisc.so.11
#3 0x00002aaaab63fda2 in isc_socketmgr_destroy () from /usr/lib64/libisc.so.11
#4 0x00002aaaac1ee307 in start_thread () from /lib64/libpthread.so.0
#5 0x00002aaaac4d3ded in clone () from /lib64/libc.so.6

The command of course runs fine from shell, but seems to be failing in the code. Do you know why or where the script may be calling the host command?
Title: Re: Core dumps being created
Post by: 青山 素子 on July 17, 2008, 03:42:35 PM
Quote from: MissRancher on July 17, 2008, 03:01:43 PM
The command of course runs fine from shell, but seems to be failing in the code. Do you know why or where the script may be calling the host command?

Disable hostname lookups. However, that command shouldn't be dumping core.

Also, it is recommended that you not use Fantastico for production applications. It installs some weird configs (for SMF, it forces UTF even if the server doesn't support it and it installs a ton of languages) and wants to manage updates, which it usually winds up breaking.
Title: Re: Core dumps being created
Post by: metallica48423 on July 17, 2008, 03:45:16 PM
not to mention that -- not only does it force UTF, it does so only halfway -- the Database and settings are set to UTF, but the tables remain ISO-8859-1.  This mx of collations can cause fatal errors on some MySQL setups, and will generally cause problems with characters outside the ISO-8859-1 set
Title: Re: Core dumps being created
Post by: MissRancher on July 17, 2008, 04:59:41 PM
Quote from: Motoko-chan on July 17, 2008, 03:42:35 PM
Disable hostname lookups. However, that command shouldn't be dumping core.

Thanks. I will post back if that does indeed fix the problem, just for a reference for anyone else.

I assume the database structure for non-Fantastico is the same as for Fantastico, so I can just export all my tables and data? I guess I'll be finding out shortly!
Title: Re: Core dumps being created
Post by: metallica48423 on July 17, 2008, 05:17:13 PM
the structure is the same, yes, but the SMF installer itself properly sets the database and table collations.  However, your backup would probably be backed up with the incorrect collations :(
Title: Re: Core dumps being created
Post by: greyknight17 on July 19, 2008, 12:00:17 AM
Hmm...thought I posted a followup on this. I asked the host what they did recently since it resolved the issue. They upgraded cPanel and recompiled Apache and PHP. That got rid of this core dump headache.
Title: Re: Core dumps being created
Post by: ellion on July 28, 2008, 02:48:37 PM
is there anything i should or could do about these core dumps.

my host is using apache 2.2.8 and php 5.2.6 but i am still getting these core dumps about 30 a day at 30mb each
Title: Re: Core dumps being created
Post by: 青山 素子 on July 28, 2008, 02:58:49 PM
If you have shell access, you could try to study them and see what the problem is, but otherwise only the host can do anything.
Title: Re: Core dumps being created
Post by: ellion on July 28, 2008, 04:09:58 PM
i dont have shell access [not that i am aware of]
what should the host do?
Title: Re: Core dumps being created
Post by: 青山 素子 on July 28, 2008, 05:48:08 PM
Check the core dumps and find out what is causing them.
Title: Re: Core dumps being created
Post by: MrPhil on July 28, 2008, 11:17:37 PM
Quote from: ellion on July 28, 2008, 04:09:58 PM
i dont have shell access [not that i am aware of]
what should the host do?

You definitely need to do something about it. 900MB a day is too much. Your host, if they're the least bit competent, should be able to figure out which binary program is going belly up (most likely, PHP or MySQL), and then what it was doing at the time. If the program is very down-level, they should get up-to-date. Otherwise, they'll need to work with the program's developers to see what's causing the program to die (what scripts or input are the problem).
Title: Re: Core dumps being created
Post by: ellion on July 29, 2008, 08:49:26 AM
well, i contacted my host letting them know some of the information i found here.

the host repsonds thus

Quote from: TechSupportThis problem is created by software version you are using. This is a beta version and beta always creates nuisance.

Please revert back to most stable version or the tech may feel that you are misusing the resources. This amy cause more trouble.

Please ask the developer to provide a fix or solution to this. We are not going to modify anything in php or apache just for the sake of this script.

the funny thing about it is i am not using 2.0beta version i am running 1.1.5

even funnier is the host seems to have crashed  the server. i cannot get access  to any of their sites. http://damnx.us or anything that i had hosted with them

luckily i had a working back ups and recent copies of the database.

i'll have to see how things unfold.

Title: Re: Core dumps being created
Post by: 青山 素子 on July 29, 2008, 10:45:53 AM
Wow. Some hosts are just very incompetent... A script isn't the problem because it can't do anything that the code running it won't allow. Read response 25 on this topic (http://www.simplemachines.org/community/index.php?topic=241376.msg1592772#msg1592772) for a bit of a long discussion on why the script isn't really the problem.

All I can do from here is to wish you luck.
Title: Re: Core dumps being created
Post by: ellion on July 29, 2008, 10:55:12 AM

i updated my support ticket informing the host that i was not using a beta version of smf. i got the following reply

Quote from: [email]TechSupport@damnx.us[/email]Wworking on your issue caused server down for last 12 hours and many refunds are raised causing heavy financial loss just to resolve your issue.

Tech is still working but cannot assure solution as server and all cl;ients are more important

Support Team
Title: Re: Core dumps being created
Post by: 青山 素子 on July 29, 2008, 11:32:39 AM
Wow...

I think they are just trying to use use you as a scapegoat. Seriously, running gdb for a backtrace shouldn't cause an outage.

I don't want to sound mean or anything, but I encourage you to find a better host that won't offer up such poor excuses and blame you for things.
Title: Re: Core dumps being created
Post by: ellion on July 29, 2008, 11:38:52 AM
i have been looking for a host this morning, to be honest i am fed up with hosts. i dont know what to do for the best. the one i am with has been okay except for this problem. i have good uptime and the server is never slow. i hear horror stories about the larger hosting companies. right now i am on a free host as a back up until i can be sure it is safe to go back to my paid host.
Title: Re: Core dumps being created
Post by: 青山 素子 on July 29, 2008, 11:43:23 AM
You can always ask over at our Hosts, Hosting and Commercial Services (http://www.simplemachines.org/community/index.php?board=4.0) board. I also recommend doing som research over at Web Hosting Talk.

Every host has problems, but some are better than others.
Title: Re: Core dumps being created
Post by: MissRancher on August 01, 2008, 12:36:49 PM
My core dumps all went away after I disabled hostname lookup.
Title: Re: Core dumps being created
Post by: metallica48423 on August 01, 2008, 02:32:26 PM
Mine persist.  I guess i'll have to give it a go at figuring out why they're being generated
Title: Re: Core dumps being created
Post by: NickC on August 01, 2008, 03:34:26 PM
I got a load of them for the first time a few days ago when the server threw a wobble. Part of my host's response:

"You can delete the core dumps from your forum folder. These files are created whenever segmentation fault occurs. I've disabled core file generation when programs crash - they're rarely useful in our environment and will save everybody hassle."

Sorted.
Title: Re: Core dumps being created
Post by: MrPhil on August 01, 2008, 11:58:36 PM
Wow, talk about putting a Band-Aid over a gaping wound! Your host should be doing something about finding out why there are so many segfaults, rather than just turning off core file generation and pretending there's nothing wrong. I can only hope that they've gotten enough dumps to look at and are busy working on a fix, and that the no-core policy is just temporary.
Title: Re: Core dumps being created
Post by: 青山 素子 on August 02, 2008, 12:17:54 AM
Agreed. Segmentation faults will occur occasionally, but multiple ones in a day is way too much and that needs to be looked at.
Title: Re: Core dumps being created
Post by: searchgr on August 12, 2008, 03:23:21 AM
Quote from: ellion on July 29, 2008, 11:38:52 AM
i have been looking for a host this morning, to be honest i am fed up with hosts. i dont know what to do for the best. the one i am with has been okay except for this problem. i have good uptime and the server is never slow. i hear horror stories about the larger hosting companies. right now i am on a free host as a back up until i can be sure it is safe to go back to my paid host.

ellion, did you find what causes the core files?
Title: Re: Core dumps being created
Post by: ellion on August 14, 2008, 03:18:35 AM
hi,

i never found the cause of them, my host eventually came to realise that it was the server that was bugging not smf. he moved my account to another server.

which has been absolutely fine. no problems at all.

during the process of migration from my buggy server to the new server there was a couple of days where i had the forum on the buggy server not being used but still accessible via ftp.

the core dumps where still being created in the public directory of the old server despite the forum not being used and the public directory not being accessed. the core dumps were being created at the same rate. i think this confirmed for me that this was a server bug not a smf bug.
Title: Re: Core dumps being created
Post by: searchgr on August 14, 2008, 03:29:45 AM
Thank you.