Simple Machines Community Forum

SMF Support => Server Performance and Configuration => Topic started by: pooya on November 20, 2008, 08:11:37 PM

Title: .CORE files - HOw to STOP THEM!
Post by: pooya on November 20, 2008, 08:11:37 PM
Guys i got joomla running on my main root.

Everyday .core files are made on its own, im guessing youll know what they are.

Now my host says he has no clue what they about and says my joomla is doing it cause  to him its only my site creating the core files.

Has anyone any knowledge to this ? i searched joomla but all i found is some termination of a service on host causing this..

Completely lost, help.
Title: Re: .CORE files - HOw to STOP THEM!
Post by: 青山 素子 on November 21, 2008, 01:15:54 AM
Core files are created when a program on the server crashes. Now, it is true that your site is causing the crash. However, normal PHP code has no ability to cause that kind of crash on its own. Core files are a symptom of a problem on the server.

Your host has the ability to determine what exactly is causing the crash by running a tool called gdb on one of those files. That should tell them the exact program crashing and even perhaps why.
Title: Re: .CORE files - HOw to STOP THEM!
Post by: pooya on November 21, 2008, 06:53:40 AM
Thankyou very much motoko.. ill let my host know **reseller lol..
Title: Re: .CORE files - HOw to STOP THEM!
Post by: pooya on November 21, 2008, 02:05:10 PM
Reply of my host :

Problem: your account is the only one creating core dumps. I have about 200 accounts on the other server, not a single core file. I can't look into the crashed processes as the processes are already terminated, there's no logging of historical processes.
Title: Re: .CORE files - HOw to STOP THEM!
Post by: 青山 素子 on November 21, 2008, 05:37:05 PM
Well, if the site seems to run fine otherwise, just delete the files when you see them. If the host refuses to do a thing, there isn't much you can do besides finding a new host.

Also, you could try disabling hostname lookups. I've seen some bugs with the default RedHat PHP and the host command that can cause segfaults (and core dumps like you are seeing). Of course, they should be updating the box to the fixed versions...

Also, moving to server support, as this is really a support topic.
Title: Re: .CORE files - HOw to STOP THEM!
Post by: LiroyvH on November 21, 2008, 06:16:28 PM
What exactly shows in the error logs?

It could be a bad PHP script too you know.
Title: Re: .CORE files - HOw to STOP THEM!
Post by: 青山 素子 on November 21, 2008, 07:29:55 PM
If it was a bad PHP script, it should be throwing PHP errors. If it is getting PHP or another app to crash, it's 99.999% of the time an issue in that app or a library.
Title: Re: .CORE files - HOw to STOP THEM!
Post by: pooya on November 22, 2008, 06:47:10 AM
Well to be honest my joomla php is truely messed up..

But like motoko says its a server error , but my host says its just my site :(

ALso  these files are over 28mb each ... upto 32mb.
Title: Re: .CORE files - HOw to STOP THEM!
Post by: LiroyvH on November 22, 2008, 09:23:07 AM
Yeah... is it by any chance a 500 internal server error that creates the .CORE files?
That can still be the cause of a bad PHP script.
Title: Re: .CORE files - HOw to STOP THEM!
Post by: pooya on November 23, 2008, 09:05:53 AM
THe 500 internal server error dosent show up anywhere on my site :(

All i have is 404 redirects lol, but thats totally different.

:( so lost and my host is more lost then i am.
Title: Re: .CORE files - HOw to STOP THEM!
Post by: LiroyvH on November 23, 2008, 09:54:57 AM
So what's in the error logs?
Title: Re: .CORE files - HOw to STOP THEM!
Post by: pooya on November 23, 2008, 03:08:08 PM
The .core error or the error log file the server generates ?

Title: Re: .CORE files - HOw to STOP THEM!
Post by: expresscolo-tom on November 24, 2008, 04:46:24 PM
I happened to be reading the forums for another customer trying to find an answer to their problem, and I saw this post and I had to create a login just to reply to it....

It saddens me that a hosting provider doesn't know what a core file is or how to properly handle them. They are actually fairly invaluable sources of information; often giving you a great deal of information into why something crashes and provide a great deal of detail as to exactly what crashed and why.

To answer your question... the best way to get rid of them is to fix the problem that is causing them.

You need to identify what process is actually creating them. Most likely since this is a web server, I would lean towards apache/httpd, but you can't always trust intuition. Fortunately you can use the app 'file' to help you with this information. You just need to type:

"file core.1234"

at the command line and it will tell you what application created the core file.

then you can run:

"gdb httpd core.1234" (assuming the core was created by the httpd process)

That will provide you a stack of the application as it was running when it segfaulted and dumped this core file. Using this information you would be able to determine what caused the application to segfault in the first place.

Using this information your provider should be able to properly diagnose the problem and fix it.

The other... uglier option, is to turn off core dumps in the OS, and apache.

Title: Re: .CORE files - HOw to STOP THEM!
Post by: metallica48423 on November 24, 2008, 05:27:27 PM
Keep in mind that in many cases you will need shell/SSH access to do the above

Though, if joomla is causing these problems, have you considered contacting the folks at joomla?  It may be, perhaps, a known issue with a certain setting which can cause this.
Title: Re: .CORE files - HOw to STOP THEM!
Post by: expresscolo-tom on November 24, 2008, 07:30:17 PM
I should have been more specific... in that my recommendation was to be provided to the hosting provider via the OP, not that the OP would run those specific commands. They generally require root permission to a certain degree.
Title: Re: .CORE files - HOw to STOP THEM!
Post by: metallica48423 on November 25, 2008, 12:00:38 AM
of course :)

I'm more than not making a note for anyone who might find that post in the future -- this forum also doubles as a searchable support database ;) -- That is, we don't remove old posts and topics
Title: Re: .CORE files - HOw to STOP THEM!
Post by: pooya on November 25, 2008, 02:31:36 PM
Awesome post express.. Im forwading this to my host hoping it finds a cure.

metallica i search on joomla, they said there script cant be possible making .core files as its a server related command and no php can do so ...
Title: Re: .CORE files - HOw to STOP THEM!
Post by: 青山 素子 on November 25, 2008, 06:15:54 PM
Quote from: pooya on November 25, 2008, 02:31:36 PM
metallica i search on joomla, they said there script cant be possible making .core files as its a server related command and no php can do so ...

Well... It certainly isn't their script, but there could be some library PHP uses for a certain command, which just happens to be something their script calls. Likewise for SMF. So, yes, a PHP script can cause a core dump to happen. However, it isn't the problem, but rather it exposes a problem with one of the underlying things it uses.

I know one issue was a bad version of the "host" command on some CentOS/RHEL/Fedora machines that would coredump when being used. This was fixed, and the host should have updated their server since then, getting a fixed edition, but some still have not done so (and are very insecure since there have been security patches for the OS since then!).
Title: Re: .CORE files - HOw to STOP THEM!
Post by: LiroyvH on November 26, 2008, 05:12:06 AM
Quote from: expresscolo-tom on November 24, 2008, 04:46:24 PM
To answer your question... the best way to get rid of them is to fix the problem that is causing them.

Lol, i'm going to remember that one as the universal answer to everything.
No offence by the way :P
Title: Re: .CORE files - HOw to STOP THEM!
Post by: pooya on November 26, 2008, 12:31:52 PM
Hey buddy,

Lol, this company sounds like it's quite upset with this whole core file issue :) the reason i don't know what it is and how to handle them is cause i've never seen them before - i guess i've never had such poorly written software running on my servers before ;)

so yeah, here's the output:
root@power [/home/xxxt/public_html]# file core.10213
core.10213: ELF 64-bit LSB core file AMD x86-64, version 1 (SYSV), SVR4-style, from 'w}'

as for gdb - says command not found.

Feel free to advise further, since these guys are pros with dealing with running their buggy software on reliable webservers ;)


***Sounds like host is fired up :P
Title: Re: .CORE files - HOw to STOP THEM!
Post by: pooya on November 26, 2008, 12:34:18 PM
Okay how do i make them stop .. its really annoying me now.
Title: Re: .CORE files - HOw to STOP THEM!
Post by: 青山 素子 on November 26, 2008, 01:15:21 PM
You can't. Only the host can either by disabling core dumps or fixing the issue.

How big are the files?
Title: Re: .CORE files - HOw to STOP THEM!
Post by: pooya on November 27, 2008, 02:10:56 AM
28 to 30mb each.. in a day it makes like 10 core files.


Title: Re: .CORE files - HOw to STOP THEM!
Post by: 青山 素子 on November 27, 2008, 10:20:15 AM
Can you send me one of the files via FTP? I probably won't be able to determine the cause, but I can at least figure out the program that created them.
Title: Re: .CORE files - HOw to STOP THEM!
Post by: pooya on November 28, 2008, 08:24:55 AM
Sure, moving it to a special folder and giving you the link.
Title: Re: .CORE files - HOw to STOP THEM!
Post by: Joseph7 on November 29, 2008, 03:23:22 AM
I feel for ya. I am having the same problem on 2 different sites with SMF integrated. One site is a Subdreamer CMS with SMF integration. The other site is a Drupal CMS with SMF integration. I am assuming it is an SMF mod that is causing this problem. Until I can figure out which one I just delete the core dumps everyday.
Title: Re: .CORE files - HOw to STOP THEM!
Post by: metallica48423 on November 29, 2008, 06:17:15 AM
i had core dumps occuring due to the AEVAC mod at one point... i think it was AEVAC 3
Title: Re: .CORE files - HOw to STOP THEM!
Post by: pooya on November 29, 2008, 11:19:38 AM
Crap i too have the addon .. uninstalling it to see :O
Title: Re: .CORE files - HOw to STOP THEM!
Post by: 青山 素子 on November 29, 2008, 11:44:15 AM
Okay, I analyzed the dump as best I could. Since I don't have the same binaries as the server, I can't get a reliable back trace.


Core was generated by `/usr/bin/php /xxxxxx/xxxxxxxxxx/public_html/index.php'.
Program terminated with signal 11, Segmentation fault.


So, the problem is in PHP or one of the libraries it uses.
Title: Re: .CORE files - HOw to STOP THEM!
Post by: pooya on November 29, 2008, 04:46:22 PM
oooooooo so it is one of the scripts which is casuing this ?
Title: Re: .CORE files - HOw to STOP THEM!
Post by: aldo on November 29, 2008, 05:24:57 PM
Possibly... I remember I was getting core dumps (like you) in my SMF directory, I uninstalled a modification and it stopped. I don't remember which one it was...
Title: Re: .CORE files - HOw to STOP THEM!
Post by: pooya on November 30, 2008, 03:01:18 PM
Well nope the avc didnt help much and yes motoko, when u open the page with firefox you can see the TONS of errors it shows and ie as well :(
Title: Re: .CORE files - HOw to STOP THEM!
Post by: swtdivalove on December 08, 2008, 09:49:22 AM
I just read something on this yesterday.  .CORE files are an HTTPD (Apache) Dump.  It means that there is an error in the Apache files and is causing Apache to crash.

Apache is a multi-thread, multi-server application.  It means that it can replicate itself if there is a high enough load on the the Apache daemon.

What the issue could be is that they do not have an allowance set high enough for reserve servers to run.  And, when Apache hits this limit, it crashes the HTTPD processes and you get a core dump.

I would find out which version of Apache is running on the server.

But, also... If there are any running processes that exceed the limit amount in Apache, this can also cause issues.

If it is a new version of Apache, it seems to be the cause.

I have Apache 2.2.10, the newest version and I'm seeing roughly similar things, but I am not getting any crashes on my web server.

But... .CORE files are directly related to Apache.
Title: Re: .CORE files - HOw to STOP THEM!
Post by: 青山 素子 on December 08, 2008, 10:43:05 AM
Quote from: swtdivalove on December 08, 2008, 09:49:22 AM
I just read something on this yesterday.  .CORE files are an HTTPD (Apache) Dump.  It means that there is an error in the Apache files and is causing Apache to crash.

Although Apache itself might be generating the dumps, it doesn't have to be the cause. Also, many hosts run things like PHP outside the Apache process and they could be responsible for the dump (as the original poster's dump showed).


Quote from: swtdivalove on December 08, 2008, 09:49:22 AM
Apache is a multi-thread, multi-server application.  It means that it can replicate itself if there is a high enough load on the the Apache daemon.

It depends on what MPM you are using. On Windows, it defaults to the WinNT MPM (a threaded model), but on Linux and UNIX systems, it uses the much more stable pre-fork MPM. Some hosts also run more experimental MPMs like worker (threaded) or perchild.


Quote from: swtdivalove on December 08, 2008, 09:49:22 AM
What the issue could be is that they do not have an allowance set high enough for reserve servers to run.  And, when Apache hits this limit, it crashes the HTTPD processes and you get a core dump.

I would find out which version of Apache is running on the server.

The core dump I examined showed the problem was with PHP itself, not Apache. Investigating the wrong thing won't help the poster.


Quote from: swtdivalove on December 08, 2008, 09:49:22 AM
But... .CORE files are directly related to Apache.

No, they can be generated by any process. The most common cause is when an application uses an invalid memory reference (SIGSEGV), although there are other causes.

You might want to get your knowledge straight before tossing out comments like this. I suggest reading the following manpages:

core(5) (http://linux.die.net/man/5/core)
signal(7) (http://linux.die.net/man/7/signal)
Title: Re: .CORE files - HOw to STOP THEM!
Post by: swtdivalove on December 08, 2008, 11:29:36 AM
What I read was dealing strictly with Apache.  Don't remember the site otherwise I would point it out as I looked at a myriad of sites trying to deal with my own issue.  The only reason it caught my attention was that someone was asking about .CORE files.  The posters did not get into other areas.

But, since you want to jump on me for posting what someone else said, be my guest.  It seems to be the modus operandi.
Title: Re: .CORE files - HOw to STOP THEM!
Post by: 青山 素子 on December 08, 2008, 11:49:06 AM
I am sorry if you took offense to my post. The thing is that bad information (as in this case) is often worse than no information. Having bad information can cause you to waste time in the wrong area or even fiddle with something that isn't causing issues, breaking that too.

Once again, I am sorry if I was too harsh.
Title: Re: .CORE files - HOw to STOP THEM!
Post by: swtdivalove on December 08, 2008, 12:29:59 PM
I agree... But, one of the things you could do, as advice, is to show the person where they are wrong and give them a direction to go.

Consider it training someone.  You would not just tell them they are wrong, you would show them what they did is wrong.  Then point them in the right direction.  In this case, a link to a site that shows them information on the subject.  You don't necessarily need to give the answer, but let them discover on their own.

I apologize for coming off a tad rash.

But, I will point out that the information that I was looking at and reading in regards to this issue was the HTTPD daemon.  I now know that other processes and other applications can now cause this too.  Which will be furthering my training in web server admin'ing.

I do thank you for the information.

Also, I hope that this helps you as you are helping me.

Scratch your back and in turn mine gets scratched.

Have a good day Motoko-Chan.
Title: Re: .CORE files - HOw to STOP THEM!
Post by: Nao 尚 on January 03, 2009, 06:30:15 AM
Just my 2 cents here. No solution offered -- just explaining how complicated it can be to get these fixed.

Quote from: metallica48423 on November 29, 2008, 06:17:15 AM
i had core dumps occuring due to the AEVAC mod at one point... i think it was AEVAC 3
Regarding this.
As Karl mentioned in another Core dump-related topic in the same board (thanks BTW), I managed to fix the issue in Aeva version 5.1.

I haven't been too vocal about the fix because (1) I never was able to get core dumps on either my server (even though it has PHP 5.2.6) or anyone else's, (2) it was too soon to be certain that it was fixed. But since I haven't got any complaint on that side since the release, I'll consider it as definitely fixed.

The bug was indeed due to the PHP stack getting filled too quick with a relatively lazy regular expression -- which was originally brought by the official PHP 5 documentation. I came up with two successives fixes which I applied to it (a ":" turned to ">" and an additional "?" to turn the regexp into a greedy one), and it fixed the bug. The crash would only happen when you had a long quote/code block inside a post. I don't remember the specifics. But on my server, it crashed with a 500 error code, which is possibly the vaguest error code you could find. So there is no "fixed" solution to the Core dumps -- if your server is simply not fit for your software, it might crash too soon. Regular expressions are a possible reason for the crashes, but not only these.

From what I could read in other core-related topics here, it can happen pretty much anywhere. The point is to be able to narrow down the piece of code that generates the crash, and then try to make it stricter (for instance in my case, making a regexp greedy instead of lazy). Even then, there are moments when Error 500 will happen outside the scope of a normal user visit. Not much can be done to debug these.

All I can say is, good luck, or if your errors aren't viewable by human beings, simply try to have Core dump automatic generation disabled.
Title: Re: .CORE files - HOw to STOP THEM!
Post by: metallica48423 on January 03, 2009, 02:38:19 PM
Nao, FWIW,i have had absolutely zero coredumps since upgrading to 5.1, and i used to get them daily :)
Title: Re: .CORE files - HOw to STOP THEM!
Post by: Nao 尚 on January 03, 2009, 05:03:08 PM
Good, good, thanks :)
I really spent two full days on this bug, it turned me crazy and everything... One of my most epic bug fights in the last few months ;)
Title: Re: .CORE files - HOw to STOP THEM!
Post by: deathnight on May 14, 2009, 06:05:19 PM
tanx , I`m wait solution for this problem