Forum stopped working, possible hack or bug, Settings.php modified to 0 bytes.

Started by Skybuck, December 04, 2018, 05:13:18 AM

Previous topic - Next topic

Skybuck

Hello,

PascalCoin Forum stopped working:

Probably since 21 november 2018. Today I decided to investigate why this is.

An error report was sent to my e-mail at the same date:

<snip>
"
There has been a problem with the database!

This is a notice email to let you know that SMF could not connect to the database, contact your host if this continues.
"

I just contacted the webhost.

However since the forum was under attack from a "spammer" I think more might be going on.

The ftp server or forum might have been hacked or somebody caused a delibrate misconfiguration or it's simply a bug.

Connecting to the ftp server I see something suspicious (I was actually looking for any log files but have not found any yet, where are they if any ?):

What I can see is the following:

Settings_bak.php   (filesize 0, modified at same date 21 november 2018, same time as well)
Settings.php (filesize 0, modified at same date 21 november 2018, same time as well)

Cache folder is also last modified during this date.

Some hypotheses how this may have happened:

Hypothesis A:

1. Perhaps it was me changing some settings, this is plausible cause I was deleting spam and disabling new registrations.

2. Perhaps somehow all the spam caused the server to run out of disk space. Perhaps the host was under attack by multiple sites attacks and ran out of space.

3. Somehow SMF software try to created a backup of settings.php, this failed, then it tried to store it and also failed.

Because of out of disk space both files ended up being zero ?

Hypothesis B:

Somebody hacked into the forum, possibly by cache exploit and caused Settings.php to become zero ?

Hypothesis C:

FTP server was hacked or it was delibrately done by an employee of host ?

Hypothesis D:

Settings.php had wrong permissions and was overwritten by an attacker ?

I currently cannot see how much disk space is available on this webhost/server, not sure if it's possible to see and how to do that ?

I am also not sure if Settings.php is needed but it's most likely needed.

So hypothesis for a possibly solution:

Settings.php has to be re-uploaded to fix the problem.

I still have all files from original installation on my local C drive.

Is it safe/recommended to re-upload the Settings.php ?

Is there perhaps something else I should do.. like reconfigure permissions on this file ?

One last question, kinda already indicated above:

Are there any logs files anywhere to see what happened ?

For now it seems 99.9% a sure thing that this is probably the cause of total forum failure... it just shows a blank.

In the case that this is not the cause, what other diagnosis procedures do you recommend ?

Is there perhaps some PHP file that can be run to diagnose failures/problems ?

Bye,
  Skybuck.

P.S.: (Not allowed to post links ? This is the web for crying out loud, risky to disallow posting if text is lost...)

Arantor

First, your PS... not allowed links in first post to discourage spammers.

As to the rest... there is a bug in SMF when multiple DB errors happen together (e.g. the host can't handle all the DB connection requests) that have it try to update Settings.php and sufficient requests close enough together that all fail can cause the file to be wiped out including its backup.

Reason: it stores the time it last sent you the "could not connect" email, inside Settings.php. But to do this it has to read then write the file, and too close together can cause this to fail.

This is fixed in 2.1 by moving the time out of Settings.php into its own file.

Skybuck

Thx for your answer, some questions about this:

1. My webhost reported that their engine switched to running PHP 7.1 on 1-11-2018. Is SMF 2.1 compatible/working with PHP 7.1 ? It would seem so since my forum was working up to later in november 2018.

2. How to proceed with installing/updating the forum software now that it is no longer working ? Do I simply wipe SMF from the webhost and re-install and re-use database ?! How can I make sure that the contents of the database/postings is kept intact and not whiped/reset/deleted and such ?

One last remark:

It may be helpfull to include a version.txt file in the SMF distribution, it's not particularly clear what version it is, though the readme.html seems to indicate this at the top. Little bit more version redundancy can't hurt ;) though extra updating work required for SMF developer(s).

P.S.:
Not allowing links in messages is indeed a smart way to discourage spammers somewhat ;)

Though perhaps the forum software should warn about this before trying to post the message, so the chance of losing text is minimized ;)

Arantor

1. Both SMF 2.0 and 2.1 are compatible with PHP 7.1. Compatibility with mods and add ons is not guaranteed.

2. Copy the Settings.php file from the 2.0 install zip and put the settings back in. Your host should be able to help with this.

3. It already is, look in index.php for $forum_version. Putting it elsewhere is not feasible given the way patches are handled.

Skybuck

Quote from: Arantor on December 05, 2018, 07:35:50 AM
1. Both SMF 2.0 and 2.1 are compatible with PHP 7.1. Compatibility with mods and add ons is not guaranteed.

2. Copy the Settings.php file from the 2.0 install zip and put the settings back in. Your host should be able to help with this.

3. It already is, look in index.php for $forum_version. Putting it elsewhere is not feasible given the way patches are handled.

1. Ok good to know this.

2. I checked both files from zip and install folder, they are exactly the same so the installation of SMF did not change these files.

I will try re-uploading these overwriting the zero byte files and see what happens.

3. How am I suppose to know this ? I am not much of a PHP programmer a version.txt file would be a bit more clear or so... but ok a readme.html will do for now as well though this requires a webbrowser to be functional :) as long as it's in there clearly stated it's not to bad.

An alternative could be a little comment in the index.php file somewhere at the time for easy finding. Don't want to go scan an entire php file for some obscure $forum_version variable which is very rare and SMF specific... all kinds of software should state their version is some clear and consistent easy to read/access manner ;) :)

Anyway no further directions were given how to update to 2.1. I'll google this and check the forum for further information about this. Perhaps it's as simple as click "update" somewhere as soon as the forum is working again.

I know the settings for mysql and such somewhere in a file.

I am not sure I am going to be successfull at placing them in the correct spots in the php files and such.

Perhaps a little php form should be coded to help users of SMF 2.0 software that run into this bug to get their settings back into the correct place.

For now I will try and see if I can find the correct lines to modify and such.

The original installation was done via forms and such... settings were specified there.

There is no backup of these settings.

This could be an idea to make an additional backup of these settings in the future of SMF software, settings that should remain untouched and unused by the SMF software once everything is running and installed.

In case this zero bytes bug returns in the future =D

That might make it a bit easier to get the forum back up and running. I wish there was a backup somewhere... that would save me some time have to fiddle with php and settings and such... but will try.

Illori

this is why we recommend users do a backup, then if something does fail you can restore the files/database and move forward.

Skybuck

I am going to need a little bit of guidance how to setup Settings.php correctly (or preferably a script to get it working again, though that would require some coding time for you guys):

I will snipe out the parts I think don't matter, like code and comments and such:

########## Maintenance ##########
# Note: If $maintenance is set to 2, the forum will be unusable!  Change it to 0 to fix it.
$maintenance = 0;      # Set to 1 to enable Maintenance Mode, 2 to make the forum untouchable. (you'll have to make it 0 again manually!)

^ This seems a bit interesting, but can probably leaved untouched for now, not sure what this is but ok.


$mtitle = 'Maintenance Mode';      # Title for the Maintenance Mode message.
$mmessage = 'Okay faithful users...we\'re attempting to restore an older backup of the database...news will be posted once we\'re back!';      # Description of why the forum is in maintenance mode.

########## Forum Info ##########
$mbname = 'My Community';      # The name of your forum.

^ Ok this seems easy enough to change.

$language = 'english';      # The default language file set for the forum.
$boardurl = ' snipped to circumvent link posting issue';      # URL to your forum's folder. (without the trailing /!)

^ Think this might have to be my webhost address and then folder or so

$webmaster_email = 'snipped to circumvent link postage problem';      # Email address to send emails from. (like [email protected].)

^ Not sure what to put here...

$cookiename = 'SMFCookie11';      # Name of the cookie to set for authentication.

########## Database Info ##########
$db_type = 'mysql';
$db_server = 'localhost';

^ Not sure, should this remain like this ?

$db_name = 'smf';

^ Not sure, did install program ask me to change this ?

$db_user = 'root';

^ Not sure, webhoster login details for mysql ?

$db_passwd = '';

^ probably related to login above.

$ssi_db_user = '';
$ssi_db_passwd = '';

^ No idea what this is.

$db_prefix = 'smf_';

^ Not sure if this has to be changed.

$db_persist = 0;
$db_error_send = 1;


Illori

try this instead


but if you don't know what your database username/password or prefix is you may be best to ask your host if they can restore that file from a backup
What is repair_settings.php?

Skybuck

After I installed SMF forum software I did create a text file which contains the database settings for SMF forum.

So now I need to figure out what to copy where...

First I will try and access phpadmin on my webhost to see if the database is still there... what the table names/prefixes are that kind of thing...

Just to see if it's still there or if perhaps a hacker or bug ;) deleted it :)

Currently have a bit of problems logging into phpadmin... tried to do it directly... perhaps I first need to login to my webhost... so I will try that... then I will proceed with the php link you posted.

What is this php link/file ? A repair php file for this particular problem ?

* Update *

Ok PHPadmin is working via webhost.

The prefix for the database I chose was PCFD_     

I guess this was Pascal Coin Forum Database, yeah pretty sure.

Many tables with that prefixes so tables are there... so seems like database is still there, working somewhat and in good shape... though not yet sure how to use phpadmin to actually see content. (Ok I see content now, last time I used phpadmin was ages ago.. funny to see this stuff .. it's grayish/silverish... not sure if this is something custom from this webhost or the latest and great phpadmin... perhaps it was also updated to complement php engine 7.1)

Illori


Skybuck

Quote from: Illori on December 05, 2018, 02:04:53 PM
did you even read the page linked?

Ok I read it, thought it was a link to some source file or something.

Anyway what is a little concerning is it will display database password... why ? for retrieval ? hmm I already know the password so this is less ideal, because then others might see it too ?

Is there any security for this repair tool ? So that only my webbrowser session can see it and nobody else ? ;)

Also what if I set maintance mode in settings...

Will this get the forum/menu things back up and running so I can go in and change these settings ? Like in integrated repair tool or so ? Hmmm...

Or is the Settings.php so screwed up that even maintance mode will not run ? I could try it... but will follow guide lines for now to now cause any weird problems... no idea what maintance mode does... ;) I have a suspicion but for all I know it could start computing weird files =D

*UPDATE*

I have set all settings that seemed to matter, the only one I am unsure of is this one:

$cookiename = 'SMFCookie11';      # Name of the cookie to set for authentication.

Should I change this for more security ? Or is it security-wise ok to leave this the same ?

I am going to upload the changed Settings.php file now and see if it works.

* UPDATE 2 *:

Ok the forum is working again ! Did not need the repair script ! =D

Thanks for the help.

I'd still like to know the answer to the cookie though.

Anyway, are there any other more serious bugs in SMF 2.0 ? How urgent is it that I update the forum to 2.1 ?

Is the database at risk of being corrupted if I don't upgrade to 2.1 or vice versa ? Hmmm..

Also a good question is:

How to download/backup the database ? This I truely don't know... would be cool if SMF has some "download database" option for admins... and also "upload database" option.

Not sure if it already has this... though this is a bit outside the scope of this thread... may have to search forum or documentation for this.

* Update 3 *

Cheered a bit too soon, still seems to be a problem once I tried to access a forum:

"
Not Found

The requested URL /PascalCoin/Forum/www.skybuck.org/PascalCoin/Forum/index.php was not found on this server.
"

Seems like I miss-configured the url setting...

* Update 4*:

Ok I fixed it by including http:// in front of it

Now it seems to be working ok :)

Hmmm maybe somebody figured out how to trigger this 0 bytes settings bugged and nuked my forum... could also be by chance :)

Perhaps there is a nuke_smf_2.0_.php script or tool somewhere on google/the web ?... hmmm...

Performed a quick google search... top results so far yield nothing... though this spammer was probably pretty good with php... so he may know how to do it... I would not be surprised if it goes down again.

Did not understand bug exactly...

Something with many connections triggers it ? May have to re-read that...

A link to a technical description of what causes the bug might be nice for a somewhat interesting read (?).

Or a proof of concept... just to see if this indeed caused the bug/forum crash in the first place... or if it was me that somehow triggered it or both... kinda confused about this... but so far so good.

* slight bug analysis *:

Person above wrote:

"
Reason: it stores the time it last sent you the "could not connect" email, inside Settings.php. But to do this it has to read then write the file, and too close together can cause this to fail.
"

Why would this fail ? Very weird... php engine issue ? SMF php scripting bug issue ? webhost to slow issue ? Kinda weird...
Though I understand PHP and webhosts and web technology not too reliable... but still a bit weird...

Maybe related to having to do this multiple times with multiple db connection/bug/errors... hmmm..

* Additional *:

Adminstration center/news center on SMF forum software says:

"
SMF 2.1 Beta 3 released on May 31, 2017, 09:21:59 PM
    Simple Machines is proud to announce the third beta of the next version of SMF, which contains many bugfixes and a few new features since 2.1 Beta 2.
"

So SMF 2.1 is still in beta ?

Could perhaps this particular Settings.php bug be fixed in 2.0.15 too like 2.0.16 or did it involve many changes ? Too much to put back into 2.0.15 ?

So what is recommended running 2.0.15 ? Or 2.1 beta 3 ? Hmmm...

At least now I can easily set back Settings.php file... so for now this is a little bit workable :) Not having a forum crash would be little bit better though.

Though if 2.1 beta 3 is very buggy and screwing things up with the database then I rather not run it for now... hmmm...

(At least now until I know/learn how to backup the database to either webhost copy or local copy... preferably local copy).

Kindred

Version 2.1 is still in Beta. This means that unless you know what you're actually doing with coding and our reporting, you should not be using a Beta version especially not on a production site
Слaва
Украинi

Please do not PM, IM or Email me with support questions.  You will get better and faster responses in the support boards.  Thank you.

"Loki is not evil, although he is certainly not a force for good. Loki is... complicated."

Arantor

Why would that fail? If you have too many connections, it blows through the limit assigned by your host and connections stop being made, so it starts erroring. At which point it could be sufficiently frequent that it causes a race condition about overwriting the file.

Backporting to 2.0 isn't entirely simple.

Should you use beta 3? No, it's a year behind the current development and unless you can keep it up to date and triage issues, don't use it in production.

Sir Osis of Liver

The bug that wipes Settings.php seems to be server dependent, it's never happened to my forums, or any of the many forums I've worked on.  Simplest fix is to copy Settings_bak.php to Settings.php, the _bak file is not affected by the bug (per Illori), or restore it from a backup (you have backups, don't you?). 

There's a database backup in 2.0 Forum Maintenance, but it doesn't work on many hosts and is not recommended.  I believe it's been removed in 2.1.  First choice is cpanel backup, if you have one, second choice is phpmyadmin, but it's subject to timeouts with large databases on slow servers, third choice is third party app like MySQLDumper, doesn't work on some servers.  Any decent host should have an automatic backup utility that does daily backups of all databases and files (mine does), and you can supplement this with cron job backups (I do).
Ashes and diamonds, foe and friend,
 we were all equal in the end.

                                     - R. Waters

Arantor

It's not server dependent, it never ever has been. It's traffic dependent, high bursts of traffic at a time when there is some circumstance causing a DB connection error. Less busy forums, especially those on high quality hosts, will simply never see it.

Sir Osis of Liver

Ashes and diamonds, foe and friend,
 we were all equal in the end.

                                     - R. Waters

Arantor

No, not really, because this can happen on even the best hosts if you get swamped with way too much traffic, far beyond resource capacity.

Sir Osis of Liver

Ashes and diamonds, foe and friend,
 we were all equal in the end.

                                     - R. Waters

Illori


Arantor

So that if there was a DB error that would need to track time stamp, it never has write contention on the Settings.php file.

Though I half wonder if updateSettingsFile could just flock(LOCK_EX) the file and be done with it.

Advertisement: