2.0.2 Unresponsive, Possible Security Breach?

[Vince S]

G'day, our SMF forum is suddenly not accessible, appearing to hang; this is at http://hunterdogs.org/DiscussionBoard/index.php. I cannot get to it via either Firefox8 or ie9; one user reports they can get to it in Chrome and has managed to post OK, complete with emailed notices to me about the posts (since I am subscribed to all boards); another user reports they got this message:
"Warning: Something's Not Right Here!
hunterdog.org.au contains content from globalstatupdate.com, a site known to distribute malware. Your computer might catch a virus if you visit this site.
Google has found malicious software may be installed onto your computer if you proceed....."

I can visit the admin page of the forum but it will only show me the log in page and then hangs. I can look at it in CPanel file manager where I see recent files of a 1Gb error log and some weird named attachments, these seem to be related to the successful posts judging from size, time, post content and qty. I am downloading the error log at the moment but it will be a couple of hours to get and I may not be any wiser.

The last time I touched the site was to put 2.0.2 on via package mgr, it went very easily and nothing seemed wrong. This was just after Chrissie if I remember correctly.

Can you please give me a clue what to do? thank you.

[Illori]

have you checked all your files for anything from that website that has been added? if there is nothing, then you need to check your database for anything from that website.

[Vince S]

I have contacted the host (SiteGround) to see what they could see, they found this:

I have scanned your hosting account for malicious files and found the following file which represents a backdoor to your hosting account:
/home/hunterdo/public_html/default.php
I strongly recommend that you delete the file and perform a security audit of your website.

That file was small (169 bytes) and dated 9 hours ago, which could be 21 hours ago if it was server time, both after the problem was noted by another member (about 22 hours ago)so it will be a consequence not a cause. Obviously I deleted it but no apparent effect, still broken.

I have done a CPanel search for globalstatupdate of all files, no result. Not sure how to do "check your database for anything from that website", can you give me a clue please?

I am trying to figure out how to do a "security audit" which Google helps with but the one I have clicked on for the free trial (http://www.acunetix.com/cross-site-scripting/links.htm) still hasn't sent the promised download link so I will have to try another one if it doesn't come soon. Anyone had experience solving this kind of problem? Thank you.

[Illori]

if you have phpmyadmin you should be able to search for that string in your database.

[Vince S]

Streuth that phpmyadmin is a pretty cool tool, first time I've seen it. So I picked on the SMF database and did a search of all tables, but no matches. Does that mean its a dead end?

Darn, I have just realised all the database style add-ons we have are in the same condition. ie we use a calendar and a photo gallery and have a couple of experimentals loaded (eg zen cart) that aren't active.

In that 1Gb error file it has one heck of a lot of lines that look like this:
[05-Dec-2011 00:26:15] PHP Warning:  mysql_real_escape_string() expects parameter 2 to be resource, boolean given in /home/hunterdo/public_html/DiscussionBoard/Sources/Subs-Db-mysql.php on line 132

and these stretch back over many months both pre and post 2.0RC5 conversion from 1.1.about14. It is hard to handle a 1Gb text file. I am trying to delete great swathes of similar errors so I can save it as a smaller file and manipulate it to see what is going on. I can see that at least 2/3rds of the content is similar to the above, but this is going to be a separate problem that I have just become aware of. I do suspect deleting the error file on the server will be helpful as just addressing that size file will be a problem, particularly if it is still tipping errors in at such a rate....

[Illori]

well if you cant find it in the database that means it must be in your files somewhere.

[Vince S]

I have been trying unsuccessfully to even count how many times these PHP warnings appear in the error log, whatever else is going on this is a really bad situation. It looks like about 50 times a second but I am yet to count how many times in a day or pick what the initiation or end points are. it may be a gazillion or two!!! All errors are identical, only the time changes Aha, I managed to find it was 28 screens for the one second time interval from:
[27-Dec-2011 10:32:24] PHP Warning:  mysql_real_escape_string() expects parameter 2 to be resource, boolean given in /home/hunterdo/public_html/DiscussionBoard/Sources/Subs-Db-mysql.php on line 132
[27-Dec-2011 10:32:25] PHP Warning:  mysql_real_escape_string() expects parameter 2 to be resource, boolean given in /home/hunterdo/public_html/DiscussionBoard/Sources/Subs-Db-mysql.php on line 132

But is this related possibly or something else that needs dealing with separately?

[Vince S]

and now that I have whittled the error log down and got some control I can see the period the errors get generated for vary wildly, and there is no pattern to the intervals between starting and stopping. Sometimes they only happen for half a screen per second, others are much longer. Here is an eg of a short break:
[31-Dec-2011 12:02:21] PHP Warning:  mysql_real_escape_string() expects parameter 2 to be resource, boolean given in /home/hunterdo/public_html/DiscussionBoard/Sources/Subs-Db-mysql.php on line 132
[01-Jan-2012 05:40:13] PHP Warning:  mysql_real_escape_string() expects parameter 2 to be resource, boolean given in /home/hunterdo/public_html/DiscussionBoard/Sources/Subs-Db-mysql.php on line 132

I have deleted the file on the server but that didn't get access to the forum back, so I have (at least) two problems, darn...!

[Kays]

Deleting that file won't help as it's required.

What I would suggest to do is to backup both your files and database. Then download the large upgrade package, delete the files in the Sources and default theme, then re-upload fresh files from the large upgrade package. Also replace the index.php file at the root of your firoms.

[Vince S]

earlier today I checked the site from an ie7 PC, no problems. and now it is fine in ie9 & FF9. Yet I have taken no further action, bizarre. My last full back-up was 3 months ago, I have just pulled off another one.

The deleted error log file is now 4Mb after a day, this HAS to be a fault of some kind that needs to be separately resolved. I like the idea of doing the large upgrade refresh, thank you, which might fix all issues - but I have realised that I have a persistent virus on my PC. It is one of the obfuscator family and I am yet to find and delete the cause and regularly MSE is telling me about various trojans it deleted plus I get Win 7 Home Security SPAM ware which I can only seem to kill off for a few days at a time.

I probably had the core virus when I did the 2.02 upgrade although that may not be an issue. I am reluctant to touch something that currently isn't broken until I find permanent resolution of the virus issue, which I am suspecting will be a full reinstallation of W7. I will update this topic when I am able to do something more.

[Vince S]

aha, it turns out the website had been compromised, I got a note about it. I have figured out that a script got added to the bottom of most .html files on the website on Dec 28 2011 7:33pm so I progressively went round and replacing any with that status via FTP (from back-ups).

Then onto Google and found http://sitecheck.sucuri.net/scanner/?scan=hunterdog.org.au yet the security audit within CPanel says it is clean. Whatever is going on I now have to deal with the forum retrieval and would ask for assistance with that please. The securi scan originally listed all the public_html directory html files as well, but they are now fixed.

I am trying to figure exactly when I did the 2.02 upgrade, I see the index.php file is dated Dec 28 2011 2:59am, several other files are 12 hours earlier and everything else is older (except the attachments directory related to posts). If that means I did it prior then the infection may have come from me, otherwise I don't know how it got there or how to prevent recurrence. The fact that it was only initially noticeable via SMF may be significant, or may not.

I have attached the script that I have stripped from the end of news_readme.html in the SMF directory, in case anyone is curious. It is plain text only.

I see the SMF error log is up to 29Mb now, this has to be a bad thing, any suggestions about that too?

This is the note, I am quite impressed that someone noticed:
Dear Domain Owner,

CERT Australia has received a report indicating that the following site has been compromised and may now be infecting visitors to the
site:

  *** DO NOT VISIT THIS SITE IN A WEB BROWSER ***

    hunterdog[DOT]org[DOT]au:80

The following pages were identified as containing, or redirecting to malicious code:

    http://hunterdog[DOT]org[DOT]au/DiscussionBoard/index[DOT]php?topic=362[DOT]0
    http://hunterdog[DOT]org[DOT]au/DiscussionBoard/index[DOT]php?topic=756[DOT]0
    http://hunterdog[DOT]org[DOT]au/DiscussionBoard/index[DOT]php?topic=767[DOT]0
    http://hunterdog[DOT]org[DOT]au/DiscussionBoard/index[DOT]php?topic=673[DOT]0
    http://hunterdog[DOT]org[DOT]au/DiscussionBoard/index[DOT]php?topic=563[DOT]0
    http://hunterdog[DOT]org[DOT]au/DiscussionBoard/index[DOT]php?board=1[DOT]0
    http://hunterdog[DOT]org[DOT]au/DiscussionBoard/index[DOT]php?topic=383[DOT]0
    http://hunterdog[DOT]org[DOT]au/DiscussionBoard/index[DOT]php?topic=618[DOT]0
    http://hunterdog[DOT]org[DOT]au/DiscussionBoard/index[DOT]php?topic=480[DOT]0
    http://hunterdog[DOT]org[DOT]au/DiscussionBoard/index[DOT]php?topic=688[DOT]0
    http://hunterdog[DOT]org[DOT]au/DiscussionBoard/index[DOT]php?topic=293[DOT]0
    http://hunterdog[DOT]org[DOT]au/DiscussionBoard/index[DOT]php?topic=399[DOT]0
    http://hunterdog[DOT]org[DOT]au/DiscussionBoard/index[DOT]php?topic=39[DOT]0
    http://hunterdog[DOT]org[DOT]au/DiscussionBoard/index[DOT]php?topic=636[DOT]0
    http://hunterdog[DOT]org[DOT]au/DiscussionBoard/index[DOT]php?topic=415[DOT]0
    http://hunterdog[DOT]org[DOT]au/DiscussionBoard/index[DOT]php?topic=612[DOT]0
    http://hunterdog[DOT]org[DOT]au/DiscussionBoard/index[DOT]php?topic=755[DOT]0
    http://hunterdog[DOT]org[DOT]au/DiscussionBoard/index[DOT]php?topic=638[DOT]0
    http://hunterdog[DOT]org[DOT]au/DiscussionBoard/index[DOT]php?topic=248[DOT]0
    http://hunterdog[DOT]org[DOT]au/DiscussionBoard/index[DOT]php?topic=575[DOT]0
    http://hunterdog[DOT]org[DOT]au/DiscussionBoard/index[DOT]php?topic=175[DOT]0
    http://hunterdog[DOT]org[DOT]au/DiscussionBoard/index[DOT]php?topic=725[DOT]0
    http://hunterdog[DOT]org[DOT]au/DiscussionBoard/index[DOT]php?topic=192[DOT]0
    http://hunterdog[DOT]org[DOT]au/DiscussionBoard/index[DOT]php?topic=558[DOT]0
    http://hunterdog[DOT]org[DOT]au/DiscussionBoard/index[DOT]php?topic=364[DOT]0
    http://hunterdog[DOT]org[DOT]au/DiscussionBoard/index[DOT]php?topic=583[DOT]0
    http://hunterdog[DOT]org[DOT]au/DiscussionBoard/index[DOT]php?topic=644[DOT]0
    http://hunterdog[DOT]org[DOT]au/DiscussionBoard/index[DOT]php?topic=72[DOT]0
    http://hunterdog[DOT]org[DOT]au/DiscussionBoard/index[DOT]php?topic=629[DOT]0
    http://hunterdog[DOT]org[DOT]au/DiscussionBoard/index[DOT]php?board=25[DOT]0
    http://hunterdog[DOT]org[DOT]au/DiscussionBoard/index[DOT]php?topic=70[DOT]0
    http://hunterdog[DOT]org[DOT]au/DiscussionBoard/index[DOT]php?action=search
    http://hunterdog[DOT]org[DOT]au/DiscussionBoard/index[DOT]php?topic=354[DOT]0
    http://hunterdog[DOT]org[DOT]au/DiscussionBoard/index[DOT]php?topic=682[DOT]0
    http://hunterdog[DOT]org[DOT]au/DiscussionBoard/index[DOT]php?topic=741[DOT]0
    http://hunterdog[DOT]org[DOT]au/DiscussionBoard/index[DOT]php?board=24[DOT]0
    http://hunterdog[DOT]org[DOT]au/DiscussionBoard/index[DOT]php?topic=10[DOT]0
    http://hunterdog[DOT]org[DOT]au/DiscussionBoard/index[DOT]php?topic=476[DOT]0
    http://hunterdog[DOT]org[DOT]au/DiscussionBoard/index[DOT]php?topic=720[DOT]0

As at 05/01/2012 this site resolved to:

    184.154.234.3

Where possible, we have provided a copy of this report to the abuse contact for this address, and the website owner.

CERT Australia requests your assistance in resolving this matter, and in providing copies of any log files for this site, information which shows how this compromise occurred, or copies of files placed on this system by an attacker.

Who are we?

CERT Australia is Australia's official national computer emergency response team (CERT). CERT Australia is managed by the Australian Government Attorney-General's Department. The Attorney-General's Department coordinates Australian Government cyber security policy and manages Australia's cyber crisis management arrangements.

[Kindred]

well, here's the deal...

If they added a backdoor file at any time, then it is likely that your entire site has been compromised. This includes any/all smf files and any/all other scripts that you may have running.
If you are running several scripts, without the server logs, we can't tell which one gave them access to the site...
(for example 2 forums on one of my sites were hit through access via a forgotten installation of zenPhoto)

You will have to clean out your ENTIRE site...
this means, checking EVERY file html, php, etc. for extraneous code
checking EVERY directory for extra sub-directories and extra files (like that default.php that your host found)
and changing EVERY password that you use for that site...

If you have a backup of the site prior to the infection, you can restore that (backup - often! and save 2-3 prior backups)


Unfortunately, recovering from a compromised site can be a LOT of work, if you don't have a known, clean backup

[Vince S]

Thanks Kindred, I get it and am struggling along with that.

For the benefit of anyone else looking for answers one very useful thing has turned out to be running the 2.02 upgrade.php script. Normally I just do the upgrades via the admin panel which is fine, but this one I did through Softaculous (a Fantastico competitor that our host Siteground has moved to). But I did it after I edited the .htaccess file to use php version to 5.2 with subhosin extension (was on 5.1). When the upgrade file ran it did a lot of database file rewriting and hey presto I seem to have a clean working forum again. I am not saying this is a definitive answer, and also I have done a lot of research into what a proper complete sitewide answer might involve - as Kindred says it is a lot of work! So I have changed the site p/w & shelled out the $100 for a security audit & fix by SiteGround and will see what else that reveals.

Whilst it is not my purpose to promote anything particular I have found that for cheap hosting I have tried a few over the years and for the last 4 years or so with SiteGround they haven't put a foot wrong, and they were happy to negotiate a 5 year deal at even better rates. Just thought I would mention one person's experience as it is so hard to find reliable cheap hosting with frills!

PS: The reason I thought to run the 2.02 script again (as I was on 2.02) was that Softaculous said it wasn't the current version. It seems to report in a meaningful way, unlike the rubbish that Fantastico used to report.

[JimM]

Glad you were able to clean the site by using the large upgrade as Kays recommended.  Are things still running ok?

If this is solved, please mark it solved by clicking the Mark Topic Solved link at the bottom left.

[Vince S]

G'day, sorry for my tardiness here. I thought I had it all fixed then bolted for a 3 week holiday to HK, KL & environs. Back a few hours ago and its still broke, at least from the perspective that people can't get to the forum or site due to Google barriers. Possibly this is due to Google not updating, but I know before leaving I found that Google's so called review request system doesn't actually let you put in a review request. It will be a few days until I can look into this and either close or put in a meaningful post.........

Also by way of explanation and apology after posting my last info I realised I hadn't specifically thanked Kays (Thank You Kays) who was already on the money but figured I would do that in the closing rather than add more noise (whistfully wondering where is that elusive balance point and why do I sometimes miss the sucker?).

[JimM]

As Kindred mentioned, it's best to clean your entire site when this happens.  That means deleting all files and folders and recreating them.  With all the files on the site, it's difficult to tell if an extra one wasn't placed there by the intruder. 

So are your members still receiving the warning message when they attempt to visit your site?

[Vince S]

It seems to be fine now. Marked as solved. For whatever reason there was some ghosting of the prior issues that was reported by various people and seemed to be something other than a refresh issue. However it cleared itself along the way, or at least the complaints stopped coming and when I had a serious crack at finding problems there were none.

For anyone else reading this my ISP (SiteGround) had some additional security suggestions
(http://kb.siteground.com/article/Basic_security_guidelines_for_the_shared_hosting_server.html)
of which I implemented the following:

Configure your site to use the secure PHP 5.2 with Suhosin patch by adding the following line to your .htaccess file:

AddHandler application/x-httpd-php52s .php .php5 .php4 .php3

(PHP 5.2 has an improved handling of remote code which reduces greatly security problems. The php 5.2.17 with Suhosin patch offers additional security against remote web exploit attacks but note that if your application has vulnerability, this particular PHP version will not fix the security hole in the application. Thus it is better to keep all web applications up to date.)

Note I picked the above rather than 5.3, to switch to PHP 5.3 it is:

    AddHandler application/x-httpd-php53 .php .php5 .php4 .php3

   Deny perl and other bots from accessing your site. This can be easily done with the following rules in your .htaccess:

SetEnvIfNoCase User-Agent libwww-perl bad_bots
order deny,allow
deny from env=bad_bots

    If you are not using Perl scripts, add a bogus handler for these files. In your home directory create a .htaccess file with the following content:

##Deny access to all CGI, Perl, Python and text files
<FilesMatch "\.(cgi|pl|py|txt)">
Deny from all
</FilesMatch>
##If you are using a robots.txt file, please remove the
# sign from the following 3 lines to allow access only to the robots.txt file:
#<FilesMatch robots.txt>
#Allow from all
#</FilesMatch>

The above will prevent Perl scripts to be executed. Many exploits / backdoors are writtent in Perl and the above will prevent them from running. This directive will apply to all your subdirectories.