Constants attacks to server

[Yorel]

Forum: www.mocosoftx.com
Version: 1.1.15

Hi all:

Hope someone could help us with a problem in our forum. Several weeks ago, an user is attacking the forum, we don't know how he does it. When the attack is launched we get the error:

Sorry, SMF was unable to connect to the database. This may be caused by the server being busy. Please try again later.

We thought the attacker floods the server with many connections but, for instance, only 397 connections are active in that moment:

root:/etc# netstat -punta | wc -l
397

We discarded the SYN Flood attack then. There was 147 apache processes and 131 www-data processes, is it normal?. Right now the forum is down for 1 hour and I can't see the number of processes when the forum is up, sorry.

This is the output of vmstat:

root:/var/log# vmstat 5
procs -----------memory---------- ---swap-- -----io---- -system-- ----cpu----
r b swpd free buff cache si so bi bo in cs us sy id wa
0 80 0 521080 0 0 0 0 3 18 0 3 55 6 0 39
0 76 0 538068 0 0 0 0 10 0 0 2357 12 8 0 80
0 78 0 555760 0 0 0 0 0 0 0 1602 8 4 0 88
0 81 0 573800 0 0 0 0 14 0 0 843 18 12 0 69
8 73 0 583532 0 0 0 0 8 0 0 1719 16 12 0 72
0 81 0 582928 0 0 0 0 67 0 0 328 6 5 0 90
0 79 0 580512 0 0 0 0 2053 0 0 674 10 5 0 84
0 78 0 575196 0 0 0 0 5215 0 0 1257 9 5 0 86
0 75 0 565748 0 0 0 0 4150 0 0 1562 11 8 0 82
0 74 0 568000 0 0 0 0 2985 0 0 1624 12 9 0 79
0 72 0 577404 0 0 0 0 9376 0 0 1641 8 7 0 85
0 70 0 597376 0 0 0 0 2045 0 0 998 8 5 0 87
1 79 0 596256 0 0 0 0 42 0 0 340 24 4 0 72

I checked out the dmesg file, messages, syslog, mysql, mysqlerr and several more files but I didn't find any abnormal.

Someone knows what can I do for finding out the source of the attack?, what commands to launch?. We don't have deep knowledge on apache or mysql and, therefore, need someone guides one on this issue.

Many thanks,
Regards

[Ricky.]

It means your mysql is causing problem and its crashing .

If you can show us your status.php then we can suggest you better.
Here you can download it from tool section : http://download.simplemachines.org/?tools , upload status.php to your forum root and give us its link.

Also, do you have any firewall installed on your server ?

[Yorel]

Thanks for replying Ricky. Now we know it's an attack because of this person said he will attack every day. The problem is we don't know what kind of attack is doing.

IPTABLES is installed but not configured properly.

Attach the SQL Status.

BTW, someone knows why our server has so many open connections with simplemachines.org server:

netstat -plan|grep :80 | awk {'print $5'}|cut -d: -f 1|sort|uniq -c|sort -n
   830 66.71.247.134

Thanks again.

[Illori]

BTW, someone knows why our server has so many open connections with simplemachines.org server:

netstat -plan|grep :80 | awk {'print $5'}|cut -d: -f 1|sort|uniq -c|sort -n
   830 66.71.247.134

in 1.1.1* the forum will ask this site for an update for several files to check that they are updated, unlike in 2.0 there is no set time for this to happen, so it may happen many times a day.

[Ricky.]

Well, I don't see any load on your server, unless the person is an pro in all this, he can't let your server down on his own.. if a simple firewall is there. Believe me, I have been in such situation and I was generally facing DOS and then DDOS but properly configured firewall solved it easily. Install CSF / ask your host to do that or do it yourself..

Also, I was expecting link to full status.php .. what  you showed here tells nothing other than that you have very little load on server.

Also, as illori said, not to worry about connection to SMF server !

[Yorel]

Hello:

When the attack is active we don't see huge connections on server, it's very strange. It looks like this person sends a few requests but specially modify.

Ricky, wouldn't be enough to configure IPTABLES properly or is it better CSF?.

I'll upload the link of status.php this morning, now it's impossible.

Thanks!

[butchs]

Interesting.  I had the same issue from unknown attackers.  Did you try my Forum Firewall mod?  Chances are he will show up in the log and you can find out what he is doing and create even more measures.

[Ricky.]

Hello:

When the attack is active we don't see huge connections on server, it's very strange. It looks like this person sends a few requests but specially modify.

Ricky, wouldn't be enough to configure IPTABLES properly or is it better CSF?.

I'll upload the link of status.php this morning, now it's impossible.

Thanks!

Well, yes Iptables can do that, infact CSF uses iptables for that but that requires deep understanding of iptables to configures something similar to CSF ..

[Yorel]

Here goes the link to status.php -> http://mocosoftx.com/foro/status.php

Thanks butchs, we're gonna install that mod, any additional information will help us.

Ricky, I'm gonna study how to configure CSF properly, thanks for your help.

Cheers!

EDIT: butchs, we've installed the mod. The most records we see are "Bypass Try!", what does it mean exactly?.

[Ricky.]

Here goes the link to status.php -> http://mocosoftx.com/foro/status.php

Thanks butchs, we're gonna install that mod, any additional information will help us.

Ricky, I'm gonna study how to configure CSF properly, thanks for your help.

Cheers!

EDIT: butchs, we've installed the mod. The most records we see are "Bypass Try!", what does it mean exactly?.

Alright.. but your status.php do not show us load averages.. ...

[LiroyvH]

Hmm... If your httpd would not be responding properly i'd say this is a SlowLoris attack...
Would of checked *extended* httpd status to see what kind of requests it is processing.
However, since your httpd server appears to still be responding it's probably not a SlowLoris, but: I would still check your httpd extended status.

Is your mySQL port closed for outside connections...?

[Yorel]

We're waiting to collect the extended status when an attack is taking place.

The mysql port is closed for everyone, only there is someting strange when I launch the iptables -L command:

root:~# iptables -L
Chain INPUT (policy ACCEPT)
target     prot opt source               destination         
FATAL: Could not load /lib/modules/2.6.32-4-pve/modules.dep: No such file or directory
FATAL: Could not load /lib/modules/2.6.32-4-pve/modules.dep: No such file or directory

I didn't find useful information on that, is it a critical error?

[LiroyvH]

That depends.

Can you run:
cat /etc/sysconfig/iptables-config | grep MODULES

[Yorel]

root:~# cat /etc/sysconfig/iptables-config | grep MODULES
cat: /etc/sysconfig/iptables-config: No such file or directory
root:~#



We know IPTABLES is working because, last week, we closed the mysql port. Before it, anyone could connect through 3306, once the rule was configured, the port is closed.

[butchs]

Here goes the link to status.php -> http://mocosoftx.com/foro/status.php

Thanks butchs, we're gonna install that mod, any additional information will help us.

If you take the time to set-up the mod and properly test it the job should get done.  Please test it for a few days before enabling it to prevent from blocking yourself.  Once you get it running, I would change all of the passwords for the forum, php, admin and moderators.

I recommend that you set up your DOS protection as per Adjusting DOS Protection HELP.  Then reinstall the mod.  Upon installation the mod will read your robots,txt file and use it to protect your site.

EDIT: butchs, we've installed the mod. The most records we see are "Bypass Try!", what does it mean exactly?.

Someone tried to access the admin account who was not from the configured domain and ip range.  If this is true, you may have his information and can file a complaint to his internet provider & etc.

Your Bypass protection may not be properly set-up and if you enable the mod you may block yourself.  See Bypass protection Help for instructions.  You can search for other Helps in the thread for detailed assistance.

[Yorel]

Right now the navigation across the forum is very very slow, we think the attack is back.

Attach a txt file with the SQL status, right now there are almost 109k connections!

Also attached the my.cnf and status

[butchs]

Do you have cpanel?  If so can you show us your last 300 connections and error log?

[Ricky.]

Looks like still you don't have any firewall installed !

[Yorel]

It doesn't make sense, right now we have 161k connections and the forum goes perfect O.O. I'll ask to my colleague about the cpanel, I dont know if it's installed.

The error.log is flooded of the same message, it appears several times in the same second:

[error] [client 66.71.247.134] File does not exist: /var/www/mods

I launched a grep -v "File does not exist" error.log and you can see the result in attached file (we had to reinstall the server yesterday from zero).

Due to the forum was very very slow we installed the APC, I can see many errors messagess like this:

[apc-warning] Potential cache slam averted for key ...

No fw is installed right now, we dont know how to configure IPTABLES properly. What is your recomendation?, Shorewall, Firestarter?..

Thanks for your help

[butchs]

Ot looks like they can not frond:

index.php - is not in your root folder.
Sources/Load.php -  Line 2649 has cache slam averted - info here.

Getting rid of the errors should help things.