Bad Behavior for SMF mod

[Etcher]

Yum...  I am the only one in my house who loves it for breakfast.  Thank goodness they now offer it in single pack servings.  Perfect to brown to golden perfection for breakfast!

Amen to that. They are especially good when mixed with eggs.

[butchs]

The Bad Behavior core site was down for a week due to a HD failure and is now back up.  Click here for an explanation.

[djkimmel]

Any idea when the latest version of the Bad Behavior mod will be available for download again? Is Bad_Behavior_1.5.10 the latest, safe version to use? I was going to install it on my version 1.16 forum but hesitated when I saw that Bad Behavior isn't available for download apparently? Thanks.

[djkimmel]

Never mind. I just read the other thread on your website. Jeez. I have 1.5.9 installed. Guess I will stay with that version for now. Hope this gets worked out.

[TheListener]

Good to see the mods back where it should be.

[butchs]

I do my utmost to strive for perfection...  my standards are held to the highest level.  So when there was a claim that my mod was unsafe, I immediately pulled the mod.  Tested the heck out of it...  Revised some code...  Then asked for a peer review by SMF.  The mod passed the review so it is back.

[TheListener]

I do my utmost to strive for perfection...  my standards are held to the highest level.

As should all mod authors.

[Sudhakar Arjunan]

I is really a premium link spam killer which i enjoy after installing the mod.

Love it.

[djkimmel]

Something changed this morning in Bad Behavior. I'm using SMF 1.1.16. I have hundreds of pages of errors for various Google bots and Bing Bots. All the IP addresses are in ranges assigned to Google and Microsoft. I had Bad Behavior 1.5.9 installed but I upgraded to 1.5.11 this afternoon to see if that would make a difference though I haven't changed anything else in some time. Bad Behavior is still blocking a large number, possibly all since I haven't seen any get through yet, of search engine bot requests?

Most of the errors are this one (or the msnbot one KEY:   e4de0453): KEY:   f1182195
DENIED REASON:   User-Agent claimed to be Googlebot, claim appears to be false.
EXPLANATION:   An invalid request was received. You claimed to be a major search engine, but you do not appear to actually be a major search engine.

IP:   131.253.41.246
DATE:   2012-08-23 19:03:50
METHOD:   GET
URI:   /vid/Jordan+Jansen/
PROTOCOL:   HTTP/1.1
HEADERS:   GET /vid/Jordan+Jansen/ HTTP/1.1 Accept: text/html, text/plain, text/text, text/javascript, text/x-javascript, text/js, text/x-js, text/jscript, text/ecmascript, text/xml, xml/xml, image/*, audio/*, video/*, application/* Accept-Encoding: gzip, deflate Connection: Close From: msnbot(at)microsoft.com Host: w w w.greatlakesbass.com User-Agent: msnbot-media/1.1 (+http://search.msn.com/msnbot.htm)
AGENT:   msnbot-media%2F1.1%20%28%2Bhttp%3A%2F%2Fsearch.msn.com%2Fmsnbot.htm%29
ENTITY:   
KEY:   e4de0453
DENIED REASON:   User-Agent claimed to be msnbot, claim appears to be false
EXPLANATION:   An invalid request was received. You claimed to be a major search engine, but you do not appear to actually be a major search engine.

Tons of rejected IP 157.55. and 157.56., 131.253.39-.41, 131.256.46-.47 and some 66.249.73 (see below where Bad Behavior says a Mediapartners-Googlebot is on the blacklist too?). User-Agent: Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm) AGENT:   Mozilla%2F5.0%20%28compatible%3B%20bingbot%2F2.0%3B%20%2Bhttp%3A%2F%2Fwww.bing.com%2Fbingbot.htm%29 being blocked?

IP:   66.249.73.137
crawl-66-249-73-137.googlebot.com
DATE:   2012-08-30 17:46:46
METHOD:   GET
URI:   /
PROTOCOL:   HTTP/1.1
HEADERS:   GET / HTTP/1.1 Accept: */* Accept-Encoding: gzip,deflate Connection: Keep-alive From: googlebot(at)googlebot.com Host: w w w.greatlakesbass.com User-Agent: Mozilla (compatible; Mediapartners-Googlebot/2.1;(+http://www.google.com/bot.html)
AGENT:   Mozilla%20%28compatible%3B%20Mediapartners-Googlebot%2F2.1%3B%28%2Bhttp%3A%2F%2Fwww.google.com%2Fbot.html%29
ENTITY:   
KEY:   17f4e8c8
DENIED REASON:   User-Agent was found on blacklist

Also a bunch of errors like the one below from non-bot IP addresses around the world that do not say they are Googlebot, but are being rejected with the same code. They just appear to be referred from search:
IP:   50.140.***.***
c-50-140-***-***.hsd1.in.comcast.net
DATE:   2012-08-30 17:47:15
METHOD:   GET
URI:   /forum/index.php?topic=3597.0
PROTOCOL:   HTTP/1.1
HEADERS:   GET /forum/index.php?topic=3597.0 HTTP/1.1 Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8 Accept-Encoding: gzip,deflate Host: w w w.greatlakesbass.com Referer: http://w w w.google.com/search User-Agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/536.8 (KHTML, like Gecko; Google Web Preview) Chrome/19.0.1084.36 Safari/536.8 X-Forwarded-For: 50.140.***.***
AGENT:   Mozilla%2F5.0%20%28X11%3B%20Linux%20x86_64%29%20AppleWebKit%2F536.8%20%28KHTML%2C%20like%20Gecko%3B%20Google%20Web%20Preview%29%20Chrome%2F19.0.1084.36%20Safari%2F536.8
ENTITY:   
KEY:   f1182195
DENIED REASON:   User-Agent claimed to be Googlebot, claim appears to be false.
EXPLANATION:   An invalid request was received. You claimed to be a major search engine, but you do not appear to actually be a major search engine.
ERROR:   403

It also appears that a bunch more than normal bad registration attempts and requests like this - //index.php?option=com_user&view=reset&layout=confirm - that are usually blocked are being permitted now. I'm going through my server looking for changes I didn't make and not finding anything so far. I added some spaces in a few spots to not create links.

[djkimmel]

Not sure if I should or shouldn't have posted all the stuff above. I did find that the 1 Google IP Bad Behavior block as blacklisted was listed in CBL 9 days ago claiming to be infected with Zeus.

[butchs]

Just because they are in your log it does not make them errors.

The first ip "131.253.41.246" clearly does not belong to MSN. 

For the google webpreview bots try leaving "Engine DNS" unchecked.

[djkimmel]

I think I've always had Search Engine DNS unchecked. It was unchecked when I upgraded from 1.5.9 to 1.5.11. Looking through my records to see if I made a note of that or not. I'm still going through the ~800 pages of Bad Behavior rejected requests and the ~800 pages of forum errors that go with them to figure out what could have happened. Almost all my Google, Yahoo and Bing search engine traffic went from response 200 (including the entire 131.253.38-47 Microsoft range) to 403 on ~8/30/2012. Also those Google and Bing Preview user agents started getting rejected. I usually have ~110 pages of Bad Behavior rejections per week, not 800 pages in 1 1/2 days.

I usually have 12 to 18 pages of my old "Database Error: No database selected
File: /home/djkimmel/public_html/forum/Sources/bad-behavior/BadBehavior-SMF.php
Line: 74" per day (never did get that issue figured out) not ~800 pages in one day! Something changed. Since I'm seeing a mix of good RDNS and bad RDNS, and IP addresses in the Bad Behavior search engine file and IP addresses not in the Bad Behavior search engine file all rejected along with the preview user agents starting the morning of 8/30/2012 I have not figured out any pattern yet. Especially considering the 131.253.41 (and .40) that you think don't belong to MSN were all going through Bad Behavior from 8/1/2012 until 8/29/2012 without a problem?

I'm also seeing my normal smattering of these type that Bad Behavior normally did a really good job of rejecting starting about the same time - 202.57.0.19   2012-08-31 21:25:57   /?option=com_myblog&Itemid=12&task=../../../../../../../../../../../../..//proc/self/environ%0000   all ending up in PERMITTED #00000000 instead. I've only had 4 of them during this time but they were all permitted. I normally did not see that in the past.

Not seeing any file changes on my server I didn't make. Not seeing any unauthorized FTP or even a root connection by anyone other than me during this time (including none from my host). I added the search engine IP ranges I wanted to the Bad Behavior whitelist to get them to stop getting 403. I wish I could figure out why almost all the search engine traffic started getting rejected by Bad Behavior all of sudden? Just can't see a pattern. Of course I barely know how Bad Behavior works.

[Kindred]

Probably because it's not real search engine traffic...  You seem to have made it onto some spammer or hacker list.

[butchs]

I usually have 12 to 18 pages of my old "Database Error: No database selected
File: /home/djkimmel/public_html/forum/Sources/bad-behavior/BadBehavior-SMF.php
Line: 74" per day (never did get that issue figured out) not ~800 pages in one day! Something changed.

I reviewed the mod and as far as I know there was no change there.  This could be an error in your settings or a bot.  The mod uses $db_prefix.  I am interested in eliminating this error. Start by looking at your "Settings.php" in your root directory and insure that $db_prefix is correctly defined.  You may want to try repair settings.

[butchs]

Probably because it's not real search engine traffic...  You seem to have made it onto some spammer or hacker list.

I subscribe myself  to that list. 

Now I must admit there was an error in older versions of Bad Behavior for SMF performed a reverse DNS test wether or not "Search Engine DNS" was checked.  This was a bug fix after the Core Author bashed me about it's existence.  The latest version of BB for SMF obeys the check mark.

If you wish to return to the way the last version worked check "Search Engine DNS".

I have been watching "google web preview" with "Search Engine DNS" checked.  So far, I found only one way to duplicate the visit on my site.   I forced it by logging in the google web-masters site and previewed my own site.  Attachment1 is the result.

Note attachement1 has only one X-Forwarded_for address.

Lets look more closely at some who failed.  See attachment2.

Notice Cf-Connecting-Ip: 209.85.224.99 "far right of x-forwarded-for".  This is actually the last ip in a proxy list:  X-Forwarded-For: 68.35.128.190,209.85.224.99.

There are two Ip addresses are from the list.  This is a ip spoof attempt where the first address is the client.  The first address has a project honey-pot threat rating of 17 (http://www.projecthoneypot.org/ip_68.35.128.190).  More important it is not a google address.

Bad Behavior only tests the proxy address on the far right of the "X-Forwarded-For" list.  Anyone in this list can have access to your site.

Lets look at attachmnet3.  Cf-Connecting-Ip: 74.125.178.86 "far right".  This is actually the last ip in a proxy list:  X-Forwarded-For: 108.50.185.118,74.125.178.86

This is a ip spoof attempt where the first address is the client.    108.50.185.118 is from the UNKNON REALM of the internet.

After some research I was able to go to google web-masters and duplicated the error.  Google did not report the error in their log.  I validated and tried to preview my site and to get error in attachement4.

The first "censored" address in the X-Forwarded-For is mine.  The second is google.  The mod blocked it because, when it performed the reverse DNS it found my address/ host, NOT Google.  The preview image was blocked.

It only happens when I visit the home page of web-master tools where my domain names are listed.

Google reports no 403 errors.  I have full ability to use the tools as I wish.

I can neither confirm nor deny without a shadow of a doubt that the "google web preview" visits are bad guys.  I believe they are after reading djkimmel's posts.

The detriment I see when the "Search Engine DNS" feature works correctly on a non-Ubuntu Server run site is that you can block yourself from google web-masters preview.

[djkimmel]

I usually have 12 to 18 pages of my old "Database Error: No database selected
File: /home/djkimmel/public_html/forum/Sources/bad-behavior/BadBehavior-SMF.php
Line: 74" per day (never did get that issue figured out) not ~800 pages in one day! Something changed.

I reviewed the mod and as far as I know there was no change there.  This could be an error in your settings or a bot.  The mod uses $db_prefix.  I am interested in eliminating this error. Start by looking at your "Settings.php" in your root directory and insure that $db_prefix is correctly defined.  You may want to try repair settings.

I will try repair settings after I look in my Settings.php file. Thank you. I use a 'hand-customized' version of the 1.1.16 default theme that may very well be the cause which is one reason I have not continued to bug you about the 'No database selected' errors. They could very well be entirely my fault by doing something to the theme index.template.php that my lack of in-depth knowledge and time to figure it out is solely the cause. I use SSI in other parts of my website and I believe I may have made a couple 'minor' edits to the SSI.php file some time ago (not sure - my memory is beyond terrible and I don't always keep meticulous notes).

I don't want you to think I believe it is your fault or that I'm not appreciative of your Mod. It has helped me a ton. A TON! I'm also extra paranoid about hackers right now because a number of my clients and friends have recently been hacked and I was hacked back in March overnight before I caught and started to fix it. So I'm more sleep deprived than normal searching logs, file changes and databases for any sign of trouble. So far, sleep is the only thing I have lost. So much to learn. Not enough time. Thanks for offering to help with these errors.

[djkimmel]

Well, thank you for going above and beyond in your reply! I do appreciate it. I will look more closely at some of the log information to see if I can spot any information that helps me figure out what changed that day. I do figure the more spam and attempts at bad stuff I get, the more popular I must be!  Might not be the best way to look at it.

I was told by someone else online who seems to know a lot more than me that the 131.253.40-41 IP's are all Microsoft media search bots too. Arin says they are Microsoft-owned? They could be something bad but they follow the same exact patterns and behavior that the 131.253.46-47 IP's have when searching media on my site. I may get a Bing Webmaster account to see if I get a clue from their about why some of the 131.253 addresses have proper RDNS and some don't. It was the combination of them all getting 200 responses from 8/1/2012 through 8/29/2012 and then suddenly starting to get 403 responses along with the good RDNS 131.253.46-47 range AND known Google AND Yahoo that really threw me.

I now see after looking closer that some IP addresses claiming to be Yahoo that look good to me were getting rejected as not Yahoo Slurp! by SMF Bad Behavior before 8/30/2012. Since so many of them hit other parts of my website that does not come under SMF BB I just saw the numerous 200 responses. I see a 403 back on 7/31/2012 for IP 98.137.72.251 - - [31/Jul/2012:22:01:38 -0500] "GET /forum/index.php?topic=2630.0 HTTP/1.1" 403 690 "-" "Mozilla/5.0 (compatible; Yahoo! Slurp/3.0; http://help.yahoo.com/help/us/ysearch/slurp)". That IP appears to me to legitimately be from Yahoo and in the BB search engine range of 98.136.0.0/14 but maybe my logs don't capture enough information?

I will look closer at more of the Googlebot and Yahoo Slurp! user agent traffic later today.

I'd estimate 85% of the rejects are bingbot/MSNbot-media with a good mix of the IP addresses with proper RDNS and in the Bad Behavior search engine ranges and those with bad RDNS/not in the BB search engine ranges. I will look closer at more of those with your extremely helpful additional information. This is definitely not my area of expertise. I may have to ask my host if there is additional log information my setup does not capture that might explain some of the rejects that seem legitimate?

I enacted some of the SMF forum performance improvement suggestions back, I think, in June (move avatars, etc). I know I changed some host look up stuff then. I may have unchecked the Bad Behavior Search Engine DNS option at that time, if I ever had it checked, to save on resources. I'd prefer not to use it anyway for that reason. I want to be indexed well but I have better performance for visitors ranked higher. I did not know hardly anything when I first installed SMF back in December 2005 about PHP and all this stuff. I do need to redo some things for sure I know.

I did go through last night and confirm all the SMF BB files and edits are updated and correct on my domain. Thanks for the help.

[butchs]

...I was told by someone else online who seems to know a lot more than me that the 131.253.40-41 IP's are all Microsoft media search bots too. Arin says they are Microsoft-owned?...

The ip list of addresses that are blocked by Bad Behavior are maintained by the Core Author.  I do mess with the list.

He just released a new version and the addresses you list are included in BB 2.2.9 which was released this morning.  Until I get a chance to update the mod please open "searchengine.inc.php" and make the following changes:

Search:

Code Select Expand

(match_cidr($package['ip'], array("207.46.0.0/16", "65.52.0.0/14", "207.68.128.0/18", "207.68.192.0/20", "64.4.0.0/18", "157.54.0.0/15", "157.60.0.0/16", "157.56.0.0/14")) === FALSE)

Replace with:

Code Select Expand

(match_cidr($package['ip'], array("207.46.0.0/16", "65.52.0.0/14", "207.68.128.0/18", "207.68.192.0/20", "64.4.0.0/18", "157.54.0.0/15", "157.60.0.0/16", "157.56.0.0/14", "131.253.21.0/24", "131.253.22.0/23", "131.253.24.0/21", "131.253.32.0/20")) === FALSE)

If you see other ip addresses like the above please report them to the Core Author.  I do not maintain the list.  According to the core website:
To report an issue with Bad Behavior, send email to bad . bots at ioerror dot us.

I see a 403 back on 7/31/2012 for IP 98.137.72.251 - - [31/Jul/2012:22:01:38 -0500] "GET /forum/index.php?topic=2630.0 HTTP/1.1" 403 690 "-" "Mozilla/5.0 (compatible; Yahoo! Slurp/3.0; http://help.yahoo.com/help/us/ysearch/slurp)". That IP appears to me to legitimately be from Yahoo and in the BB search engine range of 98.136.0.0/14 but maybe my logs don't capture enough information?

As I explained in my last post "98.137.72.251" due to the recent bug fix that ip should be ok with the latest version of BB for SMF with "Search Engine DNS" unchecked. BB will not notice if it was spoofed...

The search engine match CIDR range test is strictly maintained by the core author.  So if you see a problem with the latest version for SMF with "Search Engine DNS" unchecked, then please report it to the Core Author.   According to the core website:
To report an issue with Bad Behavior, send email to bad . bots at ioerror dot us.

[butchs]

New version.  Upgraded to 2.2.9.

Known Changes:

• Several robots associated with spam and malicious activity have been identified and blocked.
• Several patterns associated with malicious activity such as SQL injection and vulnerability scanning have been identified and blocked.
• Microsoft-owned address check fixed.
• Corrected error where the malicious activity address variable was incorrect in the core 2.2.9 release (SMF ONLY).

[djkimmel]

Great! I understand things a lot better now. I had wanted to ask about clarification about who to ask which questions. I should have seen that already. Sorry. You've set me right on that. Thank you. I will upgrade tomorrow. Went fishing today instead of looking at any more files and logs. Thanks updating your mod to include the new BB with new scanning patterns. It does appear in my logs that some hackers have updated their methods recently which added to my personal confusion.

I don't use the Search Engine DNS checked option but I'm curious to see if he changed his Yahoo round trip check from crawl.yahoo.net to what they appear to use now - yst.yahoo.net. Not a big deal for me either way though it will add a little to my understanding of how your mod and his core works.

Thank you so much for keeping this most helpful mod going and updated. I want to remove some of the IP addresses I recently whitelisted and let your mod and BB handle things.