block bad bos

Started by mehrtadbir, November 25, 2014, 06:40:25 PM

Previous topic - Next topic

mehrtadbir

Hi

how to block bad bots?

It's my last visit of bots

Googlebot41,897+44481.46 MB25 Nov 2014 - 00:59
Bingbot206,606+4532.70 GB25 Nov 2014 - 00:59
Yahoo Slurp38,092+47490.58 MB25 Nov 2014 - 00:59
Unknown robot (identified by 'spider')4243.66 MB25 Nov 2014 - 00:44
Unknown robot (identified by 'bot' followed by a space or one of the following characters _+:,.;/\-)7,070+49780.76 MB25 Nov 2014 - 00:30
Unknown robot (identified by 'crawl')9,332+144149.60 MB25 Nov 2014 - 00:19
Unknown robot (identified by empty user agent string)660+716.02 MB25 Nov 2014 - 00:12
Bing Preview bot3,60644.45 MB25 Nov 2014 - 00:04
Feedfetcher-Google8442.16 MB24 Nov 2014 - 23:42
Yandex Bot0+30274.25 KB24 Nov 2014 - 23:12
Googlebot-Image2421.18 MB24 Nov 2014 - 22:16
Common *nix tool for automating web document retireval. Most likely a bot.2412.75 MB24 Nov 2014 - 21:54
yie87,57975.58 MB24 Nov 2014 - 21:23
Powermarks15377.38 KB24 Nov 2014 - 21:11
FavIconizer861019.29 KB24 Nov 2014 - 20:08
Alexa (IA Archiver)17+1042.26 MB24 Nov 2014 - 19:53
WebFilter71+2591.51 MB24 Nov 2014 - 18:56
Sogou Spider10+1591.42 MB24 Nov 2014 - 18:44
Grapeshot Crawler1+1651.73 KB24 Nov 2014 - 18:43
A6-Indexer9+9198.96 KB24 Nov 2014 - 18:20
Unknown robot (identified by 'checker')5431.26 KB24 Nov 2014 - 17:18
A PHP script55815.16 MB24 Nov 2014 - 15:23
CFNetwork2137.55 KB24 Nov 2014 - 14:58
Proximic Spider0+44403.86 KB24 Nov 2014 - 14:21
Unknown robot (identified by hit on 'robots.txt')0+165654.99 KB24 Nov 2014 - 14:05
MJ12bot278+35716.51 MB24 Nov 2014 - 13:35
windows-rss-platform3471.96 KB24 Nov 2014 - 12:05
Unknown robot (identified by 'robot')4+14131.07 KB24 Nov 2014 - 11:01
FaceBook bot2,1027.22 MB24 Nov 2014 - 10:16
MagpieRSS1,538+2312.89 MB24 Nov 2014 - 09:13
TITAN42476.35 KB24 Nov 2014 - 08:52
arks9864.56 MB24 Nov 2014 - 08:05
archive.org bot2+13228.06 KB24 Nov 2014 - 02:03
twitterbot16+10271.19 KB24 Nov 2014 - 00:13
Exabot0+1844.73 KB23 Nov 2014 - 21:43
Phantom1514.17 KB23 Nov 2014 - 21:12
SemrushBot21+5260.97 KB23 Nov 2014 - 14:21
Suspected bot masqurading as Mozilla1363.55 MB23 Nov 2014 - 13:10
SeznamBot0+1434.72 KB23 Nov 2014 - 05:11
Java (Often spam bot)6465.46 KB23 Nov 2014 - 04:39
rpt-httpclient14122.56 KB23 Nov 2014 - 04:14
oBot2175.85 KB23 Nov 2014 - 03:23
MSNBot-media8+8159.65 KB22 Nov 2014 - 20:20
Python-urllib88+3926.59 KB22 Nov 2014 - 17:42
Fish search235.15 KB22 Nov 2014 - 16:41
Eniro Sverige, email: search (at) eniro.com4+359.72 KB22 Nov 2014 - 14:23
Tarantula2170.25 KB22 Nov 2014 - 14:05
butterfly1661.16 MB22 Nov 2014 - 08:05
Nutch5+673.91 KB21 Nov 2014 - 16:04
MSNBot876+11.22 MB21 Nov 2014 - 00:55
BaiDuSpider26+7440.84 KB20 Nov 2014 - 16:13
360spider224.30 KB20 Nov 2014 - 08:32
Suspected bot masqurading as Mozilla71951.23 KB20 Nov 2014 - 07:46
Site Explorer299+221.84 MB20 Nov 2014 - 00:12
VORTEX28201.09 KB19 Nov 2014 - 14:20
Web Core / Roots152864.21 KB19 Nov 2014 - 11:38
Google AdSense225.42 KB19 Nov 2014 - 03:21
Jakarta commons-httpclient2+185.52 KB18 Nov 2014 - 20:14
Zend Http Client21.47 KB18 Nov 2014 - 19:24
AhrefsBot2+10203.29 KB18 Nov 2014 - 18:03
SearchmetricsBot0+19.19 KB18 Nov 2014 - 08:15
WordPress1371.68 MB16 Nov 2014 - 18:09
Mail.ru bot0+32293.90 KB16 Nov 2014 - 02:37
Unknown robot (identified by 'bot' preceded by a space or one of the following characters _+:,.;/\-)349+13.86 MB15 Nov 2014 - 12:46
SurveyBot1+121.11 KB13 Nov 2014 - 14:16
SEOprofiler Bot0+12.51 KB13 Nov 2014 - 00:35
Perl tool2894 Bytes11 Nov 2014 - 07:52
ips-agent Verisign(?) - no reliable information found.11+21.20 MB11 Nov 2014 - 01:41
WGet tools0+19.09 KB08 Nov 2014 - 20:12
Microsoft Office Protocol Discovery/Microsoft Office Existence Discovery218.61 KB08 Nov 2014 - 19:03
Unknown robot (identified by 'discovery')49513.95 KB08 Nov 2014 - 11:24
MSIECrawler3102.21 KB08 Nov 2014 - 01:10
Abonti WebSearch128+11.05 MB06 Nov 2014 - 21:57
W3C Validator111.88 KB06 Nov 2014 - 21:25
تالار گفتگوی تخصصی متا    دفتر تلفن مشاغل   سامانه پیام کوتاه ویژه اعضای دفتر تلفن مشاغل     شبکه تجارت الکترونیک البرز

mehrtadbir

How do I stop unknown robots

I use below cod in htaccess but unknown robots crawles my site still.

Options All -Indexes
RewriteEngine on
# Block Bad Bots & Scrapers
SetEnvIfNoCase User-Agent "MJ12bot" bad_bot
SetEnvIfNoCase User-Agent "pingdom" bad_bot
SetEnvIfNoCase User-Agent "Aboundex" bad_bot
SetEnvIfNoCase User-Agent "80legs" bad_bot
SetEnvIfNoCase User-Agent "360Spider" bad_bot
SetEnvIfNoCase User-Agent "^Java" bad_bot
SetEnvIfNoCase User-Agent "^Cogentbot" bad_bot
SetEnvIfNoCase User-Agent "^Alexibot" bad_bot
SetEnvIfNoCase User-Agent "^asterias" bad_bot
SetEnvIfNoCase User-Agent "^attach" bad_bot
SetEnvIfNoCase User-Agent "^BackDoorBot" bad_bot
SetEnvIfNoCase User-Agent "^BackWeb" bad_bot
SetEnvIfNoCase User-Agent "Bandit" bad_bot
SetEnvIfNoCase User-Agent "^BatchFTP" bad_bot
SetEnvIfNoCase User-Agent "^Bigfoot" bad_bot
SetEnvIfNoCase User-Agent "^Black.Hole" bad_bot
SetEnvIfNoCase User-Agent "^BlackWidow" bad_bot
SetEnvIfNoCase User-Agent "^BlowFish" bad_bot
SetEnvIfNoCase User-Agent "^BotALot" bad_bot
SetEnvIfNoCase User-Agent "Buddy" bad_bot
SetEnvIfNoCase User-Agent "^BuiltBotTough" bad_bot
SetEnvIfNoCase User-Agent "^Bullseye" bad_bot
SetEnvIfNoCase User-Agent "^BunnySlippers" bad_bot
SetEnvIfNoCase User-Agent "^Cegbfeieh" bad_bot
SetEnvIfNoCase User-Agent "^CheeseBot" bad_bot
SetEnvIfNoCase User-Agent "^CherryPicker" bad_bot
SetEnvIfNoCase User-Agent "^ChinaClaw" bad_bot
SetEnvIfNoCase User-Agent "Collector" bad_bot
SetEnvIfNoCase User-Agent "Copier" bad_bot
SetEnvIfNoCase User-Agent "^CopyRightCheck" bad_bot
SetEnvIfNoCase User-Agent "^cosmos" bad_bot
SetEnvIfNoCase User-Agent "^Crescent" bad_bot
SetEnvIfNoCase User-Agent "^Custo" bad_bot
SetEnvIfNoCase User-Agent "^AIBOT" bad_bot
SetEnvIfNoCase User-Agent "^DISCo" bad_bot
SetEnvIfNoCase User-Agent "^DIIbot" bad_bot
SetEnvIfNoCase User-Agent "^DittoSpyder" bad_bot
SetEnvIfNoCase User-Agent "^Download\ Demon" bad_bot
SetEnvIfNoCase User-Agent "^Download\ Devil" bad_bot
SetEnvIfNoCase User-Agent "^Download\ Wonder" bad_bot
SetEnvIfNoCase User-Agent "^dragonfly" bad_bot
SetEnvIfNoCase User-Agent "^Drip" bad_bot
SetEnvIfNoCase User-Agent "^eCatch" bad_bot
SetEnvIfNoCase User-Agent "^EasyDL" bad_bot
SetEnvIfNoCase User-Agent "^ebingbong" bad_bot
SetEnvIfNoCase User-Agent "^EirGrabber" bad_bot
SetEnvIfNoCase User-Agent "^EmailCollector" bad_bot
SetEnvIfNoCase User-Agent "^EmailSiphon" bad_bot
SetEnvIfNoCase User-Agent "^EmailWolf" bad_bot
SetEnvIfNoCase User-Agent "^EroCrawler" bad_bot
SetEnvIfNoCase User-Agent "^Exabot" bad_bot
SetEnvIfNoCase User-Agent "^Express\ WebPictures" bad_bot
SetEnvIfNoCase User-Agent "Extractor" bad_bot
SetEnvIfNoCase User-Agent "^EyeNetIE" bad_bot
SetEnvIfNoCase User-Agent "^Foobot" bad_bot
SetEnvIfNoCase User-Agent "^flunky" bad_bot
SetEnvIfNoCase User-Agent "^FrontPage" bad_bot
SetEnvIfNoCase User-Agent "^Go-Ahead-Got-It" bad_bot
SetEnvIfNoCase User-Agent "^gotit" bad_bot
SetEnvIfNoCase User-Agent "^GrabNet" bad_bot
SetEnvIfNoCase User-Agent "^Grafula" bad_bot
SetEnvIfNoCase User-Agent "^Harvest" bad_bot
SetEnvIfNoCase User-Agent "^hloader" bad_bot
SetEnvIfNoCase User-Agent "^HMView" bad_bot
SetEnvIfNoCase User-Agent "^HTTrack" bad_bot
SetEnvIfNoCase User-Agent "^humanlinks" bad_bot
SetEnvIfNoCase User-Agent "^IlseBot" bad_bot
SetEnvIfNoCase User-Agent "^Image\ Stripper" bad_bot
SetEnvIfNoCase User-Agent "^Image\ Sucker" bad_bot
SetEnvIfNoCase User-Agent "Indy\ Library" bad_bot
SetEnvIfNoCase User-Agent "^InfoNaviRobot" bad_bot
SetEnvIfNoCase User-Agent "^InfoTekies" bad_bot
SetEnvIfNoCase User-Agent "^Intelliseek" bad_bot
SetEnvIfNoCase User-Agent "^InterGET" bad_bot
SetEnvIfNoCase User-Agent "^Internet\ Ninja" bad_bot
SetEnvIfNoCase User-Agent "^Iria" bad_bot
SetEnvIfNoCase User-Agent "^Jakarta" bad_bot
SetEnvIfNoCase User-Agent "^JennyBot" bad_bot
SetEnvIfNoCase User-Agent "^JetCar" bad_bot
SetEnvIfNoCase User-Agent "^JOC" bad_bot
SetEnvIfNoCase User-Agent "^JustView" bad_bot
SetEnvIfNoCase User-Agent "^Jyxobot" bad_bot
SetEnvIfNoCase User-Agent "^Kenjin.Spider" bad_bot
SetEnvIfNoCase User-Agent "^Keyword.Density" bad_bot
SetEnvIfNoCase User-Agent "^larbin" bad_bot
SetEnvIfNoCase User-Agent "^LexiBot" bad_bot
SetEnvIfNoCase User-Agent "^lftp" bad_bot
SetEnvIfNoCase User-Agent "^libWeb/clsHTTP" bad_bot
SetEnvIfNoCase User-Agent "^likse" bad_bot
SetEnvIfNoCase User-Agent "^LinkextractorPro" bad_bot
SetEnvIfNoCase User-Agent "^LinkScan/8.1a.Unix" bad_bot
SetEnvIfNoCase User-Agent "^LNSpiderguy" bad_bot
SetEnvIfNoCase User-Agent "^LinkWalker" bad_bot
SetEnvIfNoCase User-Agent "^lwp-trivial" bad_bot
SetEnvIfNoCase User-Agent "^LWP::Simple" bad_bot
SetEnvIfNoCase User-Agent "^Magnet" bad_bot
SetEnvIfNoCase User-Agent "^Mag-Net" bad_bot
SetEnvIfNoCase User-Agent "^MarkWatch" bad_bot
SetEnvIfNoCase User-Agent "^Mass\ Downloader" bad_bot
SetEnvIfNoCase User-Agent "^Mata.Hari" bad_bot
SetEnvIfNoCase User-Agent "^Memo" bad_bot
SetEnvIfNoCase User-Agent "^Microsoft.URL" bad_bot
SetEnvIfNoCase User-Agent "^Microsoft\ URL\ Control" bad_bot
SetEnvIfNoCase User-Agent "^MIDown\ tool" bad_bot
SetEnvIfNoCase User-Agent "^MIIxpc" bad_bot
SetEnvIfNoCase User-Agent "^Mirror" bad_bot
SetEnvIfNoCase User-Agent "^Missigua\ Locator" bad_bot
SetEnvIfNoCase User-Agent "^Mister\ PiX" bad_bot
SetEnvIfNoCase User-Agent "^moget" bad_bot
SetEnvIfNoCase User-Agent "^Mozilla/3.Mozilla/2.01" bad_bot
SetEnvIfNoCase User-Agent "^Mozilla.*NEWT" bad_bot
SetEnvIfNoCase User-Agent "^NAMEPROTECT" bad_bot
SetEnvIfNoCase User-Agent "^Navroad" bad_bot
SetEnvIfNoCase User-Agent "^NearSite" bad_bot
SetEnvIfNoCase User-Agent "^NetAnts" bad_bot
SetEnvIfNoCase User-Agent "^Netcraft" bad_bot
SetEnvIfNoCase User-Agent "^NetMechanic" bad_bot
SetEnvIfNoCase User-Agent "^NetSpider" bad_bot
SetEnvIfNoCase User-Agent "^Net\ Vampire" bad_bot
SetEnvIfNoCase User-Agent "^NetZIP" bad_bot
SetEnvIfNoCase User-Agent "^NextGenSearchBot" bad_bot
SetEnvIfNoCase User-Agent "^NG" bad_bot
SetEnvIfNoCase User-Agent "^NICErsPRO" bad_bot
SetEnvIfNoCase User-Agent "^niki-bot" bad_bot
SetEnvIfNoCase User-Agent "^NimbleCrawler" bad_bot
SetEnvIfNoCase User-Agent "^Ninja" bad_bot
SetEnvIfNoCase User-Agent "^NPbot" bad_bot
SetEnvIfNoCase User-Agent "^Octopus" bad_bot
SetEnvIfNoCase User-Agent "^Offline\ Explorer" bad_bot
SetEnvIfNoCase User-Agent "^Offline\ Navigator" bad_bot
SetEnvIfNoCase User-Agent "^Openfind" bad_bot
SetEnvIfNoCase User-Agent "^OutfoxBot" bad_bot
SetEnvIfNoCase User-Agent "^PageGrabber" bad_bot
SetEnvIfNoCase User-Agent "^Papa\ Foto" bad_bot
SetEnvIfNoCase User-Agent "^pavuk" bad_bot
SetEnvIfNoCase User-Agent "^pcBrowser" bad_bot
SetEnvIfNoCase User-Agent "^PHP\ version\ tracker" bad_bot
SetEnvIfNoCase User-Agent "^Pockey" bad_bot
SetEnvIfNoCase User-Agent "^ProPowerBot/2.14" bad_bot
SetEnvIfNoCase User-Agent "^ProWebWalker" bad_bot
SetEnvIfNoCase User-Agent "^psbot" bad_bot
SetEnvIfNoCase User-Agent "^Pump" bad_bot
SetEnvIfNoCase User-Agent "^QueryN.Metasearch" bad_bot
SetEnvIfNoCase User-Agent "^RealDownload" bad_bot
SetEnvIfNoCase User-Agent "Reaper" bad_bot
SetEnvIfNoCase User-Agent "Recorder" bad_bot
SetEnvIfNoCase User-Agent "^ReGet" bad_bot
SetEnvIfNoCase User-Agent "^RepoMonkey" bad_bot
SetEnvIfNoCase User-Agent "^RMA" bad_bot
SetEnvIfNoCase User-Agent "Siphon" bad_bot
SetEnvIfNoCase User-Agent "^SiteSnagger" bad_bot
SetEnvIfNoCase User-Agent "^SlySearch" bad_bot
SetEnvIfNoCase User-Agent "^SmartDownload" bad_bot
SetEnvIfNoCase User-Agent "^Snake" bad_bot
SetEnvIfNoCase User-Agent "^Snapbot" bad_bot
SetEnvIfNoCase User-Agent "^Snoopy" bad_bot
SetEnvIfNoCase User-Agent "^sogou" bad_bot
SetEnvIfNoCase User-Agent "^SpaceBison" bad_bot
SetEnvIfNoCase User-Agent "^SpankBot" bad_bot
SetEnvIfNoCase User-Agent "^spanner" bad_bot
SetEnvIfNoCase User-Agent "^Sqworm" bad_bot
SetEnvIfNoCase User-Agent "Stripper" bad_bot
SetEnvIfNoCase User-Agent "Sucker" bad_bot
SetEnvIfNoCase User-Agent "^SuperBot" bad_bot
SetEnvIfNoCase User-Agent "^SuperHTTP" bad_bot
SetEnvIfNoCase User-Agent "^Surfbot" bad_bot
SetEnvIfNoCase User-Agent "^suzuran" bad_bot
SetEnvIfNoCase User-Agent "^Szukacz/1.4" bad_bot
SetEnvIfNoCase User-Agent "^tAkeOut" bad_bot
SetEnvIfNoCase User-Agent "^Teleport" bad_bot
SetEnvIfNoCase User-Agent "^Telesoft" bad_bot
SetEnvIfNoCase User-Agent "^TurnitinBot/1.5" bad_bot
SetEnvIfNoCase User-Agent "^The.Intraformant" bad_bot
SetEnvIfNoCase User-Agent "^TheNomad" bad_bot
SetEnvIfNoCase User-Agent "^TightTwatBot" bad_bot
SetEnvIfNoCase User-Agent "^Titan" bad_bot
SetEnvIfNoCase User-Agent "^True_Robot" bad_bot
SetEnvIfNoCase User-Agent "^turingos" bad_bot
SetEnvIfNoCase User-Agent "^TurnitinBot" bad_bot
SetEnvIfNoCase User-Agent "^URLy.Warning" bad_bot
SetEnvIfNoCase User-Agent "^Vacuum" bad_bot
SetEnvIfNoCase User-Agent "^VCI" bad_bot
SetEnvIfNoCase User-Agent "^VoidEYE" bad_bot
SetEnvIfNoCase User-Agent "^Web\ Image\ Collector" bad_bot
SetEnvIfNoCase User-Agent "^Web\ Sucker" bad_bot
SetEnvIfNoCase User-Agent "^WebAuto" bad_bot
SetEnvIfNoCase User-Agent "^WebBandit" bad_bot
SetEnvIfNoCase User-Agent "^Webclipping.com" bad_bot
SetEnvIfNoCase User-Agent "^WebCopier" bad_bot
SetEnvIfNoCase User-Agent "^WebEMailExtrac.*" bad_bot
SetEnvIfNoCase User-Agent "^WebEnhancer" bad_bot
SetEnvIfNoCase User-Agent "^WebFetch" bad_bot
SetEnvIfNoCase User-Agent "^WebGo\ IS" bad_bot
SetEnvIfNoCase User-Agent "^Web.Image.Collector" bad_bot
SetEnvIfNoCase User-Agent "^WebLeacher" bad_bot
SetEnvIfNoCase User-Agent "^WebmasterWorldForumBot" bad_bot
SetEnvIfNoCase User-Agent "^WebReaper" bad_bot
SetEnvIfNoCase User-Agent "^WebSauger" bad_bot
SetEnvIfNoCase User-Agent "^Website\ eXtractor" bad_bot
SetEnvIfNoCase User-Agent "^Website\ Quester" bad_bot
SetEnvIfNoCase User-Agent "^Webster" bad_bot
SetEnvIfNoCase User-Agent "^WebStripper" bad_bot
SetEnvIfNoCase User-Agent "^WebWhacker" bad_bot
SetEnvIfNoCase User-Agent "^WebZIP" bad_bot
SetEnvIfNoCase User-Agent "Whacker" bad_bot
SetEnvIfNoCase User-Agent "^Widow" bad_bot
SetEnvIfNoCase User-Agent "^WISENutbot" bad_bot
SetEnvIfNoCase User-Agent "^WWWOFFLE" bad_bot
SetEnvIfNoCase User-Agent "^WWW-Collector-E" bad_bot
SetEnvIfNoCase User-Agent "^Xaldon" bad_bot
SetEnvIfNoCase User-Agent "^Xenu" bad_bot
SetEnvIfNoCase User-Agent "^Zeus" bad_bot
SetEnvIfNoCase User-Agent "ZmEu" bad_bot
SetEnvIfNoCase User-Agent "^Zyborg" bad_bot
# Vulnerability Scanners
SetEnvIfNoCase User-Agent "Acunetix" bad_bot
SetEnvIfNoCase User-Agent "FHscan" bad_bot
# Aggressive Chinese Search Engine
SetEnvIfNoCase User-Agent "Baiduspider" bad_bot
# Aggressive Russian Search Engine
SetEnvIfNoCase User-Agent "Yandex" bad_bot
<Limit GET POST HEAD>
Order Allow,Deny
Allow from all
# Cyveillance
deny from 38.100.19.8/29
deny from 38.100.21.0/24
deny from 38.100.41.64/26
deny from 38.105.71.0/25
deny from 38.105.83.0/27
deny from 38.112.21.140/30
deny from 38.118.42.32/29
deny from 65.213.208.128/27
deny from 65.222.176.96/27
deny from 65.222.185.72/29
Deny from env=bad_bot
</Limit>
تالار گفتگوی تخصصی متا    دفتر تلفن مشاغل   سامانه پیام کوتاه ویژه اعضای دفتر تلفن مشاغل     شبکه تجارت الکترونیک البرز

mehrtadbir

And this one in robots.txt


User-Agent: W3C-checklink
Disallow: /

User-Agent: W3C_Validator
Disallow: /

User-Agent: W3C_CSS_Validator
Disallow: /

User-agent: Yandex
Disallow: /

User-agent: Baiduspider
Disallow: /

User-agent: archive.org_bot
Disallow: /

User-agent: ia_archiver
Disallow: /

User-agent: FavIconizer
Disallow: /

User-agent: AhrefsBot
Disallow: /

User-agent: A6-Indexer/1.0
Disallow: /

User-agent: butterfly
Disallow: /

User-agent: Exabot
Disallow: /

User-agent: grapeshot
Disallow: /

User-agent: Mail.Ru
Disallow: /

User-agent: nutch
Disallow: /

User-agent: pingdom
Disallow: /

User-agent: proximic
Disallow: /

User-agent: Python
Disallow: /

User-agent: SearchmetricsBot
Disallow: /

User-agent: SeznamBot
Disallow: /

User-agent: Sogou web spider
Disallow: /

User-agent: sogou spider
Disallow: /

User-agent: Sogou
Disallow: /

User-agent: YoudaoBot
Disallow: /

User-agent: Sosospider
Disallow: /

User-agent: spbot
Disallow: /

User-agent: Wget
Disallow: /

User-agent: yie8
Disallow: /

User-agent: MJ12bot
Disallow: /

User-agent: 360Spider
Disallow: /

User-agent: TurnitinBot
Disallow: /

User-agent: BLEXBot
Disallow: /

User-agent: hivaBot
Disallow: /

User-agent: NerdyBot
Disallow: /

User-agent: updown_tester
Disallow: /

User-agent: AndroidDownloadManager
Disallow: /

User-agent: voltron
Disallow: /

User-agent: Xenu's Link Sleuth 1.1c
Disallow: /

User-agent: SemrushBot
Disallow: /

User-agent: SemrushBot-SA
Disallow: /

User-agent: oBot
Disallow: /

User-agent: Riddler
Disallow: /

User-agent: Mail.RU_Bot
Disallow: /

User-agent: Mail.Ru
Disallow: /


Please give me advice
Thank you
تالار گفتگوی تخصصی متا    دفتر تلفن مشاغل   سامانه پیام کوتاه ویژه اعضای دفتر تلفن مشاغل     شبکه تجارت الکترونیک البرز

Kindred

In short.... You can't actually stop them
Слaва
Украинi

Please do not PM, IM or Email me with support questions.  You will get better and faster responses in the support boards.  Thank you.

"Loki is not evil, although he is certainly not a force for good. Loki is... complicated."

mehrtadbir

Thank you for your answer

Would you advise me to do, (however small), That I can do better.

One recommendation
Or link
Or anything else that I can do better.

Thanks.
تالار گفتگوی تخصصی متا    دفتر تلفن مشاغل   سامانه پیام کوتاه ویژه اعضای دفتر تلفن مشاغل     شبکه تجارت الکترونیک البرز

Sam308

A PHP solution by Sam308

Block Unwanted Robots/Spiders visitors

Instructions:
Place the following PHP Code in the beginning of your index.php file.

<?php
// ---------------------------------------------------------------------------------------------------------------

// Banned IP Addresses and Bots - Redirects banned visitors who make it past the .htaccess and or robots.txt files to an URL.
// The $banned_ip_addresses array can contain both full and partial IP addresses, i.e. Full = 123.456.789.101, Partial = 123.456.789. or 123.456. or 123.
// Use partial IP addresses to include all IP addresses that begin with a partial IP addresses. The partial IP addresses must end with a period.
// The $banned_bots, $banned_unknown_bots, and $good_bots arrays should contain keyword strings found within the User Agent string.
// The $banned_unknown_bots array is used to identify unknown robots (identified by 'bot' followed by a space or one of the following characters _+:,.;/\-).
// The $good_bots array contains keyword strings used as exemptions when checking for $banned_unknown_bots. If you do not want to utilize the $good_bots array such as
// $good_bots = array(), then you must remove the the keywords strings 'bot.','bot/','bot-' from the $banned_unknown_bots array or else the good bots will also be banned.
   
$banned_ip_addresses = array('41.','64.79.100.23','5.254.97.75','148.251.236.167','88.180.102.124','62.210.172.77','45.','195.206.253.146');
   
$banned_bots = array('.ru','AhrefsBot','crawl','crawler','DotBot','linkdex','majestic','meanpath','PageAnalyzer','robot','rogerbot','semalt','SeznamBot','spider');
   
$banned_unknown_bots = array('bot ','bot_','bot+','bot:','bot,','bot;','bot\\','bot.','bot/','bot-');
   
$good_bots = array('Google','MSN','bing','Slurp','Yahoo');
   
$banned_ip_address_url 'http://english-1329329990.spampoison.com';

// Visitor's IP address and Browser (User Agent)
   
$ip_address $_SERVER['REMOTE_ADDR'];
   
$browser $_SERVER['HTTP_USER_AGENT'];

// Declared Temporary Variables
   
$ipfound $piece $banned_piece $botfound $gbotfound $ubotfound '';

// Checks for Banned IP Addresses and Bots
   
if($banned_ip_address_url != ''){
     
// Checks for Banned IP Address
        
if(!empty($banned_ip_addresses)){
          if(
in_array($ip_address$banned_ip_addresses)){$ipfound 'found';}
          if(
$ipfound != 'found'){
            
$ip_pieces explode('.'$ip_address);
            foreach (
$ip_pieces as $value){
              
$piece $piece.$value.'.';
              if(
in_array($piece$banned_ip_addresses)){$banned_piece 'found'; break;}
            }
          }
          if(
$banned_piece == 'found'){header("location: $banned_ip_address_url"); exit();}
        }

     
// Checks for Banned Bots
        
if(!empty($banned_bots)){
          foreach (
$banned_bots as $bbvalue){
            
$pos1 stripos($browser$bbvalue);
            if(
$pos1 !== false){$botfound 'found'; break;}
          }
          if(
$botfound == 'found'){header("location: $banned_ip_address_url"); exit();}
        }

     
// Checks for Banned Unknown Bots
        
if(!empty($good_bots)){
          foreach (
$good_bots as $gbvalue){
            
$pos2 stripos($browser$gbvalue);
            if(
$pos2 !== false){$gbotfound 'found'; break;}
          }
        }
        if(
$gbotfound != 'found'){
          if(!empty(
$banned_unknown_bots)){
            foreach (
$banned_unknown_bots as $bubvalue){
              
$pos3 stripos($browser$bubvalue);
              if(
$pos3 !== false){$ubotfound 'found'; break;}
            }
            if(
$ubotfound == 'found'){header("location: $banned_ip_address_url"); exit();}
          }
        }
   }

// ---------------------------------------------------------------------------------------------------------------
?>

Sam308

Minor Correction to A PHP solution by Sam308
Replace in the above php code with the following:


// Declared Temporary Variables
   $ipfound = $piece = $botfound = $gbotfound = $ubotfound = '';

// Checks for Banned IP Addresses and Bots
   if($banned_ip_address_url != ''){
     // Checks for Banned IP Address
        if(!empty($banned_ip_addresses)){
          if(in_array($ip_address, $banned_ip_addresses)){$ipfound = 'found';}
          if($ipfound != 'found'){
            $ip_pieces = explode('.', $ip_address);
            foreach ($ip_pieces as $value){
              $piece = $piece.$value.'.';
              if(in_array($piece, $banned_ip_addresses)){$ipfound = 'found'; break;}
            }
          }
          if($ipfound == 'found'){header("location: $banned_ip_address_url"); exit();}
        }

Advertisement: