News:

Want to get involved in developing SMF, then why not lend a hand on our github!

Main Menu

.htaccess / robots.txt

Started by pjn, July 27, 2010, 04:41:21 AM

Previous topic - Next topic

pjn

i want to block "Twiceler" but i whant get to googlebot to spy on web

i do this:
in file: robots.txt :
QuoteUser-agent: *
Allow: /

in file: .htaccess :
QuoteRewriteEngine on
RewriteCond %{HTTP_USER_AGENT} ^cuill.com
RewriteRule ^.* - [F,L]


but is still spy on my web...


what i need to do ????????

thanx
...
Quick Portal - Free hosting & portal & forums system

Many designs + many extras + server = FREE!

Ask from us your free web now:
http://qp-free.com/index.php/board,2.0.html

chilly

Quote from: pjn on July 27, 2010, 04:41:21 AM

in file: robots.txt :
QuoteUser-agent: *
Allow: /

I'm not used to .htaccess handling but I know that you allow every user agent in your robots.txt
so why should that bot stop to crawl your website

Language Coordinator

pjn

did you can give me a-code to stop that spy from robots?...
Quick Portal - Free hosting & portal & forums system

Many designs + many extras + server = FREE!

Ask from us your free web now:
http://qp-free.com/index.php/board,2.0.html

busterone

User-agent: Twiceler
Disallow: /

pjn

OK

delete this lines??? :
User-agent: *
Allow: /


and delete this file:
.htaccess

?

or what i need to do?
Quick Portal - Free hosting & portal & forums system

Many designs + many extras + server = FREE!

Ask from us your free web now:
http://qp-free.com/index.php/board,2.0.html

pjn

Quick Portal - Free hosting & portal & forums system

Many designs + many extras + server = FREE!

Ask from us your free web now:
http://qp-free.com/index.php/board,2.0.html

YogiBear

The problem is, over time, Twiceler doesn't keep to any specific IP range but take a look at this...


http://www.cuil.com/info/webmaster_info/


...apparently, you can contact them and ask them not to crawl your pages.
SMF v2.1.3  Mods : Snow & Garland v1.4,  PHP  v.7.4.33

pjn

Quick Portal - Free hosting & portal & forums system

Many designs + many extras + server = FREE!

Ask from us your free web now:
http://qp-free.com/index.php/board,2.0.html

Adish - (F.L.A.M.E.R)

Any updates over this? Do you still require any assistance?

Advertisement: