How to Block Cuil twiceler Spider ?

[philcross]

I have had this Cuil Spider sat on my forum for a few days now.

Ive read that it can be very resource hungry and wish to stop it visiting the forum.

Is there a way to do so please?

I know I can add to robots.txt but I read that it ignores robots.txt for at least the first 14 days.

Ideas anyone?

Thanks

[tyty1234]

I think despite the day thing, you'll still need to do the robots.txt method.

[mtindor]

If you have root access to the machine your site is on and you have a firewall in place, add the IP address(es) of the spider to your firewall.    My site often gets poked by Twiceler - and it's always from the same IP, 67.218.116.166.   On your server, they may be hitting it with a different address.   Just firewall all the IPs that twiceler is hitting your server with.

If you don't require immediate and brute force relief, or if you don't have root access, block the IP address via your .htaccess file or block the spider itself based upon the User Agent that is being sent.

To block an IP (using your .htaccess file):

Code Select Expand

order allow,deny
deny from 67.218.116.166
allow from all

You can block more than one IP address by adding multiple "deny from" lines.


If you want to block the bot based upon the User Agent it reports, you can block anything whose User Agent begins with "Twiceler" by adding this to your .htaccess file:

Code Select Expand

   RewriteEngine on
   RewriteCond %{HTTP_USER_AGENT} ^Twiceler
   RewriteRule ^.* - [F,L]

You can block more than one bot if you know its User Agent by just adding multiple RewriteCond lines with [OR] after all but the last one.   Example:

Code Select Expand

   RewriteEngine on
   RewriteCond %{HTTP_USER_AGENT} ^Twiceler
   RewriteCond %{HTTP_USER_AGENT} ^Googlebot
   RewriteCond %{HTTP_USER_AGENT} ^Yahoo
   RewriteRule ^.* - [F,L]


The above examples assume the server you are hosting on has mod_rewrite installed (and it probably does).

The .htaccess file that you create should go either in the root of your website (i.e. your public_html folder as an example) or in the top directory of your forum - (ex:  if your forum is in /forum, then put a .htaccess file in the /forum directory with the appropriate info from above.

Mike

[philcross]

Thank You both for your help !!

[pjn]

thanx!
i need it too...

[pjn]

but

it's not work ): :
http://pjn-il.com/index.php?action=who;sort=time

i do this:
1. made a-new file:
.htaccess

2.write this:
"
RewriteEngine on
RewriteCond %{HTTP_USER_AGENT} ^Twiceler
RewriteRule ^.* - [F,L]
"

3. upload to :
ftp://pjn-il.com/domains/pjn-il.com/public_html/




i do somesing wrong?

[YogiBear]

Take a look at this thread...


http://www.simplemachines.org/community/index.php?topic=392796.0