News:

Wondering if this will always be free?  See why free is better.

Main Menu

Stop Twiceler from seeing forum

Started by casp3r, May 06, 2008, 05:24:10 PM

Previous topic - Next topic

casp3r

Is there any way to prevent the Twiceler spider from idexing my forum? I've banned all the IPs listed here
http://cuill.com/twiceler/robot.html
but it stills shows in the logs.
Aviation forum & gallery
www.niaviation.co.uk

karlbenson

#1
Add to your robots.txt

User-agent: Twiceler
Disallow: /

Note:
It should appear BEFORE any section with the * for useragent. And with a blank line between them.
like

User-agent: Twiceler
Disallow: /

User-agent: *
Disallow:

casp3r

Done. Will post back in a few days and let you know how it goes.

Many thanks
Aviation forum & gallery
www.niaviation.co.uk

Advertisement: