News:

Bored?  Looking to kill some time?  Want to chat with other SMF users?  Join us in IRC chat or Discord

Main Menu

Stop Twiceler from seeing forum

Started by casp3r, May 06, 2008, 05:24:10 PM

Previous topic - Next topic

casp3r

Is there any way to prevent the Twiceler spider from idexing my forum? I've banned all the IPs listed here
http://cuill.com/twiceler/robot.html
but it stills shows in the logs.
Aviation forum & gallery
www.niaviation.co.uk

karlbenson

#1
Add to your robots.txt

User-agent: Twiceler
Disallow: /

Note:
It should appear BEFORE any section with the * for useragent. And with a blank line between them.
like

User-agent: Twiceler
Disallow: /

User-agent: *
Disallow:

casp3r

Done. Will post back in a few days and let you know how it goes.

Many thanks
Aviation forum & gallery
www.niaviation.co.uk

Advertisement: