Is there any way to prevent the Twiceler spider from idexing my forum? I've banned all the IPs listed here
http://cuill.com/twiceler/robot.html
but it stills shows in the logs.
Add to your robots.txt
User-agent: Twiceler
Disallow: /
Note:
It should appear BEFORE any section with the * for useragent. And with a blank line between them.
like
User-agent: Twiceler
Disallow: /
User-agent: *
Disallow:
Done. Will post back in a few days and let you know how it goes.
Many thanks