Advertisement:

Author Topic: Stop Twiceler from seeing forum  (Read 2228 times)

Offline casp3r

  • Jr. Member
  • **
  • Posts: 187
    • Aviation forum & gallery
Stop Twiceler from seeing forum
« on: May 06, 2008, 05:24:10 PM »
Is there any way to prevent the Twiceler spider from idexing my forum? I've banned all the IPs listed here
http://cuill.com/twiceler/robot.html
but it stills shows in the logs.
Aviation forum & gallery
www.niaviation.co.uk

Offline karlbenson

  • SMF Friend
  • SMF Super Hero
  • *
  • Posts: 15,629
  • Gender: Male
    • Criminal Brief UK
Re: Stop Twiceler from seeing forum
« Reply #1 on: May 06, 2008, 05:30:20 PM »
Add to your robots.txt

User-agent: Twiceler
Disallow: /

Note:
It should appear BEFORE any section with the * for useragent. And with a blank line between them.
like

User-agent: Twiceler
Disallow: /

User-agent: *
Disallow:
« Last Edit: May 06, 2008, 05:32:51 PM by karlbenson »

Offline casp3r

  • Jr. Member
  • **
  • Posts: 187
    • Aviation forum & gallery
Re: Stop Twiceler from seeing forum
« Reply #2 on: May 06, 2008, 05:32:51 PM »
Done. Will post back in a few days and let you know how it goes.

Many thanks
Aviation forum & gallery
www.niaviation.co.uk