Advertisement:

Author Topic: How to stop spider from crawling my site  (Read 6580 times)

Offline patricia2019

  • Semi-Newbie
  • *
  • Posts: 19
How to stop spider from crawling my site
« on: August 22, 2010, 11:42:05 AM »
Idon't mind google and yahoo spiders.
Please tell me how I could stop a specific spider from crawling my site.
Thanks


Offline Oranos

  • Jr. Member
  • **
  • Posts: 276
  • Gender: Male
Re: How to stop spider from crawling my site
« Reply #1 on: August 22, 2010, 11:58:00 AM »
google up a file called robots.txt

Offline Masterd

  • SMF Hero
  • ******
  • Posts: 3,919
  • Gender: Male
Re: How to stop spider from crawling my site
« Reply #2 on: August 22, 2010, 12:24:26 PM »
And upload it to your root directory.

Offline Kill Em All

  • SMF Friend
  • SMF Hero
  • *
  • Posts: 8,144
  • Gender: Male
  • Something has to go wrong. Just has to.
    • KEAGaming
Re: How to stop spider from crawling my site
« Reply #3 on: August 22, 2010, 12:40:34 PM »
Also search the forums on robots.txt files, you can find a lot of examples of them around here.
My Site: KEAGaming.com

Manual Installation of Mods
Prevent Spam and Forum Attacks
Please do not PM or email me for support unless offered, help should be publicly displayed to others.

Offline Masterd

  • SMF Hero
  • ******
  • Posts: 3,919
  • Gender: Male
Re: How to stop spider from crawling my site
« Reply #4 on: August 22, 2010, 12:45:49 PM »
Also search the forums on robots.txt files, you can find a lot of examples of them around here.


Or some popular sites.

Offline Kill Em All

  • SMF Friend
  • SMF Hero
  • *
  • Posts: 8,144
  • Gender: Male
  • Something has to go wrong. Just has to.
    • KEAGaming
Re: How to stop spider from crawling my site
« Reply #5 on: August 22, 2010, 12:49:38 PM »
Well depending on the popular site.. different software packages maybe have different looking robots.txt.

example from smf:
Code: [Select]
User-agent: *
Disallow: /forum/index.php?action=activate*
Disallow: /forum/index.php?action=admin*

example from microsoft
Code: [Select]
User-agent: *
Disallow: /*TOCLinksForCrawlers*
Disallow: /*/mac/help.mspx

So sort of different styles.
My Site: KEAGaming.com

Manual Installation of Mods
Prevent Spam and Forum Attacks
Please do not PM or email me for support unless offered, help should be publicly displayed to others.

Offline patricia2019

  • Semi-Newbie
  • *
  • Posts: 19
Re: How to stop spider from crawling my site
« Reply #6 on: August 29, 2010, 10:26:58 AM »
Thank you all for your answers, I would have to post a job to get this done.
Is it a BAD thing when these spiders crawl your site.  I did had a visit from Google, yahoo and Alexa as well?  Could they harm my site at all?

Offline YogiBear

  • SMF Hero
  • ******
  • Posts: 1,936
Re: How to stop spider from crawling my site
« Reply #7 on: August 29, 2010, 10:43:42 AM »
The three you mention are your best friends if you want folk to find you ie. your forum isn't restricted to family and friends.

Baidu can be a pain - it doesn't do any harm but crawls you relentlessly meaning it can eat up your bandwidth if you're running short on your monthly allocation.

On the 'Who's On Line' you'll sometimes see they are registering of logging-in but all that means is they're reading those pages.
SMF v2.0.15  Mods : Ohara YouTube Embed v1.2.9,  Custom Tab v1.2 , Bakers' Dozen Pages v1.2 , SMFPacks Shoutbox v1.1,  Board Color & Icons v1.1 , Simple Image Upload , Snow & Garland v1.4

Offline Kill Em All

  • SMF Friend
  • SMF Hero
  • *
  • Posts: 8,144
  • Gender: Male
  • Something has to go wrong. Just has to.
    • KEAGaming
Re: How to stop spider from crawling my site
« Reply #8 on: August 29, 2010, 05:49:14 PM »
I like spiders.... just not the creepy ones in the place some call the outside.
My Site: KEAGaming.com

Manual Installation of Mods
Prevent Spam and Forum Attacks
Please do not PM or email me for support unless offered, help should be publicly displayed to others.

Offline IceXaos

  • Full Member
  • ***
  • Posts: 673
    • Runedev
Re: How to stop spider from crawling my site
« Reply #9 on: August 29, 2010, 05:52:51 PM »
Lol, usin' spiders to find what you need to stop spiders.  ???

Offline manis99

  • Semi-Newbie
  • *
  • Posts: 69
  • Gender: Male
    • myemobiles
Re: How to stop spider from crawling my site
« Reply #10 on: August 29, 2010, 06:02:28 PM »
usually spiders dont do any harm. Then why u want to stop them.

Offline Kill Em All

  • SMF Friend
  • SMF Hero
  • *
  • Posts: 8,144
  • Gender: Male
  • Something has to go wrong. Just has to.
    • KEAGaming
Re: How to stop spider from crawling my site
« Reply #11 on: August 29, 2010, 06:03:31 PM »
My guess, as YogiBear has said, its because of limited bandwidth set by host.
My Site: KEAGaming.com

Manual Installation of Mods
Prevent Spam and Forum Attacks
Please do not PM or email me for support unless offered, help should be publicly displayed to others.

Offline manis99

  • Semi-Newbie
  • *
  • Posts: 69
  • Gender: Male
    • myemobiles
Re: How to stop spider from crawling my site
« Reply #12 on: August 29, 2010, 06:23:47 PM »
lol may be bt i dont need to worry at all cuz i hv unlimited bandwidth. :)

Offline Kill Em All

  • SMF Friend
  • SMF Hero
  • *
  • Posts: 8,144
  • Gender: Male
  • Something has to go wrong. Just has to.
    • KEAGaming
Re: How to stop spider from crawling my site
« Reply #13 on: August 29, 2010, 06:31:41 PM »
lol may be bt i dont need to worry at all cuz i hv unlimited bandwidth. :)
No such thing...
My Site: KEAGaming.com

Manual Installation of Mods
Prevent Spam and Forum Attacks
Please do not PM or email me for support unless offered, help should be publicly displayed to others.

Offline androidhelpersdotcom

  • Semi-Newbie
  • *
  • Posts: 34
  • Gender: Male
    • Android Helpers
Re: How to stop spider from crawling my site
« Reply #14 on: August 29, 2010, 07:59:05 PM »
I also have unlimited bandwidth, so it does exist =D Unlimited storage as well.
Using 2.0 RC3, Glassy Curve Theme

Offline busterone

  • SMF Hero
  • ******
  • Posts: 2,150
  • Gender: Male
  • Devil Dog
    • The Demon's Den
Re: How to stop spider from crawling my site
« Reply #15 on: August 29, 2010, 08:07:03 PM »
No, sorry, it does not exist. It is a marketing lie told by overselling hosts to get your business. These hosts promise it, but if you carefully read the fine print, it doesn't really exist, and they will suspend your account for ridiculous reasons. See this topic from the hosting boards -
http://www.simplemachines.org/community/index.php?topic=117303.0

Offline IceXaos

  • Full Member
  • ***
  • Posts: 673
    • Runedev
Re: How to stop spider from crawling my site
« Reply #16 on: August 29, 2010, 08:07:43 PM »
I also have unlimited bandwidth, so it does exist =D Unlimited storage as well.
Your host may tell you it's unlimited, but technically it's not and cannot be.

Offline Kill Em All

  • SMF Friend
  • SMF Hero
  • *
  • Posts: 8,144
  • Gender: Male
  • Something has to go wrong. Just has to.
    • KEAGaming
Re: How to stop spider from crawling my site
« Reply #17 on: August 29, 2010, 11:33:30 PM »
Oh, well my host claims to have unlimited everything, so ha! *rolls eyes*

Read your webhost's TOS, or ask them what exactly is unlimited, if they are at least decent, they will tell you what it really is. If they say unlimited, then they are just talking out of their asses.
My Site: KEAGaming.com

Manual Installation of Mods
Prevent Spam and Forum Attacks
Please do not PM or email me for support unless offered, help should be publicly displayed to others.

Offline patricia2019

  • Semi-Newbie
  • *
  • Posts: 19
Re: How to stop spider from crawling my site
« Reply #18 on: August 31, 2010, 06:16:20 PM »
Thanks you guys  ;)

Offline butchs

  • SMF Hero
  • ******
  • Posts: 1,728
  • Lost 7GB bandwidth!
    • EastCoastRollingThunder
Re: How to stop spider from crawling my site
« Reply #19 on: August 31, 2010, 09:15:24 PM »
Bad Behavior mod confirms that the main spiders are correct and does block many bad spiders.
I have been truly inspired by the SUGGESTIONS as I sit on my throne and contemplate the wisdom imposed upon me.