Simple Machines Community Forum

SMF Support => SMF 2.0.x Support => Topic started by: patricia2019 on August 22, 2010, 11:42:05 AM

Title: How to stop spider from crawling my site
Post by: patricia2019 on August 22, 2010, 11:42:05 AM
Idon't mind google and yahoo spiders.
Please tell me how I could stop a specific spider from crawling my site.
Thanks

Title: Re: How to stop spider from crawling my site
Post by: Oranos on August 22, 2010, 11:58:00 AM
google up a file called robots.txt
Title: Re: How to stop spider from crawling my site
Post by: Masterd on August 22, 2010, 12:24:26 PM
And upload it to your root directory.
Title: Re: How to stop spider from crawling my site
Post by: Kill Em All on August 22, 2010, 12:40:34 PM
Also search the forums on robots.txt files, you can find a lot of examples of them around here.
Title: Re: How to stop spider from crawling my site
Post by: Masterd on August 22, 2010, 12:45:49 PM
Also search the forums on robots.txt files, you can find a lot of examples of them around here.


Or some popular sites.
Title: Re: How to stop spider from crawling my site
Post by: Kill Em All on August 22, 2010, 12:49:38 PM
Well depending on the popular site.. different software packages maybe have different looking robots.txt.

example from smf:
Code: [Select]
User-agent: *
Disallow: /forum/index.php?action=activate*
Disallow: /forum/index.php?action=admin*

example from microsoft
Code: [Select]
User-agent: *
Disallow: /*TOCLinksForCrawlers*
Disallow: /*/mac/help.mspx

So sort of different styles.
Title: Re: How to stop spider from crawling my site
Post by: patricia2019 on August 29, 2010, 10:26:58 AM
Thank you all for your answers, I would have to post a job to get this done.
Is it a BAD thing when these spiders crawl your site.  I did had a visit from Google, yahoo and Alexa as well?  Could they harm my site at all?
Title: Re: How to stop spider from crawling my site
Post by: YogiBear on August 29, 2010, 10:43:42 AM
The three you mention are your best friends if you want folk to find you ie. your forum isn't restricted to family and friends.

Baidu can be a pain - it doesn't do any harm but crawls you relentlessly meaning it can eat up your bandwidth if you're running short on your monthly allocation.

On the 'Who's On Line' you'll sometimes see they are registering of logging-in but all that means is they're reading those pages.
Title: Re: How to stop spider from crawling my site
Post by: Kill Em All on August 29, 2010, 05:49:14 PM
I like spiders.... just not the creepy ones in the place some call the outside.
Title: Re: How to stop spider from crawling my site
Post by: IceXaos on August 29, 2010, 05:52:51 PM
Lol, usin' spiders to find what you need to stop spiders.  ???
Title: Re: How to stop spider from crawling my site
Post by: manis99 on August 29, 2010, 06:02:28 PM
usually spiders dont do any harm. Then why u want to stop them.
Title: Re: How to stop spider from crawling my site
Post by: Kill Em All on August 29, 2010, 06:03:31 PM
My guess, as YogiBear has said, its because of limited bandwidth set by host.
Title: Re: How to stop spider from crawling my site
Post by: manis99 on August 29, 2010, 06:23:47 PM
lol may be bt i dont need to worry at all cuz i hv unlimited bandwidth. :)
Title: Re: How to stop spider from crawling my site
Post by: Kill Em All on August 29, 2010, 06:31:41 PM
lol may be bt i dont need to worry at all cuz i hv unlimited bandwidth. :)
No such thing...
Title: Re: How to stop spider from crawling my site
Post by: androidhelpersdotcom on August 29, 2010, 07:59:05 PM
I also have unlimited bandwidth, so it does exist =D Unlimited storage as well.
Title: Re: How to stop spider from crawling my site
Post by: busterone on August 29, 2010, 08:07:03 PM
No, sorry, it does not exist. It is a marketing lie told by overselling hosts to get your business. These hosts promise it, but if you carefully read the fine print, it doesn't really exist, and they will suspend your account for ridiculous reasons. See this topic from the hosting boards -
http://www.simplemachines.org/community/index.php?topic=117303.0 (http://www.simplemachines.org/community/index.php?topic=117303.0)
Title: Re: How to stop spider from crawling my site
Post by: IceXaos on August 29, 2010, 08:07:43 PM
I also have unlimited bandwidth, so it does exist =D Unlimited storage as well.
Your host may tell you it's unlimited, but technically it's not and cannot be.
Title: Re: How to stop spider from crawling my site
Post by: Kill Em All on August 29, 2010, 11:33:30 PM
Oh, well my host claims to have unlimited everything, so ha! *rolls eyes*

Read your webhost's TOS, or ask them what exactly is unlimited, if they are at least decent, they will tell you what it really is. If they say unlimited, then they are just talking out of their asses.
Title: Re: How to stop spider from crawling my site
Post by: patricia2019 on August 31, 2010, 06:16:20 PM
Thanks you guys  ;)
Title: Re: How to stop spider from crawling my site
Post by: butchs on August 31, 2010, 09:15:24 PM
Bad Behavior mod confirms that the main spiders are correct and does block many bad spiders.