Author Topic: How to stop spider from crawling my site (Read 6709 times)

patricia2019 · « **on:** August 22, 2010, 11:42:05 AM »

Idon't mind google and yahoo spiders.
Please tell me how I could stop a specific spider from crawling my site.
Thanks

Oranos · « **Reply #1 on:** August 22, 2010, 11:58:00 AM »

google up a file called robots.txt

Masterd · « **Reply #2 on:** August 22, 2010, 12:24:26 PM »

And upload it to your root directory.

Kill Em All · « **Reply #3 on:** August 22, 2010, 12:40:34 PM »

Also search the forums on robots.txt files, you can find a lot of examples of them around here.

Masterd · « **Reply #4 on:** August 22, 2010, 12:45:49 PM »

Quote from: Kill Em All on August 22, 2010, 12:40:34 PM

Also search the forums on robots.txt files, you can find a lot of examples of them around here.

Or some popular sites.

Kill Em All · « **Reply #5 on:** August 22, 2010, 12:49:38 PM »

Well depending on the popular site.. different software packages maybe have different looking robots.txt.

example from smf:

Code: [Select]

User-agent: *
Disallow: /forum/index.php?action=activate*
Disallow: /forum/index.php?action=admin*

example from microsoft

Code: [Select]

User-agent: *
Disallow: /*TOCLinksForCrawlers*
Disallow: /*/mac/help.mspx

So sort of different styles.

patricia2019 · « **Reply #6 on:** August 29, 2010, 10:26:58 AM »

Thank you all for your answers, I would have to post a job to get this done.
Is it a BAD thing when these spiders crawl your site. I did had a visit from Google, yahoo and Alexa as well? Could they harm my site at all?

YogiBear · « **Reply #7 on:** August 29, 2010, 10:43:42 AM »

The three you mention are your best friends if you want folk to find you ie. your forum isn't restricted to family and friends.

Baidu can be a pain - it doesn't do any harm but crawls you relentlessly meaning it can eat up your bandwidth if you're running short on your monthly allocation.

On the 'Who's On Line' you'll sometimes see they are registering of logging-in but all that means is they're reading those pages.

Kill Em All · « **Reply #8 on:** August 29, 2010, 05:49:14 PM »

I like spiders.... just not the creepy ones in the place some call the outside.

IceXaos · « **Reply #9 on:** August 29, 2010, 05:52:51 PM »

Lol, usin' spiders to find what you need to stop spiders.

manis99 · « **Reply #10 on:** August 29, 2010, 06:02:28 PM »

usually spiders dont do any harm. Then why u want to stop them.

Kill Em All · « **Reply #11 on:** August 29, 2010, 06:03:31 PM »

My guess, as YogiBear has said, its because of limited bandwidth set by host.

manis99 · « **Reply #12 on:** August 29, 2010, 06:23:47 PM »

lol may be bt i dont need to worry at all cuz i hv unlimited bandwidth.

Kill Em All · « **Reply #13 on:** August 29, 2010, 06:31:41 PM »

Quote from: manis99 on August 29, 2010, 06:23:47 PM

lol may be bt i dont need to worry at all cuz i hv unlimited bandwidth.

No such thing...

androidhelpersdotcom · « **Reply #14 on:** August 29, 2010, 07:59:05 PM »

I also have unlimited bandwidth, so it does exist =D Unlimited storage as well.

busterone · « **Reply #15 on:** August 29, 2010, 08:07:03 PM »

No, sorry, it does not exist. It is a marketing lie told by overselling hosts to get your business. These hosts promise it, but if you carefully read the fine print, it doesn't really exist, and they will suspend your account for ridiculous reasons. See this topic from the hosting boards -
http://www.simplemachines.org/community/index.php?topic=117303.0

IceXaos · « **Reply #16 on:** August 29, 2010, 08:07:43 PM »

Quote from: androidhelpersdotcom on August 29, 2010, 07:59:05 PM

I also have unlimited bandwidth, so it does exist =D Unlimited storage as well.

Your host may tell you it's unlimited, but technically it's not and cannot be.

Kill Em All · « **Reply #17 on:** August 29, 2010, 11:33:30 PM »

Oh, well my host claims to have unlimited everything, so ha! *rolls eyes*

Read your webhost's TOS, or ask them what exactly is unlimited, if they are at least decent, they will tell you what it really is. If they say unlimited, then they are just talking out of their asses.

patricia2019 · « **Reply #18 on:** August 31, 2010, 06:16:20 PM »

Thanks you guys

butchs · « **Reply #19 on:** August 31, 2010, 09:15:24 PM »

Bad Behavior mod confirms that the main spiders are correct and does block many bad spiders.

News:

Author Topic: How to stop spider from crawling my site (Read 6709 times)