Idon't mind google and yahoo spiders.
Please tell me how I could stop a specific spider from crawling my site.
Thanks
google up a file called robots.txt
And upload it to your root directory.
Also search the forums on robots.txt files, you can find a lot of examples of them around here.
Quote from: Kill Em All on August 22, 2010, 12:40:34 PM
Also search the forums on robots.txt files, you can find a lot of examples of them around here.
Or some popular sites.
Well depending on the popular site.. different software packages maybe have different looking robots.txt.
example from smf:
User-agent: *
Disallow: /forum/index.php?action=activate*
Disallow: /forum/index.php?action=admin*
example from microsoft
User-agent: *
Disallow: /*TOCLinksForCrawlers*
Disallow: /*/mac/help.mspx
So sort of different styles.
Thank you all for your answers, I would have to post a job to get this done.
Is it a BAD thing when these spiders crawl your site. I did had a visit from Google, yahoo and Alexa as well? Could they harm my site at all?
The three you mention are your best friends if you want folk to find you ie. your forum isn't restricted to family and friends.
Baidu can be a pain - it doesn't do any harm but crawls you relentlessly meaning it can eat up your bandwidth if you're running short on your monthly allocation.
On the 'Who's On Line' you'll sometimes see they are registering of logging-in but all that means is they're reading those pages.
I like spiders.... just not the creepy ones in the place some call the outside.
Lol, usin' spiders to find what you need to stop spiders. ???
usually spiders dont do any harm. Then why u want to stop them.
My guess, as YogiBear has said, its because of limited bandwidth set by host.
lol may be bt i dont need to worry at all cuz i hv unlimited bandwidth. :)
Quote from: manis99 on August 29, 2010, 06:23:47 PM
lol may be bt i dont need to worry at all cuz i hv unlimited bandwidth. :)
No such thing...
I also have unlimited bandwidth, so it does exist =D Unlimited storage as well.
No, sorry, it does not exist. It is a marketing lie told by overselling hosts to get your business. These hosts promise it, but if you carefully read the fine print, it doesn't really exist, and they will suspend your account for ridiculous reasons. See this topic from the hosting boards -
http://www.simplemachines.org/community/index.php?topic=117303.0 (http://www.simplemachines.org/community/index.php?topic=117303.0)
Quote from: androidhelpersdotcom on August 29, 2010, 07:59:05 PM
I also have unlimited bandwidth, so it does exist =D Unlimited storage as well.
Your host may tell you it's unlimited, but technically it's not and cannot be.
Oh, well my host claims to have unlimited everything, so ha! *rolls eyes*
Read your webhost's TOS, or ask them what exactly is unlimited, if they are at least decent, they will tell you what it really is. If they say unlimited, then they are just talking out of their asses.
Thanks you guys ;)
Bad Behavior mod confirms that the main spiders are correct and does block many bad spiders.