How to stop spider from crawling my site

Started by patricia2019, August 22, 2010, 11:42:05 AM

Previous topic - Next topic

patricia2019

Idon't mind google and yahoo spiders.
Please tell me how I could stop a specific spider from crawling my site.
Thanks



Masterd


Kill Em All

Also search the forums on robots.txt files, you can find a lot of examples of them around here.


My Site: KEAGaming.com

Manual Installation of Mods
Prevent Spam and Forum Attacks
Please do not PM or email me for support unless offered, help should be publicly displayed to others.

Masterd

Quote from: Kill Em All on August 22, 2010, 12:40:34 PM
Also search the forums on robots.txt files, you can find a lot of examples of them around here.

Or some popular sites.

Kill Em All

Well depending on the popular site.. different software packages maybe have different looking robots.txt.

example from smf:

User-agent: *
Disallow: /forum/index.php?action=activate*
Disallow: /forum/index.php?action=admin*


example from microsoft

User-agent: *
Disallow: /*TOCLinksForCrawlers*
Disallow: /*/mac/help.mspx


So sort of different styles.


My Site: KEAGaming.com

Manual Installation of Mods
Prevent Spam and Forum Attacks
Please do not PM or email me for support unless offered, help should be publicly displayed to others.

patricia2019

Thank you all for your answers, I would have to post a job to get this done.
Is it a BAD thing when these spiders crawl your site.  I did had a visit from Google, yahoo and Alexa as well?  Could they harm my site at all?

YogiBear

The three you mention are your best friends if you want folk to find you ie. your forum isn't restricted to family and friends.

Baidu can be a pain - it doesn't do any harm but crawls you relentlessly meaning it can eat up your bandwidth if you're running short on your monthly allocation.

On the 'Who's On Line' you'll sometimes see they are registering of logging-in but all that means is they're reading those pages.
SMF v2.1.3  Mods : Snow & Garland v1.4,  PHP  v.7.4.33

Kill Em All

I like spiders.... just not the creepy ones in the place some call the outside.


My Site: KEAGaming.com

Manual Installation of Mods
Prevent Spam and Forum Attacks
Please do not PM or email me for support unless offered, help should be publicly displayed to others.

IceXaos

Lol, usin' spiders to find what you need to stop spiders.  ???

manis99

usually spiders dont do any harm. Then why u want to stop them.

Kill Em All

My guess, as YogiBear has said, its because of limited bandwidth set by host.


My Site: KEAGaming.com

Manual Installation of Mods
Prevent Spam and Forum Attacks
Please do not PM or email me for support unless offered, help should be publicly displayed to others.

manis99

lol may be bt i dont need to worry at all cuz i hv unlimited bandwidth. :)

Kill Em All

Quote from: manis99 on August 29, 2010, 06:23:47 PM
lol may be bt i dont need to worry at all cuz i hv unlimited bandwidth. :)
No such thing...


My Site: KEAGaming.com

Manual Installation of Mods
Prevent Spam and Forum Attacks
Please do not PM or email me for support unless offered, help should be publicly displayed to others.

androidhelpersdotcom

I also have unlimited bandwidth, so it does exist =D Unlimited storage as well.
Using 2.0 RC3, Glassy Curve Theme

busterone

No, sorry, it does not exist. It is a marketing lie told by overselling hosts to get your business. These hosts promise it, but if you carefully read the fine print, it doesn't really exist, and they will suspend your account for ridiculous reasons. See this topic from the hosting boards -
http://www.simplemachines.org/community/index.php?topic=117303.0

IceXaos

Quote from: androidhelpersdotcom on August 29, 2010, 07:59:05 PM
I also have unlimited bandwidth, so it does exist =D Unlimited storage as well.
Your host may tell you it's unlimited, but technically it's not and cannot be.

Kill Em All

Oh, well my host claims to have unlimited everything, so ha! *rolls eyes*

Read your webhost's TOS, or ask them what exactly is unlimited, if they are at least decent, they will tell you what it really is. If they say unlimited, then they are just talking out of their asses.


My Site: KEAGaming.com

Manual Installation of Mods
Prevent Spam and Forum Attacks
Please do not PM or email me for support unless offered, help should be publicly displayed to others.


butchs

Bad Behavior mod confirms that the main spiders are correct and does block many bad spiders.
I have been truly inspired by the SUGGESTIONS as I sit on my throne and contemplate the wisdom imposed upon me.

Advertisement: