Bot Balance..

[SleePy]

Anyone have suggestions for balancing spiders?
Yahoo has like 15 spiders to msn's 2 spiders to google's 1 spider on my site..
I dont mind the 15 spiders from yahoo.. Just would like to see msn and google go up...
Any suggestions on how I can do this? I know My site must be working for spiders because yahoo has 15 bots on.. but the others dont have really any...

[Juvenall Wilson]

For the most part, search engines will dispatch as many bots as it feels necessary to index your site and there isn't much you can do about it. Simply having "more" isn't any indication that more or less is being crawled. Remember, while Yahoo may have 15 spiders crawling around now, Google may have had 3 crawling for a few days.

It's also worth adding that more can cause some serious problems. Spiders are notorious for causing all sorts of server troubles. Say, 15 bots all running intensive processes on a small or shared server or sucking down an entire photo gallery, wasting your bandwidth. MSN's bot alone always manages to pull down around 400MB a month in bandwidth. I've also seen spiders crash servers with load. It's this reason that most spiders tend to pace themselves when crawling a page.

[TANDIONO.COM]

Sometime my forum could get up to 500+ of yahoo bots crawling on the site.

[Re@PeR]

How do you see the amount of bots on your forum without the spider/bot mod?

500 sounds insane! Someone should make a mod for SMF which can block spiders / bots or give you more control over them, i've seen a similar mod called the anti spam mod on XMB forums.

[Isaac]

How do you see the amount of bots on your forum without the spider/bot mod?

500 sounds insane! Someone should make a mod for SMF which can block spiders / bots or give you more control over them, i've seen a similar mod called the anti spam mod on XMB forums.

See what spiders have been to your forum using AwStats.