Bots & Crawlers

[Shades.]

I have the robots section in admin set to not show bots as guests but googlebot (66.249.73.46) keeps showing up in the online list as a guest. How can I fix this in smf 2.0 RC1?

Also I have the Quiz mod installed and when I go to see who's online, sometimes it shows the googlebot in the Quiz and says: "Guest (66.249.73.46) doing something unknown in the Quiz". I'm not understanding this. I'm sure the bot is probably just crawling the Quiz area but it shoudn't show up at all...should it?

Thanks

[riker]

Funny I was going to make another post about the bots which is very simliar.  I've noticed two MSN bots which are showing as guest although we do have MSN bots showing too.

65.55.109.88 & 65.55.109.88 are showing as guests.  How would we get the IP range updated ?

[greyknight17]

If you set the Show spiders in the online list to not at all, it will show them as Guests only. If you don't like them showing as guests, set that option back to name or quantity instead.

Yes, it will show the spiderbots there even if it's restricted. They will just be crawling the login page though since it's restricted.

riker, you can add the IP range by going to Admin > Members > Search Engines > Spiders.

[karlbenson]

Riker, smf by default only detects spiders by useragent, not by IP. (although you are free to add it)

Also note, Microsoft uses the same IP ranges while testing out some of its new spidering stuff (with different non-detected useragents).

Note 2: Microsoft also sometimes sends spiders out with FAKE useragents. (probably like google, trying to detect people employing blackhat techniques eg showing content to spiders not to normal guests).