SMF 2.0 Spider Recognition Spider List

Started by Smith6612, March 28, 2008, 08:40:40 AM

Previous topic - Next topic

Smith6612

Hey, I'm running SMF 2.0 Beta 3 on my forum right now and it is running great on my home server. As for the Search Engine Recognition feature on there, I was wondering either the user-agent and/or the IP addresses of the search engines, other than Google and Yahoo. Would someone be able to provide me a list of this data, or should I take a dig through my web logs to find the bot user-agents/IPs? That's going to take a while though...

The bots I'd like besides the already set Google and Yahoo are:

Live Search
MSN Search (Live search now, but there are still MSN search bots going around)
Archive.org
Ask.com

Thanks!

Oldiesmann

MSN/Live: http://www.iplists.com/misc.txt (look for "MSN")

Archive: Don't know what the IPs are, but the official FAQ says the user agent is "archive.org_bot".

Ask.com/Teoma: http://www.iplists.com/misc.txt (look for "Ask/Teoma")
Michael Eshom
Christian Metal Fans

Advertisement: