News:

Bored?  Looking to kill some time?  Want to chat with other SMF users?  Join us in IRC chat or Discord

Main Menu

SMF 2.0 Spider Recognition Spider List

Started by Smith6612, March 28, 2008, 08:40:40 AM

Previous topic - Next topic

Smith6612

Hey, I'm running SMF 2.0 Beta 3 on my forum right now and it is running great on my home server. As for the Search Engine Recognition feature on there, I was wondering either the user-agent and/or the IP addresses of the search engines, other than Google and Yahoo. Would someone be able to provide me a list of this data, or should I take a dig through my web logs to find the bot user-agents/IPs? That's going to take a while though...

The bots I'd like besides the already set Google and Yahoo are:

Live Search
MSN Search (Live search now, but there are still MSN search bots going around)
Archive.org
Ask.com

Thanks!

Oldiesmann

MSN/Live: http://www.iplists.com/misc.txt (look for "MSN")

Archive: Don't know what the IPs are, but the official FAQ says the user agent is "archive.org_bot".

Ask.com/Teoma: http://www.iplists.com/misc.txt (look for "Ask/Teoma")
Michael Eshom
Christian Metal Fans

Advertisement: