SMF Support > Server Performance and Configuration

Proper Robots.txt?

(1/3) > >>

Xarcell:
I'm not sure if this is in the right board, but I wanted to make a robots.txt for my forum SMF installation. I have never made one before so this is a learning experience for me. Does this below seem proper to you?


--- Code: ---# robots.txt for http://www.example.com/
User-agent: *
Disallow: /Sources/
Disallow: /Themes/
Disallow: index.php?action=activate
Disallow: index.php?action=admin
Disallow: index.php?action=arcade
Disallow: index.php?action=calendar
Disallow: index.php?action=collapse
Disallow: index.php?action=deletemsg
Disallow: index.php?action=editpoll
Disallow: index.php?action=help
Disallow: index.php?action=helpadmin
Disallow: index.php?action=lock
Disallow: index.php?action=login
Disallow: index.php?action=logout
Disallow: index.php?action=markasread
Disallow: index.php?action=mergetopics
Disallow: index.php?action=mlist
Disallow: index.php?action=modifykarma
Disallow: index.php?action=movetopic
Disallow: index.php?action=notify
Disallow: index.php?action=notifyboard
Disallow: index.php?action=pm
Disallow: index.php?action=post
Disallow: index.php?action=profile
Disallow: index.php?action=printpage
Disallow: index.php?action=register
Disallow: index.php?action=removetopic2
Disallow: index.php?action=reporttm
Disallow: index.php?action=search
Disallow: index.php?action=sendtopic
Disallow: index.php?action=splittopics
Disallow: index.php?action=stats
Disallow: index.php?action=sticky
Disallow: index.php?action=trackip
Disallow: index.php?action=unread
Disallow: index.php?action=unreadreplies
Disallow: index.php?action=who
Disallow: index.php?PHPSESSID=*
Disallow: index.php?*rss*
Disallow: index.php?*wap*
Disallow: index.php?*wap2*
Disallow: index.php?*imode*
Disallow: index.php?wwwRedirect*
Disallow: *.msg
Disallow: *.new
Disallow: *sort
Disallow: *topicseen
Disallow: *prev_next*
Crawl-delay: 5
--- End code ---

Thanks.

ziycon:
There is loads of information regarding robot files available on the internet. Have a look through this site to give you a better understanding, http://www.robotstxt.org/.

Xarcell:

--- Quote from: ziycon on February 07, 2012, 04:43:03 AM ---There is loads of information regarding robot files available on the internet. Have a look through this site to give you a better understanding, http://www.robotstxt.org/.

--- End quote ---

Yes, I've already been reading that site.

I'm now trying to use google's webmaster tools to check for the robots.text file. When I test "mysite.com/Sources/", it says it's allowed, even when I have "Disallow: /Sources/" in my robots.txt file.

Any pointers?

EDIT: actually it isn't making any sense to me. It says "mysite.com/robots.txt" not found. However, when I link the link "mysite.com/robots.txt" I can view it.

16melih:
what must do this for google optimize and site performance?

青山 素子:
Google caches negative results for a little bit. Try it again in a few hours.

Navigation

[0] Message Index

[#] Next page