SMF Support > Server Performance and Configuration
Proper Robots.txt?
Xarcell:
--- Quote from: 青山 素子 on February 07, 2012, 04:37:47 PM ---Google caches negative results for a little bit. Try it again in a few hours.
--- End quote ---
Thanks, I'll report back if there are any changes.
Xarcell:
Reporting back. Google now see my robots.txt file. However, google says the content of the robots.txt file is blank. Even when I test mysite.com/Sources/ it says allowed. Even when it shouldn't be. Any advice?
Here is what is EXACTLY in the file:
--- Code: ---User-agent: *
Disallow: /Sources/
Disallow: /Themes/
Disallow: /Test/
Disallow: /smf/
Disallow: index.php?action=activate
Disallow: index.php?action=admin
Disallow: index.php?action=arcade
Disallow: index.php?action=calendar
Disallow: index.php?action=collapse
Disallow: index.php?action=deletemsg
Disallow: index.php?action=editpoll
Disallow: index.php?action=help
Disallow: index.php?action=helpadmin
Disallow: index.php?action=lock
Disallow: index.php?action=login
Disallow: index.php?action=logout
Disallow: index.php?action=markasread
Disallow: index.php?action=mergetopics
Disallow: index.php?action=mlist
Disallow: index.php?action=modifykarma
Disallow: index.php?action=movetopic
Disallow: index.php?action=notify
Disallow: index.php?action=notifyboard
Disallow: index.php?action=pm
Disallow: index.php?action=post
Disallow: index.php?action=profile
Disallow: index.php?action=printpage
Disallow: index.php?action=register
Disallow: index.php?action=removetopic2
Disallow: index.php?action=reporttm
Disallow: index.php?action=search
Disallow: index.php?action=sendtopic
Disallow: index.php?action=splittopics
Disallow: index.php?action=stats
Disallow: index.php?action=sticky
Disallow: index.php?action=trackip
Disallow: index.php?action=unread
Disallow: index.php?action=unreadreplies
Disallow: index.php?action=who
Disallow: index.php?PHPSESSID=*
Disallow: index.php?*rss*
Disallow: index.php?*wap*
Disallow: index.php?*wap2*
Disallow: index.php?*imode*
Disallow: index.php?wwwRedirect*
Disallow: *.msg
Disallow: *.new
Disallow: *sort
Disallow: *topicseen
Disallow: *prev_next*
Crawl-delay: 5
--- End code ---
It also says "Line 52: Crawl-delay: 5 - Rule ignored by Googlebot"
Please advise...
Illori:
you put the robots.txt file in your forum or site root, not in each and every folder under that or any folder under that.
Roph:
This seems like a wasted effort. On useless pages (like those actions you include in your robots.txt) SMF includes a robots meta tag for noindex. For example, viewing the page source as I reply now:
<meta name="robots" content="noindex" />
Accessing subfolders results in redirection to the forum already:
http://www.simplemachines.org/community/Themes/
Ensiferous:
--- Quote from: Roph on February 08, 2012, 06:09:45 AM ---Accessing subfolders results in redirection to the forum already:
http://www.simplemachines.org/community/Themes/
--- End quote ---
But interestingly enough return a 302 status header. HTTP/1.1 302 Found.
Navigation
[0] Message Index
[#] Next page
[*] Previous page
Go to full version