Below is my robots.txt for my SMF installation. Is there anything wrong here? Will it affect my Page Ranking and Search Indexing? What are the downside of this robots.txt setup? Thanks
User-agent: *
Disallow: /cgi-bin/
Disallow: /admin/
Disallow: /attachments/
Disallow: /avatars/
Disallow: /Packages/
Disallow: /Smileys/
Disallow: /Sources/
Disallow: /Themes/
Disallow: /index.php?action=activate
Disallow: /index.php?action=admin
Disallow: /index.php?action=collapse
Disallow: /index.php?action=deletemsg
Disallow: /index.php?action=dlattach
Disallow: /index.php?action=editpoll
Disallow: /index.php?action=help
Disallow: /index.php?action=helpadmin
Disallow: /index.php?action=lock
Disallow: /index.php?action=login
Disallow: /index.php?action=logout
Disallow: /index.php?action=markasread
Disallow: /index.php?action=mergetopics
Disallow: /index.php?action=modifykarma
Disallow: /index.php?action=movetopic
Disallow: /index.php?action=notify
Disallow: /index.php?action=notifyboard
Disallow: /index.php?action=pm
Disallow: /index.php?action=post
Disallow: /index.php?action=printpage
Disallow: /index.php?action=register
Disallow: /index.php?action=reminder
Disallow: /index.php?action=removetopic2
Disallow: /index.php?action=reporttm
Disallow: /index.php?action=search
Disallow: /index.php?action=sendtopic
Disallow: /index.php?action=splittopics
Disallow: /index.php?action=sticky
Disallow: /index.php?action=trackip
Seems fine. Theres nothing wrong with it.
Query though.
Disallow: /admin/
If you've got an admin section at that point, its a REALLY bad idea to include an admin section in your robots.txt
There are bots/scripts that go around finding security/admin sections on sites by parsing peoples robots.txt.
Its basically the equivalent of putting a notice on your front door telling people you have a 'safe' in your house.
Karl, it's better to remove the disallow /admin ?
If you've got an admin section then yes.
If you can, it would be better to putting
<meta name="robots" content="noindex" />
Directly into the page of the admin site.
That way if a search engine does find your admin through some fluke (or accidental linkage), then it won't get indexed.
But your not actively pointing to it anywhere.
I don't have an admin section, actually. So I think I better remove that one. Actually I just copy the robots from a forum site suggestion.