Is there a need to put robots.txt for google or yahoo to crawl?

[joseph21]

Hello, may I know what files and directories to disallow for the robots.txt ? - I tried searching but can't find a topic about this.

[snecz]

As I know there's meta tag robots with no-index in index.template by default.

[joseph21]

I don't quite understand.. How are we going to do this with google without robots.txt?

[SleePy]

google and most other search engines will follow meta tags and robots.txt
This meta tag in SMF 1.1 should help search engines search better and not find duplicate pages.

you can search around though if you wish to use a robots.txt still.

[joseph21]

Thanks. I found this on my site already with meta name noindex:
<meta name="robots" content="noindex" />
<meta name="keywords" content="PHP, MySQL, bulletin, board, free, open, source, smf, simple, machines, forum" />
Looks like I need to change the keywords.

But how is the done with SMF that when I search in yahoo or google for a random word.. let's say "SMF daniel15" then it will show most all links to simplemachines.org directly to that thread or post having the word "SMF daniel15". It seems to me that the words inside each posts in the forum itself is a meta tag?

[SleePy]

keywords are one way to help to do this. Another is just having clean content and allowing google to index the pages.