<noob mode>
I have a robots.txt file that says:
User-agent: *
Disallow: /cgi-bin/
and its located at www.nutronic.net/clan/robots.txt is that the correct place, im wanting every search engine to find my forum? or does it need to go to www.nutronic.net/robots.txt
There is nothing on the homepage of my site, i mean that its just a blank space lol.
The reason I need to know if everything is fine is because I found that google only lists www.nutronic.net/clan and none of its posts which is not good as I have installed the GoogleTagged mod too.
Robots.txt goes in the root folder, not subfolders (site.com/robots.txt)
ok but is the text in it correct?
I think so, not sure if you need the trailing "/" or not, but only honest bots obey it anyway. You can see what other sites use, like cnn.com/robots.txt.
You should turn off raw indexing to keep snoopers out too.
htaccess:
#disable viewing raw file directories
Options -Indexes
Then if you need a folder to be viewed, you can place a new htaccess in that folder saying "+indexes" :)