General Community > Scripting Help

Setting up robots.txt effectively

(1/2) > >>

Aaron10:
Hey guys, here's my current robots.txt:


--- Code: ---User-agent: *
Allow: /
Sitemap: http://www.website.com/sitemap1.xml

User-agent: *
Disallow: http://www.website.com/forums/index.php?action=register
Disallow: http://www.website.com/forums/index.php?action=login
Disallow: http://www.website.com/forums/index.php?action=search
--- End code ---

Just want to know if the disallow part is correct and whether or not I should add any other pages? Thanks.

Arantor:
Before worrying about what's correct or recommended, what exactly are you hoping to achieve with a robots.txt?

Aaron10:
It'll save Google, Yahoo etc. from accessing those pages and trying to to fill any forms in (although spam bots will still get through) it'll save bandwidth however small. And incase Google tries indexing them.

Arantor:
It will stop them doing so, yes.

Is that all you want the robots.txt to do? Do it want it to do something else?

Aaron10:
No that's pretty much it. Just wondering though if the

Allow: /

(Which is allow all) voids the Disallows below it?

Navigation

[0] Message Index

[#] Next page