best content for robots.txt file

Started by lepidas, June 01, 2013, 12:04:12 PM

Previous topic - Next topic

lepidas

Hi to all, I was checking my /var/log/apache2/error.log file and saw that there is no robots.txt file in my /forum folder so I decided to create one but I don't know what its content should be.
My smf is 2.0.4

vbgamer45

You can keep it blank that is fine.
Community Suite for SMF - Take your forum to the next level built for SMF, Gallery,Store,Classifieds,Downloads,more!

SMFHacks.com -  Paid Modifications for SMF

Mods:
EzPortal - Portal System for SMF
SMF Gallery Pro
SMF Store SMF Classifieds Ad Seller Pro

lepidas

so you recomend to make a robots.txt and keep it blank or no file at all?
thanx anyway for answer

vbgamer45

Yeah I do on my sites just to stop the error log.
Community Suite for SMF - Take your forum to the next level built for SMF, Gallery,Store,Classifieds,Downloads,more!

SMFHacks.com -  Paid Modifications for SMF

Mods:
EzPortal - Portal System for SMF
SMF Gallery Pro
SMF Store SMF Classifieds Ad Seller Pro

Tyras

use the robots.txt I have, this is very useful and prevents spiders from crawling non-needed and private paths.

ggamers.net/robots.txt

MrPhil

It's good to have a robots.txt file, even if it's empty, to stop an error being logged every time a search engine asks for it from your site. Beyond that, it can be useful for stopping bots from sucking up resources following links to login pages, print pages, user profiles, alternate format (wap, imode) pages, etc.; and only catalog useful, non-duplicate pages from your site. Note that robots.txt should not be expected to provide any security, as it's only a suggestion to search engines of where not to look -- no one is obligated to obey it!

lepidas

Quote from: dabandan on June 01, 2013, 03:54:48 PM
use the robots.txt I have, this is very useful and prevents spiders from crawling non-needed and private paths.

ggamers.net/robots.txt
off topic: whats that mod you have different icons on each board at your forum?its very interesting :D

lepidas

well thank you all, I decided to create an empty robots.txt file and leave it empty  :D

Tyras


lepidas


lepidas

Quote from: Tyras on June 01, 2013, 04:58:10 PM
Quote from: lepidas on June 01, 2013, 04:27:28 PM
off topic: whats that mod you have different icons on each board at your forum?its very interesting :D

http://custom.simplemachines.org/mods/index.php?mod=2857
I was interested for the board.index icons, not menu icons

Advertisement: