Title: best content for robots.txt file
Post by: lepidas on June 01, 2013, 12:04:12 PM
Post by: lepidas on June 01, 2013, 12:04:12 PM
Hi to all, I was checking my /var/log/apache2/error.log file and saw that there is no robots.txt file in my /forum folder so I decided to create one but I don't know what its content should be.
My smf is 2.0.4
My smf is 2.0.4
Title: Re: best content for robots.txt file
Post by: vbgamer45 on June 01, 2013, 01:17:54 PM
Post by: vbgamer45 on June 01, 2013, 01:17:54 PM
You can keep it blank that is fine.
Title: Re: best content for robots.txt file
Post by: lepidas on June 01, 2013, 01:56:26 PM
Post by: lepidas on June 01, 2013, 01:56:26 PM
so you recomend to make a robots.txt and keep it blank or no file at all?
thanx anyway for answer
thanx anyway for answer
Title: Re: best content for robots.txt file
Post by: vbgamer45 on June 01, 2013, 02:02:55 PM
Post by: vbgamer45 on June 01, 2013, 02:02:55 PM
Yeah I do on my sites just to stop the error log.
Title: Re: best content for robots.txt file
Post by: Tyras on June 01, 2013, 03:54:48 PM
Post by: Tyras on June 01, 2013, 03:54:48 PM
use the robots.txt I have, this is very useful and prevents spiders from crawling non-needed and private paths.
ggamers.net/robots.txt
ggamers.net/robots.txt
Title: Re: best content for robots.txt file
Post by: MrPhil on June 01, 2013, 04:09:42 PM
Post by: MrPhil on June 01, 2013, 04:09:42 PM
It's good to have a robots.txt file, even if it's empty, to stop an error being logged every time a search engine asks for it from your site. Beyond that, it can be useful for stopping bots from sucking up resources following links to login pages, print pages, user profiles, alternate format (wap, imode) pages, etc.; and only catalog useful, non-duplicate pages from your site. Note that robots.txt should not be expected to provide any security, as it's only a suggestion to search engines of where not to look -- no one is obligated to obey it!
Title: Re: best content for robots.txt file
Post by: lepidas on June 01, 2013, 04:27:28 PM
Post by: lepidas on June 01, 2013, 04:27:28 PM
Quote from: dabandan on June 01, 2013, 03:54:48 PMoff topic: whats that mod you have different icons on each board at your forum?its very interesting :D
use the robots.txt I have, this is very useful and prevents spiders from crawling non-needed and private paths.
ggamers.net/robots.txt
Title: Re: best content for robots.txt file
Post by: lepidas on June 01, 2013, 04:29:18 PM
Post by: lepidas on June 01, 2013, 04:29:18 PM
well thank you all, I decided to create an empty robots.txt file and leave it empty :D
Title: Re: best content for robots.txt file
Post by: Tyras on June 01, 2013, 04:58:10 PM
Post by: Tyras on June 01, 2013, 04:58:10 PM
Quote from: lepidas on June 01, 2013, 04:27:28 PM
off topic: whats that mod you have different icons on each board at your forum?its very interesting :D
http://custom.simplemachines.org/mods/index.php?mod=2857
Title: Re: best content for robots.txt file
Post by: lepidas on June 01, 2013, 05:27:40 PM
Post by: lepidas on June 01, 2013, 05:27:40 PM
Quote from: Tyras on June 01, 2013, 04:58:10 PMthank you,thank you allQuote from: lepidas on June 01, 2013, 04:27:28 PM
off topic: whats that mod you have different icons on each board at your forum?its very interesting :D
http://custom.simplemachines.org/mods/index.php?mod=2857
Title: Re: best content for robots.txt file
Post by: lepidas on June 06, 2013, 04:34:00 AM
Post by: lepidas on June 06, 2013, 04:34:00 AM
Quote from: Tyras on June 01, 2013, 04:58:10 PMI was interested for the board.index icons, not menu iconsQuote from: lepidas on June 01, 2013, 04:27:28 PM
off topic: whats that mod you have different icons on each board at your forum?its very interesting :D
http://custom.simplemachines.org/mods/index.php?mod=2857
