Simple Machines Community Forum

SMF Support => SMF 2.0.x Support => Topic started by: sombra on November 15, 2008, 03:40:29 AM

Title: googlebot.txt ?
Post by: sombra on November 15, 2008, 03:40:29 AM
Analyzing URL: Open Window to Visit   http://.....net/
Loading...

Note: This site does NOT provide the robots.txt file or the robots.txt file is not constructed properly. It is recommended all sites provide this file.

is this a problem of smf that i need to fix  in smf or is a problem in my host?

can someone give me a hand here hehe

how can i fox this
Title: Re: googlebot.txt ?
Post by: sombra on November 15, 2008, 03:41:27 PM
im using 2.04 smf if that helps is there any way to make my forum bot friendly
Title: Re: googlebot.txt ?
Post by: H on November 15, 2008, 03:45:13 PM
Please post a link to your site.
Title: Re: googlebot.txt ?
Post by: sombra on November 15, 2008, 04:32:58 PM
www.todoilegal.net
Title: Re: googlebot.txt ?
Post by: H on November 15, 2008, 04:38:08 PM
Your site doesn't have a robots.txt. This isn't essential but it can help search engine rankings in some circumstances. Search this board for robots.txt for some example entries :)
Title: Re: googlebot.txt ?
Post by: sombra on November 15, 2008, 09:30:23 PM
mmm im a newbie can you tell me if there is some place were i can get one already made?
Title: Re: googlebot.txt ?
Post by: H on November 16, 2008, 08:47:24 AM
You can't get ready made files because it depends what you're running on your site.

If you want one for SMF, just search this site for robots.txt. You'll find lots of examples. Then you just put them in a text file (using Wordpad/Notepad etc) and then save the file as robots.txt.

Then just upload it to your website using FTP. See: How do I use FTP? / What is FTP? (http://docs.simplemachines.org/index.php?topic=480)
Title: Re: googlebot.txt ?
Post by: sombra on November 16, 2008, 09:45:07 AM
Quote from: H on November 16, 2008, 08:47:24 AM
You can't get ready made files because it depends what you're running on your site.


it depends on my host or my web?

and how come my site dosent have a robot.txt its normal in a smf instalation? or im a rare case?
Title: Re: googlebot.txt ?
Post by: sombra on November 16, 2008, 10:02:16 AM
i found this one can some one help me out to fix it i dont know to tweek it hehehe

for my site i dont care if google looks for all my site i just want him looking in my forum does this one work for me?
Title: Re: googlebot.txt ?
Post by: sombra on November 16, 2008, 12:45:33 PM
i found this site http://www.xml-sitemaps.com/ to create sitemaps for google i want to know if it works heres the sitemap that it created for me if someone can look at it and tell me if its fine
Title: Re: googlebot.txt ?
Post by: xconnect on November 16, 2008, 05:30:00 PM
http://www.mcanerin.com/EN/search-engine/robots-txt.asp
Title: Re: googlebot.txt ?
Post by: H on November 17, 2008, 01:18:20 PM
Quote from: sombra on November 16, 2008, 12:45:33 PM
i found this site http://www.xml-sitemaps.com/ to create sitemaps for google i want to know if it works heres the sitemap that it created for me if someone can look at it and tell me if its fine

I'd recommend the Sitemap Mod (http://custom.simplemachines.org/mods/index.php?mod=755) instead. It will dynamically update and won't cause bandwidth to get used up when it updates, like an external sitemap creator would.

Quote from: sombra on November 16, 2008, 10:02:16 AM
i found this one can some one help me out to fix it i dont know to tweek it hehehe

for my site i dont care if google looks for all my site i just want him looking in my forum does this one work for me?

That won't help for your site. Please do as I have suggested and run a proper search for robots.txt on this site. Not that a robots.txt is a nice extra, and is not required.
Title: Re: googlebot.txt ?
Post by: sombra on November 17, 2008, 04:30:52 PM
im using smf 2.04 that mod is comatible? and im using darkbreak theme
Title: Re: googlebot.txt ?
Post by: LOADING... on November 17, 2008, 05:23:55 PM
Quote from: H on November 17, 2008, 01:18:20 PM
Quote from: sombra on November 16, 2008, 12:45:33 PM
i found this site http://www.xml-sitemaps.com/ to create sitemaps for google i want to know if it works heres the sitemap that it created for me if someone can look at it and tell me if its fine

I'd recommend the Sitemap Mod (http://custom.simplemachines.org/mods/index.php?mod=755) instead. It will dynamically update and won't cause bandwidth to get used up when it updates, like an external sitemap creator would.

Quote from: sombra on November 16, 2008, 10:02:16 AM
i found this one can some one help me out to fix it i dont know to tweek it hehehe

for my site i dont care if google looks for all my site i just want him looking in my forum does this one work for me?

That won't help for your site. Please do as I have suggested and run a proper search for robots.txt on this site. Not that a robots.txt is a nice extra, and is not required.

Thank you for the recommendation
Title: Re: googlebot.txt ?
Post by: sombra on November 18, 2008, 12:13:30 PM
mm still confuced :-[
Title: Re: googlebot.txt ?
Post by: Deprecated on November 18, 2008, 01:33:30 PM
Please read these two sites to gain an understanding of the subject:

http://www.robotstxt.org/

http://en.wikipedia.org/wiki/Robots.txt
Title: Re: googlebot.txt ?
Post by: sombra on November 18, 2008, 07:50:03 PM
thanks helped alot :D i just made my robot.txt and now i know whats its for

but im having some trouble understanding the sitemaps since i tried to install the mod and i cudent i went again to a site so i can make it for me

i know that its like an index of my web but im asking my self do i need to update it periodically? if i dont what woud happend?
Title: Re: googlebot.txt ?
Post by: Deprecated on November 18, 2008, 08:26:33 PM
Use "SMF Sitemap (http://custom.simplemachines.org/mods/index.php?mod=755)" and you won't have to update anything manually. That's what I use.* Unfortunately it is not clear from the Mod Site if SMF Sitemap works on SMF 1.1.7 or SMF 2.0 Beta 4. Perhaps the support topic has details.




*note: I've modified my Sitemap so my experience does not indicate whether it works with Beta 4.
Title: Re: googlebot.txt ?
Post by: sombra on November 18, 2008, 08:36:36 PM
thanks for the reply im trying waiting on there reply but still i have the big question hehe if i or someone make the sitemap from an external page it will need to update frequently?
Title: Re: googlebot.txt ?
Post by: H on November 20, 2008, 03:13:14 PM
If you make the sitemap externally, then yes, you'll need to update it every time you get a new post / member.

Obviously it isn't crucial if it gets a little outdated, but it wouldn't give any benefit.