Advertisement:

Author Topic: googlebot.txt ?  (Read 6594 times)

Offline sombra

  • Full Member
  • ***
  • Posts: 596
googlebot.txt ?
« on: November 15, 2008, 03:40:29 AM »
Analyzing URL: Open Window to Visit   http://.....net/
Loading...

Note: This site does NOT provide the robots.txt file or the robots.txt file is not constructed properly. It is recommended all sites provide this file.

is this a problem of smf that i need to fix  in smf or is a problem in my host?

can someone give me a hand here hehe

how can i fox this

Offline sombra

  • Full Member
  • ***
  • Posts: 596
Re: googlebot.txt ?
« Reply #1 on: November 15, 2008, 03:41:27 PM »
im using 2.04 smf if that helps is there any way to make my forum bot friendly

Offline H

  • SMF Friend
  • SMF Super Hero
  • *
  • Posts: 21,662
  • Gender: Male
Re: googlebot.txt ?
« Reply #2 on: November 15, 2008, 03:45:13 PM »
Please post a link to your site.
-H
Former Support Team Lead
                              I recommend:
Namecheap (domains)
Fastmail (e-mail)
Linode (VPS)
                             

Offline sombra

  • Full Member
  • ***
  • Posts: 596
Re: googlebot.txt ?
« Reply #3 on: November 15, 2008, 04:32:58 PM »

Offline H

  • SMF Friend
  • SMF Super Hero
  • *
  • Posts: 21,662
  • Gender: Male
Re: googlebot.txt ?
« Reply #4 on: November 15, 2008, 04:38:08 PM »
Your site doesn't have a robots.txt. This isn't essential but it can help search engine rankings in some circumstances. Search this board for robots.txt for some example entries :)
-H
Former Support Team Lead
                              I recommend:
Namecheap (domains)
Fastmail (e-mail)
Linode (VPS)
                             

Offline sombra

  • Full Member
  • ***
  • Posts: 596
Re: googlebot.txt ?
« Reply #5 on: November 15, 2008, 09:30:23 PM »
mmm im a newbie can you tell me if there is some place were i can get one already made?

Offline H

  • SMF Friend
  • SMF Super Hero
  • *
  • Posts: 21,662
  • Gender: Male
Re: googlebot.txt ?
« Reply #6 on: November 16, 2008, 08:47:24 AM »
You can't get ready made files because it depends what you're running on your site.

If you want one for SMF, just search this site for robots.txt. You'll find lots of examples. Then you just put them in a text file (using Wordpad/Notepad etc) and then save the file as robots.txt.

Then just upload it to your website using FTP. See: How do I use FTP? / What is FTP?
-H
Former Support Team Lead
                              I recommend:
Namecheap (domains)
Fastmail (e-mail)
Linode (VPS)
                             

Offline sombra

  • Full Member
  • ***
  • Posts: 596
Re: googlebot.txt ?
« Reply #7 on: November 16, 2008, 09:45:07 AM »
You can't get ready made files because it depends what you're running on your site.


it depends on my host or my web?

and how come my site dosent have a robot.txt its normal in a smf instalation? or im a rare case?
« Last Edit: November 16, 2008, 09:57:38 AM by sombra »

Offline sombra

  • Full Member
  • ***
  • Posts: 596
Re: googlebot.txt ?
« Reply #8 on: November 16, 2008, 10:02:16 AM »
i found this one can some one help me out to fix it i dont know to tweek it hehehe

for my site i dont care if google looks for all my site i just want him looking in my forum does this one work for me?
« Last Edit: November 16, 2008, 10:05:18 AM by sombra »

Offline sombra

  • Full Member
  • ***
  • Posts: 596
Re: googlebot.txt ?
« Reply #9 on: November 16, 2008, 12:45:33 PM »
i found this site http://www.xml-sitemaps.com/ to create sitemaps for google i want to know if it works heres the sitemap that it created for me if someone can look at it and tell me if its fine

Offline xconnect

  • Semi-Newbie
  • *
  • Posts: 43
Re: googlebot.txt ?
« Reply #10 on: November 16, 2008, 05:30:00 PM »

Offline H

  • SMF Friend
  • SMF Super Hero
  • *
  • Posts: 21,662
  • Gender: Male
Re: googlebot.txt ?
« Reply #11 on: November 17, 2008, 01:18:20 PM »
i found this site http://www.xml-sitemaps.com/ to create sitemaps for google i want to know if it works heres the sitemap that it created for me if someone can look at it and tell me if its fine

I'd recommend the Sitemap Mod instead. It will dynamically update and won't cause bandwidth to get used up when it updates, like an external sitemap creator would.

i found this one can some one help me out to fix it i dont know to tweek it hehehe

for my site i dont care if google looks for all my site i just want him looking in my forum does this one work for me?

That won't help for your site. Please do as I have suggested and run a proper search for robots.txt on this site. Not that a robots.txt is a nice extra, and is not required.
-H
Former Support Team Lead
                              I recommend:
Namecheap (domains)
Fastmail (e-mail)
Linode (VPS)
                             

Offline sombra

  • Full Member
  • ***
  • Posts: 596
Re: googlebot.txt ?
« Reply #12 on: November 17, 2008, 04:30:52 PM »
im using smf 2.04 that mod is comatible? and im using darkbreak theme

Offline LOADING...

  • Jr. Member
  • **
  • Posts: 305
  • Gender: Male
  • Everyday is a New Learning Day
    • http://wwww.facebook.com/ADMobOH on Facebook
    • @ADMobz on Twitter
    • ADMob Creative Marketing
Re: googlebot.txt ?
« Reply #13 on: November 17, 2008, 05:23:55 PM »
i found this site http://www.xml-sitemaps.com/ to create sitemaps for google i want to know if it works heres the sitemap that it created for me if someone can look at it and tell me if its fine

I'd recommend the Sitemap Mod instead. It will dynamically update and won't cause bandwidth to get used up when it updates, like an external sitemap creator would.

i found this one can some one help me out to fix it i dont know to tweek it hehehe

for my site i dont care if google looks for all my site i just want him looking in my forum does this one work for me?

That won't help for your site. Please do as I have suggested and run a proper search for robots.txt on this site. Not that a robots.txt is a nice extra, and is not required.

Thank you for the recommendation

Offline sombra

  • Full Member
  • ***
  • Posts: 596
Re: googlebot.txt ?
« Reply #14 on: November 18, 2008, 12:13:30 PM »
mm still confuced :-[

Offline Deprecated

  • SMF Hero
  • ******
  • Posts: 3,499
Re: googlebot.txt ?
« Reply #15 on: November 18, 2008, 01:33:30 PM »
Please read these two sites to gain an understanding of the subject:

http://www.robotstxt.org/

http://en.wikipedia.org/wiki/Robots.txt

Offline sombra

  • Full Member
  • ***
  • Posts: 596
Re: googlebot.txt ?
« Reply #16 on: November 18, 2008, 07:50:03 PM »
thanks helped alot :D i just made my robot.txt and now i know whats its for

but im having some trouble understanding the sitemaps since i tried to install the mod and i cudent i went again to a site so i can make it for me

i know that its like an index of my web but im asking my self do i need to update it periodically? if i dont what woud happend?

Offline Deprecated

  • SMF Hero
  • ******
  • Posts: 3,499
Re: googlebot.txt ?
« Reply #17 on: November 18, 2008, 08:26:33 PM »
Use "SMF Sitemap" and you won't have to update anything manually. That's what I use.* Unfortunately it is not clear from the Mod Site if SMF Sitemap works on SMF 1.1.7 or SMF 2.0 Beta 4. Perhaps the support topic has details.




*note: I've modified my Sitemap so my experience does not indicate whether it works with Beta 4.

Offline sombra

  • Full Member
  • ***
  • Posts: 596
Re: googlebot.txt ?
« Reply #18 on: November 18, 2008, 08:36:36 PM »
thanks for the reply im trying waiting on there reply but still i have the big question hehe if i or someone make the sitemap from an external page it will need to update frequently?

Offline H

  • SMF Friend
  • SMF Super Hero
  • *
  • Posts: 21,662
  • Gender: Male
Re: googlebot.txt ?
« Reply #19 on: November 20, 2008, 03:13:14 PM »
If you make the sitemap externally, then yes, you'll need to update it every time you get a new post / member.

Obviously it isn't crucial if it gets a little outdated, but it wouldn't give any benefit.
-H
Former Support Team Lead
                              I recommend:
Namecheap (domains)
Fastmail (e-mail)
Linode (VPS)