Archived Boards and Threads... > SMF Feedback and Discussion

Making your Smf forum as SEO friendly as possible? How to Tips

<< < (2/30) > >>

masternewbie:

--- Quote from: H on August 04, 2008, 04:00:14 PM ---Indeed you just upload a file called robots.txt with that text inside. It needs to go in your main sites folder (So it can be access via http://www.mysite.example/robots.txt rather than http://www.mysite.example/forum/robots.txt)

I'd recommend you do a search as SEO is a frequently discussed topic. You may also want to do a search for robots.txt as the one confusion has linked to is missing quite a few good entries.

--- End quote ---

What if I have my cpanel redirect my site domain (www.luckie8.com) to goto, PATH: www.luckie8.com/forum/   ?
Will the robots.txt still work if I put it in the root directory?

Deprecated:

--- Quote from: masternewbie on August 05, 2008, 11:36:57 PM ---Will the robots.txt still work if I put it in the root directory?
--- End quote ---

The method used to exclude robots from a server is to create a file on the server which specifies an access policy for robots. This file must be accessible via HTTP on the local URL "/robots.txt".

I take that to mean that it must be located at www.example.com/robots.txt

I've used my Apache .htaccess file to forward my entire domain with or without the www to /forum/ but I have added a line that excludes the robots.txt file:

RewriteCond %{REQUEST_URI} !robots.txt [NC]

Even though every other access to my domain with or without the www is forwarded to /forum/ my robots.txt is available to robots in my domain's root directory with or without the www.

I suggest you do the same.

kopchev:

--- Quote from: H on August 04, 2008, 04:00:14 PM ---Indeed you just upload a file called robots.txt with that text inside. It needs to go in your main sites folder (So it can be access via http://www.mysite.example/robots.txt rather than http://www.mysite.example/forum/robots.txt)

--- End quote ---

Disallow: /forum/*?action*
Disallow: /forum/*sort=*
Disallow: /forum/*msg*
 
Is this correct? I mainly targer google bot since MSN and Yahoo have no worth for my site

karlbenson:
No
- No * is needed at the end of each line, like I said, its implied.

Disallow: /forum/*?action
Disallow: /forum/*sort=
Disallow: /forum/*msg

Also note, the above will only work for Google and Yahoo.
It is an invalid robots.txt for other sites (including msn).

Therefore I'd suggest doing different blocks for each


User-agent: Googlebot
Disallow: /forum/*?action
Disallow: /forum/*sort=
Disallow: /forum/*msg

User-agent: Slurp
Disallow: /forum/*?action
Disallow: /forum/*sort=
Disallow: /forum/*msg

User-agent: *
Disallow: /forum/index.php?action


Note, the * (catch all) user agent always comes last. 
And there isn't a way to block msg, and sort links for the catchall since the basic robots.txt which most robots follow, don't support the wildcard *.  So you leave them off.

kopchev:
So,


--- Code: ---User-agent: Googlebot
Disallow: /forum/*?action
Disallow: /forum/*sort=
Disallow: /forum/*msg

User-agent: Slurp
Disallow: /forum/*?action
Disallow: /forum/*sort=
Disallow: /forum/*msg

User-agent: *
Disallow: /forum/index.php?action

--- End code ---

is fine?

My final robots.txt looks like (I run joomla)
edit 2:

User-agent: Googlebot
Disallow: /forum/*?action
Disallow: /forum/*sort=
Disallow: /forum/*msg

User-agent: Slurp
Disallow: /forum/*?action
Disallow: /forum/*sort=
Disallow: /forum/*msg

User-agent: *
Disallow: /administrator/
Disallow: /cache/
Disallow: /components/
Disallow: /editor/
Disallow: /help/
Disallow: /images/
Disallow: /includes/
Disallow: /language/
Disallow: /mambots/
Disallow: /media/
Disallow: /modules/
Disallow: /templates/
Disallow: /installation/
Disallow: /forum/index.php?action

Navigation

[0] Message Index

[#] Next page

[*] Previous page