Thanks for any advice. I'm a web entrepreneur and have a popular blog for the apparel industry and recently added a business community with a bridged Smf community.
I have some backlinks from my own blogs and other sites. I get decent traffic with it over 30,000 pageviews per month for it to be fairly new and over 2,000 members mostly from the traffic from my main blog. I have some great articles and threads people are discussing in the forum. I want to find the best way to get them indexed into the search engines.
Any mods, tips, or general advice is greatly appreciated. I host my site with Media Temple if that helps with anything and you can look at the forum by going to http://fashionnetwork.dobizo.com/modules/smf/index.php
Thanks,
Dobizo
This is a pretty good list of SMF SEO optimization steps (http://www.jerry-bell.com/search-engine-optimization-for-the-simple-machines-forum/).
thats a really useful list of tips, i have done some of this stuff already.
does anybody know about this robots.txt ?do i just create a txt file in notepad and name it robots with the code User-agent: *
Disallow: /*?action*
Disallow: /*sort=*
Disallow: /*msg*
then upload that. i am geussin that once it is done there will be no feedback from the site or the robots as to wether the robots are making use of that so i want to check before i do it.
Indeed you just upload a file called robots.txt with that text inside. It needs to go in your main sites folder (So it can be access via http://www.mysite.example/robots.txt rather than http://www.mysite.example/forum/robots.txt)
I'd recommend you do a search as SEO is a frequently discussed topic. You may also want to do a search for robots.txt as the one confusion has linked to is missing quite a few good entries.
Re robots.txt
and
User-agent: *
Disallow: /*?action*
Disallow: /*sort=*
Disallow: /*msg*
* for use in the disallow line is NOT supported by most search engines (other than Google,Yahoo) [msn does not support it for one.]
* at the end of the disallow line is unnecessary as it is IMPLIED (the implied at the end is supported by all robots, which for the reason about is the reason why its better not to specify them if you can.
My robots.txt can be found
http://www.youposted.com/robots.txt
I have different sections for different robots and based on the parameters that each can support.
- Yahoo is too aggressive so I limit it to everything but topics.
- Google I allow free reign
Quote from: H on August 04, 2008, 04:00:14 PM
Indeed you just upload a file called robots.txt with that text inside. It needs to go in your main sites folder (So it can be access via http://www.mysite.example/robots.txt rather than http://www.mysite.example/forum/robots.txt)
I'd recommend you do a search as SEO is a frequently discussed topic. You may also want to do a search for robots.txt as the one confusion has linked to is missing quite a few good entries.
What if I have my cpanel redirect my site domain (www.luckie8.com) to goto, PATH: www.luckie8.com/forum/ ?
Will the robots.txt still work if I put it in the root directory?
Quote from: masternewbie on August 05, 2008, 11:36:57 PMWill the robots.txt still work if I put it in the root directory?
The method used to exclude robots from a server is to create a file on the server which specifies an access policy for robots. This file must be accessible via HTTP on the local URL "/robots.txt". (http://www.robotstxt.org/orig.html#method)
I take that to mean that it must be located at www.example.com/robots.txt
I've used my Apache .htaccess file to forward my entire domain with or without the www to /forum/ but I have added a line that excludes the robots.txt file:
RewriteCond %{REQUEST_URI} !robots.txt [NC]
Even though every other access to my domain with or without the www is forwarded to /forum/ my robots.txt is available to robots in my domain's root directory with or without the www.
I suggest you do the same.
Quote from: H on August 04, 2008, 04:00:14 PM
Indeed you just upload a file called robots.txt with that text inside. It needs to go in your main sites folder (So it can be access via http://www.mysite.example/robots.txt rather than http://www.mysite.example/forum/robots.txt)
Disallow: /forum/*?action*
Disallow: /forum/*sort=*
Disallow: /forum/*msg*
Is this correct? I mainly targer google bot since MSN and Yahoo have no worth for my site
No
- No * is needed at the end of each line, like I said, its implied.
Disallow: /forum/*?action
Disallow: /forum/*sort=
Disallow: /forum/*msg
Also note, the above will only work for Google and Yahoo.
It is an invalid robots.txt for other sites (including msn).
Therefore I'd suggest doing different blocks for each
User-agent: Googlebot
Disallow: /forum/*?action
Disallow: /forum/*sort=
Disallow: /forum/*msg
User-agent: Slurp
Disallow: /forum/*?action
Disallow: /forum/*sort=
Disallow: /forum/*msg
User-agent: *
Disallow: /forum/index.php?action
Note, the * (catch all) user agent always comes last.
And there isn't a way to block msg, and sort links for the catchall since the basic robots.txt which most robots follow, don't support the wildcard *. So you leave them off.
So,
User-agent: Googlebot
Disallow: /forum/*?action
Disallow: /forum/*sort=
Disallow: /forum/*msg
User-agent: Slurp
Disallow: /forum/*?action
Disallow: /forum/*sort=
Disallow: /forum/*msg
User-agent: *
Disallow: /forum/index.php?action
is fine?
My final robots.txt looks like (I run joomla)
edit 2:
User-agent: Googlebot
Disallow: /forum/*?action
Disallow: /forum/*sort=
Disallow: /forum/*msg
User-agent: Slurp
Disallow: /forum/*?action
Disallow: /forum/*sort=
Disallow: /forum/*msg
User-agent: *
Disallow: /administrator/
Disallow: /cache/
Disallow: /components/
Disallow: /editor/
Disallow: /help/
Disallow: /images/
Disallow: /includes/
Disallow: /language/
Disallow: /mambots/
Disallow: /media/
Disallow: /modules/
Disallow: /templates/
Disallow: /installation/
Disallow: /forum/index.php?action
There is an issue.
Each spider will ONLY follow ONE block.
So google won't follow Googlebot section and the * section.
You need to specify for the FULL lot for each spider.
eg
User-agent: Googlebot
Disallow: /forum/*?action
Disallow: /forum/*sort=
Disallow: /forum/*msg
Disallow: /administrator/
Disallow: /cache/
Disallow: /components/
Disallow: /editor/
Disallow: /help/
Disallow: /images/
Disallow: /includes/
Disallow: /language/
Disallow: /mambots/
Disallow: /media/
Disallow: /modules/
Disallow: /templates/
Disallow: /installation/
User-agent: Slurp
Disallow: /forum/*?action
Disallow: /forum/*sort=
Disallow: /forum/*msg
Disallow: /administrator/
Disallow: /cache/
Disallow: /components/
Disallow: /editor/
Disallow: /help/
Disallow: /images/
Disallow: /includes/
Disallow: /language/
Disallow: /mambots/
Disallow: /media/
Disallow: /modules/
Disallow: /templates/
Disallow: /installation/
User-agent: *
Disallow: /administrator/
Disallow: /cache/
Disallow: /components/
Disallow: /editor/
Disallow: /help/
Disallow: /images/
Disallow: /includes/
Disallow: /language/
Disallow: /mambots/
Disallow: /media/
Disallow: /modules/
Disallow: /templates/
Disallow: /installation/
Disallow: /forum/index.php?action
You could also block the smf source and theme files if necessary
Disallow: /forum/Themes
Disallow: /forum/Sources
Ok, I will make the changes you suggested. Thanks very much... I have another question. Will this solve the problem with the duplicate meta descriptions and title tags errors in Google Webmasters, that the forum causes?
Karl: Thank you so much for your post. I just read Jerry Bell's article (http://www.jerry-bell.com/search-engine-optimization-for-the-simple-machines-forum/)where I presume the suggested SMF robots file came from, added his mods (with the asterisks) to my site yesterday after satisfying myself with what each line did, but I had no idea the end of line wild card wasn't valid. I've updated my own robots.txt file per your suggestion. :)
Kopchev: while there's no harm, your file is probably needlessly overcomplicated. There is no need to list folders if they do not have links pointing to them on your site. Robots cannot see folders unless they are linked. I don't know which of those can be eliminated but I'm sure there is no link to your /cache/ folder, and there are probably others. Being executed by calls from other PHP files (e.g. index.php) does not qualify as being linked.
All: It is important that you must check your robots.txt file not by eyeballing it but by use of automated checkers such as that provided in the Google Webmaster tools (http://www.google.com/webmasters/tools/). The file suggested by Karl may be perfectly fine for most SMF forums, but if your forum is located in your root as mine is, the /forum/ part is incorrect. I hadn't realized this until I tested my robots.txt using the Webmaster Analyze Robots.txt tool.
You can paste your tentative robots.txt file into the Google page, add a URL you want to test against it, then hit the button and see the result, either allowed or disallowed. Go ahead and test using some action, sort and msg URLs and verify that they work right. You can assume that if you've got it right for Google that it will be right for Yahoo too.
The "official" unofficial robots.txt site is located at http://www.robotstxt.org/orig.html and I suggest that everybody should read this quasi-standard and consider your robots.txt file in that context.
Also, please remember any changes you make to your robots.txt take 24hrs to take effect (as search engines cache them).
The problem with robots.txt is that the most only support the basic robots.txt, but then there were later ones which some have adopted, but different parts. So it is a nightmare
Deprecated, the end * is valid only for search engines which support a wildcard character.
However a wildcard is already implied by default.
so
Disallow: /cache
also matches
Disallow: /cachefile
Disallow: /cache/
Google and Yahoo support the $ character to prevent the IMPLIED wildcard, eg
Disallow: /cache/$
would only match that, but none of the others I posted above.
MSN only supports $ when used in conjunction with * for filenames ONLY.
Disallow: /avatars/*.png$
I agree with your syntax although I'm not so sure there's any point in including your /cache/ directory since I doubt there is any link a searchbot could follow to discover your cache directory. I presume you just stated that as an example of syntax.
For my own forum I focus 80% of my attention towards Google. They are actively friendly in soliciting the cooperation of website operators and provide personalized tools that allow you to specify the interaction between the Googlebot and your site, and then provide you with regularly updated feedback on how they perceive your site. Google even allows you to set a sitemap.xml to aid their finding your pages.
Yahoo seems to have found me and they visit fairly regularly, but I've never been able to find any way to get their attention or how to optimize my site to get the best results with Yahoo. They're just like the weather, no point in doing anything about it except being prepared.
MSN? Pfffff!!! They are inscrutable, monolithic, impervious. Besides, who uses MSN search anyway? They can come, they can see, and then they can go conquer some place else. ;)
I'm pleased that Google and I have a very nice arrangement. My final lines for User-agent: * suffice for whatever else and I don't care what it is, devil take the hindmost. If they send me any visitors, fine. If not, that's fine too.
The only sure thing I ask is that any Disallow directories I include better well not be indexed! I've got a static site that enforces my robots.txt disallowed directories, including robot bait and auto-ban. I'm not that concerned about my forum site. When I want to feel superior to bad robots I go look at my banned robots log at the other site. :P :P :P
We have become focused on the topic of robots.txt files, so I would like to expand the discussion and get back to the original titled topic: Making your forum as SEO friendly as possible. Here are the steps I've taken so far:
1.) Register for Google Webmaster Tools (http://www.google.com/webmasters/tools/): Visit their site, sign up for a free account. Validate your authority over your site by either placing a coded META tag on your index page (not that easy for SMF) or placing a coded name file in your root (easy). Have a look around and discover the various reports including error conditions.
2.) Install SMF Sitemap (http://custom.simplemachines.org/mods/index.php?mod=755): SlammedDime's mod to add an XML sitemap to your forum, compatible with SMF 1.1.* thru 2.0b3.1p. This creates a new action for your index.php that dynamically generates an XML sitemap of the current state of your site whenever it is accessed. Then go back to your Google Webmaster Tools and enter the URL to your sitemap so that Google can find it.
3.) Install vBulletin Style Meta Tags (http://custom.simplemachines.org/mods/index.php?mod=1138): rsw686's mod that allows SMF to generate different META tags depending on the board or topic accessed. This allows each topic on your site to have different meta tags, and the tags are more relevant to those specific pages. The description tags are generated from the initial words of the post. It's not likely you will be able to educate your members to get to their point in the first sentence, but I'm pleased as long as my members don't write in textese. ;) Anyway the mod's tags are much better than static tags for the entire site. Note also that you must enter the default tags such as are used on your index page. The mod has a place to configure what you want to use.
4.) Have a close look and give a lot of thought to your .htaccess file (Apache servers only). If you have none, your site probably appears as both example.com and www.example.com (http://www.example.com) which has been said may confuse the robots. You can tell Google which you prefer in your Webmaster Tools, but why not just fix it on your end for everybody? I've got two versions of this file for two different forum sites. One causes all requests to be redirected to www.example1.com (http://www.example1.com) while the other causes all requests to be redirected to www.example2.com/forum/ (http://www.example2.com/forum/) (different URLs, different forums). Both automatically adjust to whether the www is present or not, both use a Redirect 301 permanently moved when the requested URL is wrong, to nudge the search engines into remembering. Note also that .htaccess redirections should be designed to permit your robots.txt file to function properly. If you redirect EVERYTHING to /forum/ the robots.txt file in your root is inaccessible. There is some uncertainty whether any robots.txt file in /forum/ would be honored or even found. I could test for this but honestly I couldn't be bothered. I just made sure my own file in the root is accessible.
5.) Make sure that the parts of your forum that you want indexed are accessible by guests. From your end all robots look like guests, and if you restrict your site material to require registration, the searchbots are going to get nada.
6.) Get your site linked to from other sites. If you control several websites, put links to your forum on each, and enlist friends to do the same. The more sites that link to you, the more valuable the search engines consider your content. Just remember that there's no use unless the other sites are crawled too, so you may need to take steps and at least get the page with the link indexed. I've gotta do this myself... Been too busy getting #1-#5 done.
Okay, that's my tips to share with y'all. Now what I want from youse is to tell me if I have missed anything, and share your ideas on how to get the attention of the search engines. Remember, if you aren't on the search engines your forum will find it difficult to grow. If you get lots of hits on the search engine, people will be drawn to your site from web searches, and your forum will be more likely to gain popularity.
I've done alot of seo type stuff on my forum. I've probably done everything apart from SEO-urls {which in my opinion, have little effect [and would actively result in short-medium term negative pagerank and seo impact if I switched on my 2 year old forum}.
Indeed google is the big cheese.
Yahoo / MSN are less importance, but Yahoo needs controlling purely because its the most aggressive spider in the world.
Indeed, I've wrote a sitemap generating script (similar to the smf sitemap mod), and have submitted it via google webmasters.
And I'm also linking to it in my robots.txt (http://www.youposted.com/robots.txt) with the auto discovery feature of robots.txt.
I had a lot of fun one day trying to adjust my .htaccess file and write a custom php script to make it appear that robots.txt was being accessed while instead my robots.php was spitting out the text. I wanted to be able to dynamically decide what to send depending on who asked. I almost got it working but there was one final, little flaw and I can't remember what it was now. But that's a story for another day...
I can't see how SMF's default URLs could be any more SEO friendly. I doubt SEOs care how the actual URL is spelled, whether it's http://www.example.com/index.php?topic=150.0 (http://www.example.com/index.php?topic=150.0) or http://www.example.com/what_a_pleasant_title_for_a_topic.html (http://www.example.com/what_a_pleasant_title_for_a_topic.html) ... Actually that's what I've done with my vanity cooking site, pretty thread names dynamically generated by one php script and .htaccess file, but that too is a story for another day...
I've researched the topic of SEO friendly URLs and as far as I can tell all the big search engines handle the ?query=selector form just fine.
Well my list above must have covered practically all of it since I know you're a SMF veteran and if you don't have anything to add then I must be almost there. :)
I am glad too have stirred up some discussion here :)
A few notes from my side:
1. I have observed a notable difference in the volume of traffic to my busiest forum from Google referrals after switching to pretty URLs. I had been using the old style URL's for a long time and was a bit worried about taking a hit when I switched, but I did not notice a decline.
2. The sitemaps mod does not work when using pretty urls
3. vbulletin meta tags only work for SMF 2
4. Focusing on google makes a lot of sense. Take a look at these stats from one of my sites (copied from awstats output):
- Google 7044 7081
- Windows Live 790 790
- Yahoo! 378 378
- Google (cache) 30 536
- Dogpile 21 21
- MSN Search 17 17
- AltaVista 16 16
- Unknown search engines 11 79
- AllTheWeb 10 10
- Baidu 9 9
- Swik (Social Bookmark) 7 7
- Mozbot 6 6
- MetaCrawler (Metamoteur) 4 4
- Yandex 4 4
- del.icio.us (Social Bookmark) 3 3
- Netscape 3 3
- Search.com 2 2
- AOL 2 2
- Avantfind 1 1
- WebCrawler 1 1
- Stumbleupon (Social Bookmark) 1 1
- MetaBot 1 1
- Scroogle 1 1
- Ask 1 1
It's nearly a factor of 10 difference between google referrals and the other SE's. I am #2 on google for my keyword, but #1 on just about all other SE's.
5. There are many many many off-page activities to increase SERP ranking, I was focusing only on on-page improvements. If you are looking for a good place to get some ideas for the off-page improvements, read some of the threads over at http://forums.digitalpoint.com.
Could you please describe exactly what "pretty URLs" are, and give an example? I don't want to install the mod just to be able to see what they are.
I may be mistaken but I believe that Google/Yahoo/MSN are quite capable of understanding SMF's default URLs. I couldn't be bothered with the other web crawlers.
Perhaps your 10x difference between Google and the other SEs could be due to Google's vast size?
And what about SMF's "Search engine friendly URLs" ? As I said above, the default URLs seem to work fine. Actually since Google indexes my site regularly, I could care less what the other SEs do. Google is the 800 pound gorilla.
To see pretty URL's in action, take a look at this site: http://www.syslog.org/forum/syslog-and-syslogd/
Normally, the URL's look like this: http://www.syslog.org/forum/index.php?topic=147.0
The "Search engine friendly URL" setting makes the URL's look like this: http://www.syslog.org/forum/index.php/topic,147.0.html
Pretty URLs make URLs look like this: http://www.syslog.org/forum/syslog-and-syslogd/implementation-of-plain-syslog-tcp/
Like it or not, the presence of keywords in a URL is material to the weighting search engines (even Google) give to a page. I am not asserting that Google (or any other SE) can't or won't parse the URL's used by SMF. I am saying that the appearance of the title in the URL gives additional weighting to the page.
The majority of internet users search on Google, which is partly why the numbers are so high. The other part is that technical people (my site is technical in nature) tend to shun MSN, Yahoo, and all other SE's in favor of Google.
When I look at a site I have that is not technical in nature (also not running SMF), I have the following stats:
- Yahoo! 593 593
- Google 508 508
- Windows Live 25 25
- AOL 17 17
- Unknown search engines 13 13
- MSN Search 7 7
- GoodSearch 4 4
- Avantfind 1 1
- Earth Link 1 1
- WebCrawler 1 1
- Google (Images) 1 2
- Ask 1 1
- AltaVista 1 1
- Google (cache) 0 12
So, Google is king to be sure, but I would not downplay the importance of the other search engines.
QuoteLike it or not, the presence of keywords in a URL is material to the weighting search engines (even Google) give to a page
The presence of keywords in the link bears no effect towards pagerank. (source google).
They might however come into play as general keywords on the page where they are linked.
This is why the benefit of seo-type urls is so hotly debated.
Some say there is no real benefit, some say there is alot, some say there is (and the bracket which I prescribe to) a little.
The main benefit of seo-urls is that some people think they look prettier.
(IMO again they are a perversion on what seo-urls started out at).
Thank you so much for clearing up the ... er ... confusion, confusion. ;)
Well now that you've explained it I can see your point and I agree about the desirability of the pretty URLs. However, I really like the sitemap, so now it's a choice between one or the other. It's a shame that Sitemap is not compatible. I think I'll go make a post to encourage the author to update his mod for pretty URL compatibility.
I'm going to stick with the default + Sitemap, but I think I'll add the mod to my test site to explore how it works. Who knows, maybe I can figure out how to fix Sitemap and then we'd have the best of both worlds.
I hope others will add to this thread. I had intended to start the same thread myself, but noticed your discussion and figured it would be best to concentrate the discussion to one place. Let's keep those SEO suggestions coming in folks! :)
Well I can see that Karl has added a post while I was composing.
I agree with you Karl, that SEO URL optimization is a hotly debated topic. As far as humans who look at my URLs, pfffftttt!!!! ;) I could care less. :)
However I am totally convinced that since Google encourages the use of a sitemap, practically clubs you into submitting one to their webmaster tools, that that one thing is probably the single most valuable SEO trick in the book. In my opinion when you're with the gorilla you've got 90% of the job done!
Well I've submitted sitemaps with GoogleWebmasters. It doesn't give you that much.
Overall SEO'ing a forum doesn't help as much as lots of unique Content + Backlinks.
Quote from: karlbenson on August 08, 2008, 10:17:00 AM
The presence of keywords in the link bears no effect towards pagerank. (source google).
They might however come into play as general keywords on the page where they are linked.
That is true, but Google also states that Page Rank is but one of roughly 200 variables that impact the SERP of a site for a key word. In addition to back links, as has been pointed out, there is a strong need to clearly identify to the SE's what the page is about. The contents of the URL are a factor in that equation.
Quote from: Deprecated on August 08, 2008, 10:22:03 AM
However, I really like the sitemap, so now it's a choice between one or the other. It's a shame that Sitemap is not compatible. I think I'll go make a post to encourage the author to update his mod for pretty URL compatibility.
I'm going to stick with the default + Sitemap, but I think I'll add the mod to my test site to explore how it works. Who knows, maybe I can figure out how to fix Sitemap and then we'd have the best of both worlds.
I agree. I don't like having to choose. For a while (until the load got to be too high) I was running a crawler on my site that generated a site map that I ha submitted to Google.
Yeah external crawlers can overload sites.
Its better to do it with internal scripts on the database. Although even they can lock up a forum with super-long queries.
I personally cache/save the sitemap.xml file generated for 24hrs.
To get back on track to the topic of broader SEO for SMF, the vbulletin meta tags are a very valuable thing, but are only available on SMF >= 2, so do not work for me.
There are some serious SE gurus that believe that forcing links on a forum (that are not internal links) to use the rel=nofollow tag. The logic is that forums will be penalized by SE's for having large numbers of outbound links. There is an SMF mod called NoFollow All Links (http://custom.simplemachines.org/mods/index.php?mod=1236) that apparently will do just that. I have not played with it. I have a thought on how to improve it, though. Forum owners not only want their forums to rank highly in SERPs, they also want to have an active forum. Let's be very honest, there are a non-trivial number of forum participants that post simply for the backlinks in their signature blocks, but genuinely contribute. Forcing all links to nofollow will cause some number of those people to go elsewhere, which is somewhat counter productive. Some forums I have seen take a stance of now allowing links in a signature or post until the poster gets to X posts (25 or 50 typically). A hybrid approach would be to allow such links from the start, but below a certain threshold of posts, the links would be nofollow. This would encourage participation and reduce the number of links exposed to SE's (from all of the posters who drop in and make one or two posts, never to return). I suppose that is more of a suggestion for an improvement to a module than a suggestion on SEO.
Getting back to Sitemap and robots.txt files, I just discovered something that should be shared:
This line in your robots.txt file
Disallow: /*?action
Is going to prevent Google from accessing your Sitemap
http://www.example.com/index.php?action=sitemap;xml (http://www.example.com/index.php?action=sitemap;xml)
What you need to do is to add this line to your robots.txt
Allow: /index.php?action=sitemap;xml
I have verified that it works properly.
Quote from: Deprecated on August 08, 2008, 09:25:10 AM
Could you please describe exactly what "pretty URLs" are, and give an example? I don't want to install the mod just to be able to see what they are.
They are the ones that have text in them versus a numeric ID. The argument is that when people just post the URL as the link, the text in the URL will help. So, it's a tangental benefit and not direct. Some people argue that it has value in itself, but there are plenty of studies showing this isn't the case. Personally, I think it's a waste of resources to do it, and is especially hard on the system as you grow.
Thanks Motoko. I went ahead and installed Pretty URLs and they were exactly what I had expected, .htaccess mods and all. Quite a big programming job I might add. Unfortunately it totally crashed my forum. Fortunately it was my test forum. :) It took about 20 minutes to manually uninstall it, a good lesson in itself. I decided it wasn't worth my trouble to figure out what the problem was.
There's a better way to expose intelligence in links: Show Local URL Titles (on the mod site)
I agree with you that it's a waste of resources, including a waste of my time to debug it, although I'm sure others have had better luck.
Quote from: karlbenson on August 04, 2008, 04:37:57 PM
My robots.txt can be found
http://www.youposted.com/robots.txt
I have different sections for different robots and based on the parameters that each can support.
- Yahoo is too aggressive so I limit it to everything but topics.
- Google I allow free reign
Would it be okay for me to use this robots.txt in my own forum? Can i delete the aracade and downloads parts? as i dont have those section in my forum.
.
Karl has said before that you are free to use his robots.txt and modify it to fit your needs.
Yes feel free.
thanks karl i only altered it slightly and i left credit to youposted.com
@depracted
@confusion
in previous posts you are talking about 'pretty urls' and 'sitemap' mods not being compatible. i have both working together perfectly on smf 1.1.5. are you using different version?
I used SMF 2.0b3.1 so perhaps it's a versional issue. I installed Pretty Urls and my site blew up, became unusable. I had to manually uninstall it (not that hard).
You sure your server supports both .htaccess and mod_rewrite? Noone else seems to be having major problems anymore.
Damned sure. I already had a healthy .htaccess file I wrote myself. I've been writing them for a couple years. In fact I have a vanity website written entirely in PHP that dynamically generates about 100 HTML pages, except none of the pages exist even though they appear that way. I'm using a similar technique that you used for Pretty Urls. I analyzed and fully understood the Pretty Urls .htaccess mods, and they looked okay, but in the end it just wasn't worth my trouble to debug it further since I don't think the pretty URLs make any difference to SEOs, or at least not enough to matter to me. Please note that I did not intend to deprecate the mod. It just wasn't for me.
i use it in my test site with no problems at all on SMF 2.0 beta 3.1 :)
Well I'm sorry I mentioned it. I didn't mean to start an argument. I installed it, had trouble, decided to just manually uninstall and that was it. I started with no mods other than Sitemap installed, but maybe it was just a glitch. I've seen SMF's install/remove glitch before and probably anybody who has installed/removed a lot has seen that. I began by noting which files were affected, and uploaded fresh copies of those files directly from my 2.0b3.1 local reference. Then I installed Sitemap and then I installed Pretty Urls. Actually the whole point of the exercise was that somebody had asked me if the two mods were compatible. I was curious what the mod looked like so I was willing. When my test site blew up I lost interest and just wanted to get back to testing. I bet if I tried installing it again it would probably go fine.
Actually that Pretty Urls is one hell of a coding job! I looked in the source and there's a lot there, really a lot!!! :)
i was simply stating that i had no problem with it. :P
theres a million and one things that could prevent it from working properly and such.. such is the way of this industry :P
Quote from: metallica48423 on August 14, 2008, 07:45:13 PM
i was simply stating that i had no problem with it. :P
theres a million and one things that could prevent it from working properly and such.. such is the way of this industry :P
Yeah, yeah, yeah. All the girls tell me that! ;)
:P ;)
ouch :P
I didn't want an argument either, I just want to find some of those million and one things that prevent my mod from working and fix them!
Aw let's just duke it out!!! Name your second! I declare antechinus as my second!
No really Dannii, my install just burped. I'm sure it happens all the time with every mod. And again, I commend you on a great, fearless job of programming compounded by a foray into the .htaccess file!
Sheesh, I've spent hours, even a full day developing several lines of Apache directives. That stuff is HEAVY!!!! :)
I pride myself on having a finely tuned .htaccess file. Almost every day I learn more about .htaccess. More people here at SMF forums should endeavor to understand and use their .htaccess file. I've already helped a few n00bz and I'm willing to tackle more .htaccess problems. I pore over the new posts every day wondering if .htaccess would help them.
Of course if your site doesn't run Apache then all bets are off. Yay, go Apache!!! :)
Well I want to get my mod working on other httpds eventually... some of them have rewrite engines, but I might move the processing into PHP instead. This is how wordpress does it.
Just FYI Dannii my site runs Apache 1.3.34
I still think my install just burped. No reflection on your mod. Again, a masterful programming job!!! :)
God, I just L*O*V*E the Apache Rewrite engine!!! :) :) :)
Quote from: karlbenson on August 08, 2008, 10:17:00 AM
QuoteLike it or not, the presence of keywords in a URL is material to the weighting search engines (even Google) give to a page
The presence of keywords in the link bears no effect towards pagerank. (source google).
They might however come into play as general keywords on the page where they are linked.
You are 100% correct - keywords in a URL does not improve or hurt PR. Anyone who claims otherwise does not understand what PR is. PR is not relevant for a key word, it is a ranking given to a page, based on how authoritative Google calculates that page to be.
Anyone that is well familiar with SEO will tell you that PR is not all that valuable an attribute of a site. What is really valuable to the site owner is the SERP for target keywords (essentially where your site will fall in the results for a given keyword). So, keywords in the URL are not useful in the determination of PR, but keywords in the URL are important in the SERP for a given keyword.
So, if you have 2 identical sites, with identical content (excluding for a moment any theorized duplicate content penalty), same back links, etc, but one had standard SMF urls, and the other had pretty URLs, the site with pretty URLs will rank more highly for a keyword, where the keyword appears in the URL. That is all I am trying to assert about pretty URL's.
I don't think that's correct. What matters is the link texts. So pretty URLs will help, but only when the URL itself is used for the link text.
Quote from: karlbenson on August 04, 2008, 04:37:57 PM
Re robots.txt
and
User-agent: *
Disallow: /*?action*
Disallow: /*sort=*
Disallow: /*msg*
* for use in the disallow line is NOT supported by most search engines (other than Google,Yahoo) [msn does not support it for one.]
* at the end of the disallow line is unnecessary as it is IMPLIED (the implied at the end is supported by all robots, which for the reason about is the reason why its better not to specify them if you can.
My robots.txt can be found
http://www.youposted.com/robots.txt
I have different sections for different robots and based on the parameters that each can support.
- Yahoo is too aggressive so I limit it to everything but topics.
- Google I allow free reign
Karl,
If you have your site setup in as a subdomain, is it okay to put the robots.txt in the directory of the subdomain?
And a separate one for the main domain?
Or only one.
Thanks
No it is not. Generally speaking robots look only in the / folder for the robots.txt file.
You should read this page: http://www.robotstxt.org/
Thanks depreciated.
My problem is I have WordPress installed in the public_html and the robots.txt file is being generated by the sitemap generator I use. It is a plugin and does a good job.
Google's webmaster tools allows me to treat them as different sites as do sitemaps and Google Analytics.
I used a free sitemap generator without the robots.txt and I exceeded the 500 URL that the free version of the generator allows. After putting the robots.txt file in the "home" directory of the subdomain the sitemap generator was able to map the entire board as it used the robots.txt
Maybe my wordpress plug will only append the wordpress portion of the robots.txt file.
If anyone has experience with this, please let me know. I've gotta do something, what I've been getting with site:domain.com is a total mess. :)
I also told Google that the both the domain and the subdomain where there with Google Webmaster Tools, maybe I'll sit on it for a bit and see what happens.
Been a while since I read a tutorial on robots.txt. I need to do that again, anyone know of a good one?
Sickthing--L*O*V*E your username you sick SOB ;)--I just gave you the URL for the best robots.txt site on the Internet. You sick... sick... heh, lol! Sick is good! ;)
I gotta learn wordpress myself. I just signed on to the management team of a new website, and the owner wants to add wordpress so I gotta learn WTF wordpress is. 'Druther not, but you gotta please the site owners or they get a new team...
LOL you did give me a URL. I just woke up and totally missed it. Only two lines how could I miss that. I think you edited it. :)
I'm a crazed Alice Cooper fan, thus the nick. Alice wrote a song in the 70's about his fans, it is named Sickthing's :)
PS WordPress is good for getting good SEO. I know in the right hands Joomla, SMF, are just as good but I guess my hands are not right. :) I do much better with WP.
LOL@Sick Thing. :)
Well if I edited my post, then where's the edit line? ROFLMAO! You just don't want to admit that your reading comprehension is 50%... Or less. ;)
Hey man I'm an Alice Cooper fan too. I attended their 1972 "School's Out" concert! Man that was some kick ass concert! :D It was at the Hollywood Bowl and I just love outdoor concerts. The smoke doesn't get so thick that way. ;)
Me wrong? I would agree with you but then we'd both be wrong! LOL
Wow, you went during the hey day. I was about 14 in 72. Didn't start going to concerts until 75, started with ZZ Top.
Hmmm, i prefer indoor so the smoke doesn't clear and I get to re-breath. LOL Oh wait, maybe that's why my reading comprehension is less than 50%
Check out my guitar.
(http://sickthing.com/images/stories/cooper1916a.jpg)
Nice ax! :) I used to play... We're taking this technical thread off-topic. Feel free to start a music discussion in Chit Chat. :)
..
@ karlbenson -
thanks very much for sharing your work on your robots.txt. As my SMF projects are the first to be directly advertising supported, I had been looking for such advice. I'm pretty familiar with the right structure for static sites, but knew that the script driven world would be very, very different. I feel at least 2K lumens brighter!
To all who posted, thanks for your insights on SEO and forums
/me must now agonize over pretty URLS!
domo arigato
Quote from: Deprecated on August 07, 2008, 01:07:49 PM
For my own forum I focus 80% of my attention towards Google. They are actively friendly in soliciting the cooperation of website operators and provide personalized tools that allow you to specify the interaction between the Googlebot and your site, and then provide you with regularly updated feedback on how they perceive your site. Google even allows you to set a sitemap.xml to aid their finding your pages.
Yahoo seems to have found me and they visit fairly regularly, but I've never been able to find any way to get their attention or how to optimize my site to get the best results with Yahoo. They're just like the weather, no point in doing anything about it except being prepared.
MSN? Pfffff!!! They are inscrutable, monolithic, impervious. Besides, who uses MSN search anyway? They can come, they can see, and then they can go conquer some place else. ;)
Since the karlbenson example is no longer available, would you so kind as to post a robots.txt file that I can use for my own forum?
Thanks in advance.
Here it is from my site. I only modified it to use the "forum" path. You'll need to adjust to the path your forum is viewable at:
###################################
# Robots.txt
#
# Based on:
# YouPosted.com Smart Robots v3.05
###################################
#
# My Sitemap - I don't provide it just for the fun of it
Sitemap: http://YOURDOMAINHERE/sitemap.xml
# Google - Most Important bot
# Unfortunately a robots.txt will only stop it crawling certain urls, and NOT adding any
# urls which it comes across into its index. So we're relying on a meta noindex tag.
User-agent: Googlebot
# Don't index mobile versions
Disallow: /forum/index.php?*;wap
Disallow: /forum/index.php?*;wap2
Disallow: /forum/index.php?*;imode
# Yahoo - Too aggressive
# So limit it as much as possible.
User-agent: Slurp
# Disallow Everything
Disallow: /forum/
# Now allow bits and then disallow bits
Allow: /sitemap.xml$
Allow: /robots.txt$
Allow: /forum/index.php$
Allow: /forum/index.php?topic=*.0$
Allow: /forum/index.php?topic=*.*0$
Allow: /forum/index.php?topic=*.*5$
Allow: /forum/index.php?board=*.0$
Allow: /forum/index.php?board=*.*0$
Allow: /forum/index.php?board=*.*5$
# But don't allow these
Disallow: /forum/index.php?*.msg
Disallow: /forum/index.php?topic=*.msg*0$
Disallow: /forum/index.php?topic=*.msg*5$
Disallow: /forum/index.php?*.new
# Anything with a ; disallow
Disallow: /forum/index.php?*;*
# Bad bot - Often ignores robots.txt - Waste of bandwidth
# Despite claiming on their website to be a search engine in development
# I'm suspicious as to whether they are a harvester pretending to be SE
User-agent: Twiceler
Disallow: /
User-Agent: W3C-checklink
Disallow: /
User-agent: TurnitinBot
Disallow: /
# Stop following PHPSESSID's
User-Agent: MJ12bot
Disallow: /forum/index.php?PHPSESSID
# Catch all (remainder)
# Will be followed by any bots other than ones identified above
# Uses BASIC robots.txt directives without wildcards, end-anchors etc
# So Spiders should understand these (including MSNBOT)
User-agent: *
# Default SMF Folders
Disallow: /forum/attachments/
Disallow: /forum/Packages/
Disallow: /forum/Smileys/
Disallow: /forum/Sources/
Disallow: /forum/Themes/
# Default SMF Actions
Disallow: /forum/index.php?action=activate
Disallow: /forum/index.php?action=admin
Disallow: /forum/index.php?action=calendar
Disallow: /forum/index.php?action=emailuser
Disallow: /forum/index.php?action=findmember
Disallow: /forum/index.php?action=help
Disallow: /forum/index.php?action=helpadmin
Disallow: /forum/index.php?action=login
Disallow: /forum/index.php?action=logout
Disallow: /forum/index.php?action=mlist
Disallow: /forum/index.php?action=modifykarma
Disallow: /forum/index.php?action=pm
Disallow: /forum/index.php?action=post
Disallow: /forum/index.php?action=printpage
Disallow: /forum/index.php?action=profile
Disallow: /forum/index.php?action=recent
Disallow: /forum/index.php?action=register
Disallow: /forum/index.php?action=reminder
Disallow: /forum/index.php?action=search
Disallow: /forum/index.php?action=theme
Disallow: /forum/index.php?action=unread
Disallow: /forum/index.php?action=unreadreplies
Disallow: /forum/index.php?action=verificationcode
Disallow: /forum/index.php?action=who
Disallow: /forum/index.php?theme
Disallow: /forum/index.php?action=stats;expand
Disallow: /forum/index.php?action=stats;collapse
Quote from: Motoko-chan on December 04, 2008, 11:08:53 PM
Here it is from my site. I only modified it to use the "forum" path. You'll need to adjust to the path your forum is viewable at
My forum is in the root of my domain so I would remove all of the /forum that are present and leave it as such-
# Google - Most Important bot
# Unfortunately a robots.txt will only stop it crawling certain urls, and NOT adding any
# urls which it comes across into its index. So we're relying on a meta noindex tag.
User-agent: Googlebot
# Don't index mobile versions
Disallow: /index.php?*;wap
Disallow: /index.php?*;wap2
Disallow: /index.php?*;imode
# Yahoo - Too aggressive
# So limit it as much as possible.
User-agent: Slurp
# Disallow Everything
Disallow: /forum/
# Now allow bits and then disallow bits
Allow: /sitemap.xml$
Allow: /robots.txt$
Allow: /index.php$
Allow: /index.php?topic=*.0$
Allow: /index.php?topic=*.*0$
Allow: /index.php?topic=*.*5$
Allow: /index.php?board=*.0$
Allow: /index.php?board=*.*0$
Allow: /index.php?board=*.*5$
# But don't allow these
Disallow: /index.php?*.msg
Disallow: /index.php?topic=*.msg*0$
Disallow: /index.php?topic=*.msg*5$
Disallow: /index.php?*.new
# Anything with a ; disallow
Disallow: /forum/index.php?*;*
# Bad bot - Often ignores robots.txt - Waste of bandwidth
# Despite claiming on their website to be a search engine in development
# I'm suspicious as to whether they are a harvester pretending to be SE
User-agent: Twiceler
Disallow: /
User-Agent: W3C-checklink
Disallow: /
User-agent: TurnitinBot
Disallow: /
# Stop following PHPSESSID's
User-Agent: MJ12bot
Disallow: /index.php?PHPSESSID
# Catch all (remainder)
# Will be followed by any bots other than ones identified above
# Uses BASIC robots.txt directives without wildcards, end-anchors etc
# So Spiders should understand these (including MSNBOT)
User-agent: *
# Default SMF Folders
Disallow: /attachments/
Disallow: /Packages/
Disallow: /Smileys/
Disallow: /Sources/
Disallow: /Themes/
# Default SMF Actions
Disallow: /index.php?action=activate
Disallow: /index.php?action=admin
Disallow: /index.php?action=calendar
Disallow: /index.php?action=emailuser
Disallow: /index.php?action=findmember
Disallow: /index.php?action=help
Disallow: /index.php?action=helpadmin
Disallow: /index.php?action=login
Disallow: /index.php?action=logout
Disallow: /index.php?action=mlist
Disallow: /index.php?action=modifykarma
Disallow: /index.php?action=pm
Disallow: /index.php?action=post
Disallow: /index.php?action=printpage
Disallow: /index.php?action=profile
Disallow: /index.php?action=recent
Disallow: /index.php?action=register
Disallow: /index.php?action=reminder
Disallow: /index.php?action=search
Disallow: /index.php?action=theme
Disallow: /index.php?action=unread
Disallow: /index.php?action=unreadreplies
Disallow: /index.php?action=verificationcode
Disallow: /index.php?action=who
Disallow: /index.php?theme
Disallow: /index.php?action=stats;expand
Disallow: /index.php?action=stats;collapse
Correct?
Or do I remove all of the / that are still present as well?
dlackner, that looks fine.
Thanks
One last question- Since as I mentioned, my forum is installed in my root directory- could you edit the following code to remove the /forum references from this particular section as I do not know how that should be done.
# Yahoo - Too aggressive
# So limit it as much as possible.
User-agent: Slurp
# Disallow Everything
Disallow: /forum/
# Now allow bits and then disallow bits
Allow: /sitemap.xml$
Allow: /robots.txt$
Allow: /index.php$
Allow: /index.php?topic=*.0$
Allow: /index.php?topic=*.*0$
Allow: /index.php?topic=*.*5$
Allow: /index.php?board=*.0$
Allow: /index.php?board=*.*0$
Allow: /index.php?board=*.*5$
# But don't allow these
Disallow: /index.php?*.msg
Disallow: /index.php?topic=*.msg*0$
Disallow: /index.php?topic=*.msg*5$
Disallow: /index.php?*.new
# Anything with a ; disallow
Disallow: /forum/index.php?*;*
Just remove "/forum" and leave the remaining.
No. 3 on the list below doesn't work with smf 1.1.7. Is there another work around? Also I have SMF installed in a subdirectory that google can't seem to get to. How do I work around this? path = mysite.com/forum/index.php
Quote from: Deprecated on August 07, 2008, 01:33:46 PM
We have become focused on the topic of robots.txt files, so I would like to expand the discussion and get back to the original titled topic: Making your forum as SEO friendly as possible. Here are the steps I've taken so far:
1.) Register for Google Webmaster Tools (http://www.google.com/webmasters/tools/): Visit their site, sign up for a free account. Validate your authority over your site by either placing a coded META tag on your index page (not that easy for SMF) or placing a coded name file in your root (easy). Have a look around and discover the various reports including error conditions.
2.) Install SMF Sitemap (http://custom.simplemachines.org/mods/index.php?mod=755): SlammedDime's mod to add an XML sitemap to your forum, compatible with SMF 1.1.* thru 2.0b3.1p. This creates a new action for your index.php that dynamically generates an XML sitemap of the current state of your site whenever it is accessed. Then go back to your Google Webmaster Tools and enter the URL to your sitemap so that Google can find it.
3.) Install vBulletin Style Meta Tags (http://custom.simplemachines.org/mods/index.php?mod=1138): rsw686's mod that allows SMF to generate different META tags depending on the board or topic accessed. This allows each topic on your site to have different meta tags, and the tags are more relevant to those specific pages. The description tags are generated from the initial words of the post. It's not likely you will be able to educate your members to get to their point in the first sentence, but I'm pleased as long as my members don't write in textese. ;) Anyway the mod's tags are much better than static tags for the entire site. Note also that you must enter the default tags such as are used on your index page. The mod has a place to configure what you want to use.
4.) Have a close look and give a lot of thought to your .htaccess file (Apache servers only). If you have none, your site probably appears as both example.com and www.example.com (http://www.example.com) which has been said may confuse the robots. You can tell Google which you prefer in your Webmaster Tools, but why not just fix it on your end for everybody? I've got two versions of this file for two different forum sites. One causes all requests to be redirected to www.example1.com (http://www.example1.com) while the other causes all requests to be redirected to www.example2.com/forum/ (http://www.example2.com/forum/) (different URLs, different forums). Both automatically adjust to whether the www is present or not, both use a Redirect 301 permanently moved when the requested URL is wrong, to nudge the search engines into remembering. Note also that .htaccess redirections should be designed to permit your robots.txt file to function properly. If you redirect EVERYTHING to /forum/ the robots.txt file in your root is inaccessible. There is some uncertainty whether any robots.txt file in /forum/ would be honored or even found. I could test for this but honestly I couldn't be bothered. I just made sure my own file in the root is accessible.
5.) Make sure that the parts of your forum that you want indexed are accessible by guests. From your end all robots look like guests, and if you restrict your site material to require registration, the searchbots are going to get nada.
6.) Get your site linked to from other sites. If you control several websites, put links to your forum on each, and enlist friends to do the same. The more sites that link to you, the more valuable the search engines consider your content. Just remember that there's no use unless the other sites are crawled too, so you may need to take steps and at least get the page with the link indexed. I've gotta do this myself... Been too busy getting #1-#5 done.
Okay, that's my tips to share with y'all. Now what I want from youse is to tell me if I have missed anything, and share your ideas on how to get the attention of the search engines. Remember, if you aren't on the search engines your forum will find it difficult to grow. If you get lots of hits on the search engine, people will be drawn to your site from web searches, and your forum will be more likely to gain popularity.
Welcome to SMF. Both those mods are compatible with SMF 2.x beta. If you wanted, you could Upgrade (http://docs.simplemachines.org/index.php?board=3.0;sort=subject) :)
Quote from: H on December 15, 2008, 06:08:50 AM
Welcome to SMF. Both those mods are compatible with SMF 2.x beta. If you wanted, you could Upgrade (http://docs.simplemachines.org/index.php?board=3.0;sort=subject) :)
I use yahoo for hosting, which unfortunately doesn't support the .htaccess file (wont allow it to be uploaded). How then do I adjust for this? At issue is the fact that none of my smf forum pages or post are visible when searched on google www.atlantainvesting.net/forum/index.php (http://www.atlantainvesting.net/forum/index.php). Could the issue be the fact that the real index page contains an instant redirect to the forum? (from www.atlantainvesting.net to www.atlantainvesting.net/forum/index.php)
Please give me a clue here. I.m using 1.1.7
Try using Google Webmaster Tools (http://www.google.com/webmasters/) and look at the index stats. It should let you know why no indexing is being done on that.
Quote from: Motoko-chan on December 15, 2008, 10:18:20 AM
Try using Google Webmaster Tools (http://www.google.com/webmasters/) and look at the index stats. It should let you know why no indexing is being done on that.
From google webmaster:
Sitemap summary
Most sites will not have all of their pages indexed.
Improve how Google crawls and indexes your site. More information.
Property Status
Sitemap type Web
Format Sitemap
Submitted 10 hours ago
Last downloaded by Google 10 hours ago
Status OK
Total URLs in Sitemap 57
Indexed URLs in Sitemap Help No data available. Please check back soon.
Sitemap errors and warnings
No errors or warnings found.
What does this tell me?
That isn't the correct section (Diagnostics -> Web crawl is where you want to look for errors), but it did make me think. Your main sitemap doesn't list any of your forum, so Google is ignoring it. Check http://www.atlantainvesting.net/sitemap.xml.
Sitemaps cut both ways. They let you make sure all the URLs are listed, but if you don't list something, it doesn't get indexed. I suggest you either delete the sitemap files or generate a full sitemap at that location.
Search engines are good at picking up URLs. I only suggest using a sitemap when you need to modify some behavior or you see big gaps in indexing. They only help you get indexed, they don't help rank on searches or anything like that.
Hello!
I followed some of the tips regarding the optimization of my SMF board and now i'm stuck at the following problems:
1. I use the sitemap mod and i submited mine (http://www.forum-brasov.ro/index.php?action=sitemap;xml)to google; after that i created robots.txt file, as described here (http://www.jerry-bell.com/search-engine-optimization-for-the-simple-machines-forum/). Unfortunately, it appears that /*?action* does not permit Google to follow my sitemap, or my gallery. Though, i want to prevent crawlers to follow Help, Search, Statistics etc. and other pages that, i guess, may hurt my ranking.
Now, in Webmaster's tools i have:
URL restricted by robots.txt
We encountered an error while trying to access your Sitemap. Please ensure your Sitemap follows our guidelines and can be accessed at the location you provided and then resubmit.
What do you suggest?
2. I observed in Diagnostics | Content analysis, that i have the following issues:
- duplicate meta descriptions;
- short meta descriptions;
- duplicate title tags.
The downloaded tables containing all the information may be found in the attachment.
I am most annoyed by the duplicate title tags issue and i don't understand why this is happening; by example:
http://www.forum-brasov.ro//index.php?topic=2.0;prev_next=prev (http://www.forum-brasov.ro//index.php?topic=2.0;prev_next=prev);
http://www.forum-brasov.ro/index.php?topic=3.0 (http://www.forum-brasov.ro/index.php?topic=3.0)
http://www.forum-brasov.ro/index.php?topic=3.msg%msg_id (http://www.forum-brasov.ro/index.php?topic=3.msg%msg_id)
http://www.forum-brasov.ro/index.php?topic=3.msg3 (http://www.forum-brasov.ro/index.php?topic=3.msg3)
all of them open the same page.
Motoko-chan, i would like to use your robot.txt file, but i am wondering though on how to prevent the duplicate content that appears for ?action=print instruction more than imode or wap stuff because i don't use those formats yet for my forum.
Many thanks!
[Later edit]
I forgot to mention that i enabled Search engine friendly URL's in Basic features page of SMF, and the topic put as example at question #2, is accessible by default with:
http://www.forum-brasov.ro/index.php/topic,3.0.html (http://www.forum-brasov.ro/index.php/topic,3.0.html)
Also, i installed metatags 1.1 mod for 1.1.x forums which should have solved the duplicate and short meta description issues; maybe the problem reported by google appeared before installing this mod?
You shouldn't be blocking: *action* unless you're using a rewrite mod that doesn't use action urls.
I didn't knew that; now i modified my robots.txt; i hope that it will be ok.
Thanks and Happy new year to all! :D
Hi all,
I am creating my 1st robots.txt file, I want both google and yahoo bots to be able to index my entire forum except for my staff area - Admin, Global, staff and recycle bin.
My forum is the the root dir on my host, but I dont want the robots being able to access anything else in the root dir eg like cgi or anything else just the boards on my forum.
How do I go about doing this?
Thanks in advance
Rob
If guests don't have access to those boards, the search engines will not either.
o ok, well guests do have limited access on my forum, so its a case of either, give guests full forum access and have bots (and open spammers) or little or no guest access and no bots?
Basically, a search engine bot is like a guest. They can only see the stuff that guests can see. They don't need posting permissions, however. In SMF 2.0, you can restrict recognized search engines to see less than a normal guest, but never more.
You don't want a search engine to see more than a normal guest. That technique is usually grouped with a few others that are considered cloaking. This can get you in trouble and get you removed from the index.
Quote from: Deprecated on August 07, 2008, 01:33:46 PM
We have become focused on the topic of robots.txt files, so I would like to expand the discussion and get back to the original titled topic: Making your forum as SEO friendly as possible. Here are the steps I've taken so far:
1.) Register for Google Webmaster Tools (http://www.google.com/webmasters/tools/): Visit their site, sign up for a free account. Validate your authority over your site by either placing a coded META tag on your index page (not that easy for SMF) or placing a coded name file in your root (easy). Have a look around and discover the various reports including error conditions.
2.) Install SMF Sitemap (http://custom.simplemachines.org/mods/index.php?mod=755): SlammedDime's mod to add an XML sitemap to your forum, compatible with SMF 1.1.* thru 2.0b3.1p. This creates a new action for your index.php that dynamically generates an XML sitemap of the current state of your site whenever it is accessed. Then go back to your Google Webmaster Tools and enter the URL to your sitemap so that Google can find it.
I registered a Google Webmaster Tools account and submitted my Sitemmap (Sitemap 2.0.0 Mod)
I keep getting this error after submission...
Unsupported file format
Your Sitemap does not appear to be in a supported format. Please ensure it meets our Sitemap guidelines and resubmit.
Is there a different format available? Or which one should I use?
@zerog12avity: add ;xml to your sitemap address (url/forum/index.php?action=sitemap;xml) and it will work just fine.
I tested it and it's ok, google will accept it this way.
i put the mod in.... now my entire site will not show up, and will not delete the mod.
www.DeadTix.info
Quote from: ccraciun on February 13, 2009, 02:54:17 PM
@zerog12avity: add ;xml to your sitemap address (url/forum/index.php?action=sitemap;xml) and it will work just fine.
I tested it and it's ok, google will accept it this way.
Thanks ccraciun. It worked.
Quote from: Deprecated on August 07, 2008, 01:33:46 PM
1.) Register for Google Webmaster Tools (http://www.google.com/webmasters/tools/): Visit their site, sign up for a free account. Validate your authority over your site by either placing a coded META tag on your index page (not that easy for SMF) or placing a coded name file in your root (easy). Have a look around and discover the various reports including error conditions.
2.) Install SMF Sitemap (http://custom.simplemachines.org/mods/index.php?mod=755): SlammedDime's mod to add an XML sitemap to your forum, compatible with SMF 1.1.* thru 2.0b3.1p. This creates a new action for your index.php that dynamically generates an XML sitemap of the current state of your site whenever it is accessed. Then go back to your Google Webmaster Tools and enter the URL to your sitemap so that Google can find it.
Ok, 1st one worked.
The sitemap however did not.. :(
Sitemap type Web
Format –
Submitted 7 minutes ago
Last downloaded by Google 6 minutes ago
Status ERRORS
Total URLs in Sitemap 0
Indexed URLs in Sitemap Help –
Unsupported file format
Your Sitemap does not appear to be in a supported format. Please ensure it meets our Sitemap guidelines and resubmit.
Quote from: protonxl on March 05, 2009, 09:13:12 PM
The sitemap however did not.. :(
Make sure you are using the URL to the XML version, not the html version.
whoops you're right
I just did it with ;xml at the end, and it worked.
:D
i do wish i could get pretty urls..
Why do you want "pretty" URLs? There is no direct SEO benefit.
I use pretty URLs for the users sake and for linking.
Users tend to find it easier to navigate the site and also, if they link something on my site, readers wherever the link is posted will have an easier time determing what the content of the link is.
At least, that is my take on it.
Quote from: Deprecated on August 07, 2008, 01:33:46 PM
2.) Install SMF Sitemap (http://custom.simplemachines.org/mods/index.php?mod=755): SlammedDime's mod to add an XML sitemap to your forum, compatible with SMF 1.1.* thru 2.0b3.1p. This creates a new action for your index.php that dynamically generates an XML sitemap of the current state of your site whenever it is accessed. Then go back to your Google Webmaster Tools and enter the URL to your sitemap so that Google can find it.
I have installed this, now where does the sitemap.xml file reside at? I cannot find it. I'm trying to point the google webmaster tools at this. I see a sitemap.xml file in my root directory that is from wordpress, but I don't show any sitemap.xml in my root/forums directory. Thanks for the help.
Did you look at http://yourdomain.com/locationofsmfinstallation/index.php?action=sitemap;xml ?
Yes, your right I found it now, and added. I assume this won't create an issue at all with my wordpress xml? Thanks
You'd have to ask SlammedDime (the Mod author) on the Mod support topic for that, I'm afraid. Sorry I couldn't help.
Thanks for the robot.txt tips. Very useful indeed.
dear motoko-chan
are your robot.txt can be placed in every smf sites, generally ?
how bout SMF with preety url ? should i remove it first or not ?
BTW thanx for the robot
Quote from: gisfreak on May 31, 2010, 06:07:39 PM
are your robot.txt can be placed in every smf sites, generally ?
It should work fine on any SMF site.
Quote from: gisfreak on May 31, 2010, 06:07:39 PM
how bout SMF with preety url ? should i remove it first or not ?
It probably won't help.
Quote from: gisfreak on May 31, 2010, 06:07:39 PM
BTW thanx for the robot
It was originally by Karl Benson, so all credit should go to him.
hello motoko-chan, thanx for your reply, just to make sure something ;D
i saw that the robot using standard SMF URL format, and my SMF using preety url syntax, is it compatible, although different URL style ?
second question
where should i place the robots.txt ? my forum is under /forum not in the root folder ?
and third question
i found in SEO mod, there are SEF mod and pretty URL, what is the different ? i saw that they have same purpose, to make URL more simple and Search engine Friendly. did i miss something on that ?
thanx, GISfreak
it should be pointed out that NONE of the (so-called) SEO/SEF mods have **ANY** effect on search engines. The *ONLY* effect they have is to make the URLs look "pretty" for the HUMANS viewing or linking to the site.
To tell all ya the truth:
Sitemaps aren't really needed, my forum doesn't gave on yet it's on Google, because SMF is friendly with search engines.
PrettyURLs and SimpleSEF stink, yes they do. I used them and my database increased a lot >.> after I removed them I didn't find any sort of decrease or anything.
Robot.txt isn't really needed unless you want bots to not come to your site.
The few SEO methods I know (only 1 to tell you the truth), requires access to a big database, dedicated server or VPS, and some coding. I personally never done it but I've seen a professional do this and was suprised by the traffic! Our sites are almost the same except his is a bit more referenced, anyways in about 5 months he got asmany members as mine and mine was 2,000, not to mention all those pageviews >.<
Anyways this is my two cents.
Quote from: Royalduke on June 01, 2010, 06:56:02 PM
Sitemaps aren't really needed, my forum doesn't gave on yet it's on Google, because SMF is friendly with search engines.
Sitemaps are good if you need to specifically indicate every URL you want indexed. If your pages are all linked already, it's not urgent.
Quote from: Royalduke on June 01, 2010, 06:56:02 PM
PrettyURLs and SimpleSEF stink, yes they do. I used them and my database increased a lot >.> after I removed them I didn't find any sort of decrease or anything.
URL rewriting will increase load, that's to be expected. Removing the rewriting stuff should have decreased load, however.
Quote from: Royalduke on June 01, 2010, 06:56:02 PM
Robot.txt isn't really needed unless you want bots to not come to your site.
Using special rules in robots.txt can help eliminate indexing of duplicate content pages to begin with, which should decrease load and bandwidth usage. For SMF 1.1, it can also help block duplicate content that didn't get added to the noindex list.
Quote from: Royalduke on June 01, 2010, 06:56:02 PM
The few SEO methods I know (only 1 to tell you the truth), requires access to a big database, dedicated server or VPS, and some coding. I personally never done it but I've seen a professional do this and was suprised by the traffic! Our sites are almost the same except his is a bit more referenced, anyways in about 5 months he got asmany members as mine and mine was 2,000, not to mention all those pageviews >.<
Most of the most effective SEO tactics can be done with any kind of hosting as they all deal with the content of things.
http://writingcenter4.info (Enjoy all things Harry Potter from Dark Harry, to romance and Femslash):D the webmaster is a proffesional and I know because I saw some of his old works and because I have a lot of contact with him.
Anyways, his forum is like 7 months old and he barely has any of the SEO stuff like sitemap and he's still using 2.0 RC2.
In these 7 months most of the content he added was automatically generated but manually add (it's more of a reference site). He has more content then me but not much user activity yet he managed to support a dedicated server for 5 or so months on user donations.
His pageviews and signups increased dramatically in the first few months and now it's bigger then mine. Last month the site got sponsered and the server got updated to Quand Core (I think) with 4 GB RAM and 500 GB of space.
Even though he never told me how he managed, a close friend explained and now I understand it. Even though I understand it, I doubt I can do anything like that...
PS: The prettyURLs mod increased my mySQL database because of all the saved stuff >.>
VB seo = 10/10
ipb seo = 10/10
smf seo = ? imho it's a big 0
smf knows this bt keep on releasing betas wt so many bugs in them.they are nt working on the imp stuff.
-meko
*headdesk*
There is nothing SEO worthy in vB, they provide the same stuff that SMF does but in SMF those features must be add on by mods.
meko,
what the heck are you talking about? First of all, what are you calling "seo", second, what are you using to compar 10 out of 10 or 0 out of 10?
Search engines scan, catalog and rank SMF content just fine with or without any of those silly pretty url or SEO/SEF mods.
And what do you consider to be "the important stuff"? because I feel that the work our current devs are doing is "important stuff"
i think the most important thing is the content itself,
all seo thingy will be useless if your content just copy and paste from another sites
am i correct :o
I think for making SEO friendly you want some fresh content and SEO methods that will make as smf forum as SEO Friendly.
yes thats true after all Content is King
content is really king...... Copy, modify and paste .... To avoid -10 ranking from google
Content is very important followed closely by quality relevant links. Search out other relevant forums and get natural links back via blog posts, forum posts etc.
I try to keep up with SEO as best as I can..it's been awhile since I read over this...
Google actually has a pretty big forum with help from some of Googles people with providing answers....
You gotta realise, search engines WANT your site listed. It's not like it used to be years ago. It makes their search engine more favorable. While they try to improve their search engines, they will tell you what gives them problems, or what works better with it...
Such things as conical urls......Google doesn't like identical pages of content and will drop one. So if your using www.you.com as your sites addy, and Google dropped it in favor you.com (no www and identical content), then obviously its ranking, www.you.com, is not going to be good...
You can do the redirect of www.you.com with a simple meta tag or use htaccess.
I guess I'm saying don't forget to try sites like Google with help for their search engines...
Quote from: damnitmike on August 28, 2010, 10:02:01 PM
Such things as conical urls......Google doesn't like identical pages of content and will drop one. So if your using www.you.com as your sites addy, and Google dropped it in favor you.com (no www and identical content), then obviously its ranking, www.you.com, is not going to be good...
If Google sees identical content on www and non-www, it understands and will pick whatever version has the most inbound links (I believe). You can change this using their webmaster tools.
SMF 2.0 supports canonical URLs which will help a ton with individual topic pages.
I believe you can also show this on your sites sitemap also.....
Thanks for the 2.0 info, I wasn't aware it supported the conical issues... ;)
ok... term FAIL... canonical...
next, www versus non-www does not make a difference.
Quoteok... term FAIL... canonical...
next, www versus non-www does not make a difference.
Thanks for the spelling correction Kindred! It was my 20th hr.....
Not sure what your saying, so I'll go on and make myself clearer, and you can add or take away....
This is from Googles Webmaster Help forums....
Quote============= Multiple Homepage URL Issues =============
This means that you have things like;
http://www.example.com
http://www.example.com/index.html
http://www.example.com/homepage.html
http://www.example.com/default.aspx
all showing the same content/loading the same file.
============= So why is this bad? =============
This means that you are showing the same content under numerous URLs.
Google tracks things by their URL.
That means all scores/ranks etc. are based on that 1 URL.
If you have the same content under multipel URLs - you may have your "scores" divided between those URLs, instead of consolidated to a single URL.
Google tries to avoid showing Duplication to it's users.
Searchers do not want to see tons of identical results in the SERPs.
Not a Penalty people, it just picks 1 out of X URLs and shows that.
The one it shows may/may not be your "best" or "strongest" URL - so your rankings may vary/fluctuate ... and if it picks a "weaker" URL - you may not rank at all.
Canonical issues can cause Duplication and result in a weaker site and poor rankings/performance.
Got it?
Good.
Here's the link....it is quite lengthy.
http://www.google.com/support/forum/p/Webmasters/thread?tid=227a384b1a309ba7&hl=en (http://www.google.com/support/forum/p/Webmasters/thread?tid=227a384b1a309ba7&hl=en)
They made a video to explain also....
http://www.google.com/support/webmasters/bin/answer.py?answer=139394 (http://www.google.com/support/webmasters/bin/answer.py?answer=139394)
I was too damn tired to find the links last nite....I got more if anyone is interested.
:)
That still has no application to SMF as version 2.0 actually supports canonical tagging (http://googlewebmastercentral.blogspot.com/2009/02/specify-your-canonical.html).
Also, Google is smart enough to understand about the www prefix.
Quote from: Deprecated on August 07, 2008, 01:33:46 PM
We have become focused on the topic of robots.txt files, so I would like to expand the discussion and get back to the original titled topic: Making your forum as SEO friendly as possible. Here are the steps I've taken so far:
I have followed your steps very carefully and today I have about 200 Guests in my forum plus the common 15-20 users active. I don't know if those are crawler-bots or "real users", but is good anyway I think.
Now I have a couple of questions:
1) In Google Webmaster it shows as crawl error this:
http://mysite/index.php?action=search
URL restricted by robots.txt
However, this is normally restricted in robots.txt, am I correct? Should I allow the robots access to this?
2) When I check "Who's online" I always find someone (I guess is a bot crawler) trying to "Report a topic to a moderator". This is because guests are allowed to report. How can I prevent crawlers of trying to index this?
Thanks!
Quote from: dsanchez on September 13, 2010, 07:04:40 AM
1) In Google Webmaster it shows as crawl error this:
http://mysite/index.php?action=search
URL restricted by robots.txt
However, this is normally restricted in robots.txt, am I correct? Should I allow the robots access to this?
It's a non-content page, so you can safely restrict it. In fact, it's recommended.
Quote from: dsanchez on September 13, 2010, 07:04:40 AM
2) When I check "Who's online" I always find someone (I guess is a bot crawler) trying to "Report a topic to a moderator". This is because guests are allowed to report. How can I prevent crawlers of trying to index this?
You could block the reporting action in robots.txt or remove the option for guests to report things.
Quote from: 青山 素子 on September 15, 2010, 01:31:51 AM
You could block the reporting action in robots.txt or remove the option for guests to report things.
I just did the second. Thanks!
Here are some useful SMF SEO Google Friendly (http://www.jerry-bell.com/search-engine-optimization-for-the-simple-machines-forum/) tips
Actually, that blog post is mostly useless....
URL does not matter.
Metatag description and keyords actually do not matter any more.
Global header and footers?? What use to seo??
H1 tags are properly handled in SMF 2
Your description of the robots.txt I outdated, since SMF 2 handles duplicate content correctly
Basically.. Pointless and misleading... :(
You are totally wrong by saying this.
Keywords in a URL do matter (http://www.youtube.com/watch?v=gRzMhlFZz9I=).
Meta descriptions does absolutely matter (see Google SEO start guide)
Keywords are useless for Google, maybe not for Yahoo! or Bing.
Global headers and footers mod puts extra content to the header or footer of the forum, helping Google to understand what the page is about.
You can block Googlebot from indexing unimportant pages such as ?area= ?wap2= and many others, as stated in that article which I linked to.
Am I right, or not?
re: URL - that is from 2009. It is outdated. None of the search engines care at all about the url structure any more and all of them parse argument style urls just fine these days.
Metatag descriptions ONLY matter for what the user sees as the "stub"/description in google... they do not matter for the search engine itself.
your "blog" indicates it it for improving GOOGLE SEO.... so why are you now mentioning yahoo or Bing?
However, the answer your question:
http://searchengineland.com/sorry-yahoo-you-do-index-the-meta-keywords-tag-27743
(read the end - basically, only if no other ranking criteria is present)
And who bothers to use Bing? It's useless.
Global Headers puts the SAME content on every page, thus diluting any page specific content. So, it's actually weakening your page content SEO.
SMF 2.0 already tells google that such content is duplicate by use of the
<meta content="noindex" name="robots">
and
<link href=".........." rel="canonical">
tags, which is more specific than robots.txt and is automatic.
So, basically... nope, you are not right.
QuoteSimple-to-understand URLs will convey
content information easily
URLs are displayed in search results
(1) A URL to a page on our baseball card site that a user might have a hard time
with.
(2) The highlighted words above could inform a user or search engine what the
target page is about before following the link.
(3) A user performs the query [baseball cards]. Our homepage appears as a result,
with the URL listed under the title and snippet.
Creating descriptive categories and filenames for the documents on
your website can not only help you keep your site better organized,
but it could also lead to better crawling of your documents by
search engines. Also, it can create easier, "friendlier" URLs for those
that want to link to your content. Visitors may be intimidated by
extremely long and cryptic URLs that contain few recognizable words.
Let's look what Google has to say about this. Please read page 6 and page 8 of the Search Engine Optimization Guide of Google,
which was released in NOV 2010: http://static.googleusercontent.com/external_content/untrusted_dlcp/www.google.com/en//webmasters/docs/search-engine-optimization-starter-guide.pdf
If you want URLs with words in them for HUMAN eyes, then fine, but do not think that it has any bearing at all on the search engines....
proper LINKS (i.e. title tag, etc) make a difference, but the URL really does not)
Okay, I believe you :D
Although it's also from 2009 (it'll be updated soon for 2011), the SEOmoz Search Engine Ranking Factors (http://www.seomoz.org/article/search-ranking-factors) section is a good starting guide for determining what counts most for a search engine. It's not just a single person's idea, it's based on a survey of a wide selection of SEO professionals.
If you look into the top ranking factors, you'll find that keywords in the URL is in #10 and 11 slots, with only about 37% importance, making it low importance. You'll also find the meta keywords item is in the very bottom slot, below even keywords in HTML comments.
The other thing I find utterly hilarious is the entire concept of search engine optimisation when applied to forums.
When you're in control of every element of the page, content and presentation, you can load it up with this week's fad of SEO if you like (note that Google is rolling out a major algorithm change anyway). But most importantly you can load it up with appropriately weighted, keyword rich content. Which is fantastic if SEO's your thing.
But on a forum, you can't control the content unless you're the one writing it and running it like a blog - your users create the content, and unless you sit and rewrite it post by post, you don't have the ability to control how they write it - which means you don't have the power to make it keyword rich.
Consequently, anything else you do is just tweaking, because the real meat is never being touched.
Playing about with your urls is not advisable unless you know what you are getting into. I used Pretty Urls mod for around a year. I now regret that choice a hell of a lot. Pretty urls does not a lot for forum seo, whats worse it creates far to many versions of the same page.
I have lots of links pointing to the pretty urls, but now I no longer use the mod all those links count for nothing. Anyone who knows seo knows that links are far more important than anything you can do on your site. Reverting back to the old urls is difficult as the reverter doesnt appear to function, and you must also consider what will happen if the mod developer decides to give up on the mod and not develop it for future smf releases.
Well, Pretty URLs' original developer mostly moved on, but made sure that anyone could continue to work on it by ensuring it has a non restrictive licence. And someone took it over.
2.0 handles the 'duplicate' problem a lot better with canonical tags all pointing back to the same 'page'.
I run about 30 SMF forums, and let me tell you Pretty URL is one of the most important things in SEO. It allows google to crawl every page and give it a unique path. can't live without it.
and, that statement, right there, shows how little you know about actual SEO.
Pretty URLs has ZERO, none, nada, NO effect on SEO in the least.
It may have mattered ten years ago... but with the explosion of php-based software that includes arguments in the URLs, Google and the others all made sure that they can crawl forum urls just fine... And every message and link in the forum already has a unique path... the message ID is unique per thread and message within a thread.
What matters most is good content, decent relevant links from other fellow websites.
There is some argument for SEO friendly URL's but Kindred is correct as Google for example can index almost anything these days. Build a good site with visitors in mind first and rankings will surely follow.
How pretty your URL's look is the least of your worries!
~RedOne
Actually, the only benefit I've observed from pretty URLs is directly in spite of it itself being added. What I've seen is that folks who start caring about SEO not only add things like sitemaps and pretty URLs but they start taking the time and energy to care about other things - including content. So it ends up being all part of the 'optimisation' process even though it doesn't make any practical difference in and of itself.
How more descriptive a link is, the better the chance that it will rank well in the search engines on relevant keywords of that particular topic.
Anyone got an idea how to add content to the homepage en subpages of a SMF forum?
Yes, that's often true, but how often do you have direct control of what your users post? (Which is why the entire concept of SEO on a forum is more than a little messed up)
QuoteAnyone got an idea how to add content to the homepage en subpages of a SMF forum?
Which home page? Using a portal?
No, on the normal homepage and on the sub pages (category pages of the boards)
I need to make sure every page has enough content on it.
The only thing I have now are the board descriptions.
That's the only content I have on the homepage.
Quote from: Arantor on March 18, 2011, 01:01:08 PM
Yes, that's often true, but how often do you have direct control of what your users post? (Which is why the entire concept of SEO on a forum is more than a little messed up)
How do you mean? Users can't change the URL of a topic. But if you are the forum-owner and you use descriptive URL's, search engines can better understand the content of your page and users can already see what the page is going to be about.
patrick, The URLs are, as I have already said, pointless.
Pretty urls are nice for people to look at, but the search engines don't care, at all...
The content on the page is what matters... and on a forum, in posts, you don't get to control the content.
If your site has things like a media gallery, articles, and static pages... then you have control. But over posts? The only thing you can really do is keep your users focused and get good posts.
Kindred,
You say > content is the only thing that matters on a forum.
Well, how do I get enough content on the homepage and category pages, then?
The only content I have on those pages is the description of the boards. Should this be enough for Google to recognize the subject of the forums and the subject of individual boards? Google recommends to have enough content on all of my pages.
forums are tough in that regard.
forum name, board names and board descriptions all help
for internal pages, like posts... make sure you have good posts... SEO is not something that happens overnight, especially with forums. With forums, you need to BUILD your content.
I have a question. Which file do I have to edit to add a nofollow atribute to the stats section of the page, which can be found under:
http://www.nameofyourforum.com/stats - thank you.
why do you need to do that?
I don't want to make that page findable by search engines in the search results and I don't want to flow pagerank to that section and I don't want to let Googlebot crawl this section.
I just put /stats in robots.txt
That should do it.
I fail to understand why... but use the permissions and turn off access to the stats for guests.
View forum statistics - make unchecked.
Quote from: glennk on February 07, 2011, 06:59:08 PM
Playing about with your urls is not advisable unless you know what you are getting into. I used Pretty Urls mod for around a year. I now regret that choice a hell of a lot. Pretty urls does not a lot for forum seo, whats worse it creates far to many versions of the same page.
I have lots of links pointing to the pretty urls, but now I no longer use the mod all those links count for nothing. Anyone who knows seo knows that links are far more important than anything you can do on your site. Reverting back to the old urls is difficult as the reverter doesnt appear to function, and you must also consider what will happen if the mod developer decides to give up on the mod and not develop it for future smf releases.
If the reverter package doesn't work anymore that's a shame. Maybe someone could update it? It's really a very simple little mod.
This was a really wonderful article. Thank you for your provided information.
probably a spammer... but definitely incorrect.
Although there MAY be some (small) worth to descriptive URLs, it really is not all that important in the scheme of things.
Once again... Content is king.
Sorry for digging up a old thread, but it is a good one!
I add my €0.02
note that my forum is in a subdomain so you might want to add /forum or so
My robots.txt targetting unwanted urls generated by SMF forum software:
User-agent: Googlebot
Disallow: /*?action
Disallow: *;wap
Disallow: *;wap2
Disallow: *;imode
Disallow: /index.php?www
Disallow: /?www
Disallow: /*msg
Disallow: /*sort
So no actions, means no SMF related (no content) links, wap/ imode no text based pages, the ?www should not become indexed, *msg makes Google forget anchored pages, eg, post #2, instead it will read the whole page and store it once, sort also blocked, took me a while to find, one can sort the topics, could appear as different content in Google.
Links I tested against in Webmaster Tools:
http://forum.yourdomain.xx/
http://forum.yourdomain.xx/index.php?topic=6706.0
http://forum.yourdomain.xx/index.php?topic=6576.15
http://forum.yourdomain.xx/index.php?board=2
http://forum.yourdomain.xx/index.php#2
http://forum.yourdomain.xx/index.php?board=2;wap2
http://forum.yourdomain.xx/index.php?board=2.0;sort=first_post
http://forum.yourdomain.xx/index.php?action=profile
http://forum.yourdomain.xx/index.php?www
http://forum.yourdomain.xx/?www
http://forum.yourdomain.xx/index.php?topic=6576.msg204437#msg204437
The test results of Google Webmaster Tools
Test results
Url Googlebot Googlebot-Mobile
http://forum.yourdomain.xx/
Allowed
Allowed
http://forum.yourdomain.xx/index.php?topic=6706.0
Allowed
Allowed
http://forum.yourdomain.xx/index.php?topic=6576.15
Allowed
Allowed
http://forum.yourdomain.xx/index.php?board=2
Allowed
Allowed
http://forum.yourdomain.xx/index.php#2
Allowed
Allowed
http://forum.yourdomain.xx/index.php?board=2;wap2
Blocked by line 5: Disallow: *;wap2
http://forum.yourdomain.xx/index.php?board=2.0;sort=first_post
Blocked by line 10: Disallow: /*sort
http://forum.yourdomain.xx/index.php?action=profile
Blocked by line 3: Disallow: /*?action
http://forum.yourdomain.xx/index.php?www
Blocked by line 7: Disallow: /index.php?www
http://forum.yourdomain.xx/?www
Blocked by line 8: Disallow: /?www
http://forum.yourdomain.xx/index.php?topic=6576.msg204437#msg204437
Blocked by line 9: Disallow: /*msg
---
I believe this is proper. Did I forget something? It is a 1.x SMF forum
Thanks!