SMF SEO Unfriendly?

[Eliana Tamerin]

In my opinion, Karlbenson has some of the best robots.txt ideas out there. Search for KB's posts about robots.txt.

[青山 素子]

In my opinion, Karlbenson has some of the best robots.txt ideas out there. Search for KB's posts about robots.txt.

Agreed. He's a done a lot of research on these things in relation to SMF.

[stealth645]

Thanks - I've got Karlbenson's robots.txt now...

-Steve

[ashsimmonds]

Hmmmm this is how your forum is indexed in google and its just like every other smf forum I've seen, all the pointless pages are indexed but NO post content.

Has anybody actualy seen a smf forum indexed by google?

About 5,000 pages on the forum alone.  Some of the useless stuff in there but I've just done a bunch of nofollow things etc to help stamp out pointless pages and duplicates.

http://www.google.com.au/search?q=site%3Aaussieexotics.com%2Fforum

[stealth645]

FYI.... 

Karlbenson's robots.txt help a lot!

[chadon]

If they are linked from other places your examples have no value for what we are discussing here.

But if they have no incoming links, aren't they also of no value? The only time they'll rank highly is if you search for what's essentially a unique term.

If we need to have all topics linked from other websites to have our forums crawled, then SMF has a very big problem. Links pointing to the home page should be enough to have all important pages indexed if the useless ones were not crawled. It can be done with the sitemap mod and others but not with the default configuration of SMF.
SMF is very unfriendly and when other forum softwares will become SEO friendly most people here will migrate their forum to better ones.

[Ben_S]

Use a proper set of robots.txt rules if it bothers you that much. Having the pages in the index is all well and good, but forum content without backlinks is all but useless as far as SERPS goes.

[chadon]

 I do have the robots.txt rules set and my forum is getting better since then but how many users know about it? Does it give any chance for new forums to get traffic?
I also have a question. Why is there links displayed to guests sending to unread posts with the on.gif and off.gif? All they get is a login page with different URL's. This is more duplicate content indexed resulting in penalties from search engines and don't tell me the links are useful for guests.
Duplicate content with login pages, duplicate content with help pages from all forums, that's a lot of chances to be at the bottom of the list in search results and to stay there.
The default configuration of SMF is not only SEO unfriendly but it also has major issues and no sticky or warning is displayed anywhere. Yes it can be fixed but the least that could be done is to let every one know and explain what to do if the developers ignore it.

[Ben_S]

I do have the robots.txt rules set and my forum is getting better since then but how many users know about it? Does it give any chance for new forums to get traffic?

All forum software suffers with duplicate content, SMF at least uses meta noindex tags on most of it these days.

I also have a question. Why is there links displayed to guests sending to unread posts with the on.gif and off.gif? All they get is a login page with different URL's.

Thats evidently an oversight, I'll stick it in the bugtracker, thanks for reporting it. - http://dev.simplemachines.org/mantis/view.php?id=2231

Duplicate content with login pages, duplicate content with help pages from all forums, that's a lot of chances to be at the bottom of the list in search results and to stay there.

Of course, it's not just SMF that has this issue.

The default configuration of SMF is not only SEO unfriendly but it also has major issues and no sticky or warning is displayed anywhere.

What major issues are these?

Yes it can be fixed but the least that could be done is to let every one know and explain what to do if the developers ignore it.

The developers aren't ignoring anything, point out the issues you see and they will be looked at.

[chadon]

All forum software suffers with duplicate content, SMF at least uses meta noindex tags on most of it these days.

I don't see any noindex tag in the "help pages" of the future version used by simplemachines. Do we need to have this unique content indexed?
Help page
Why do you have so many lines in the robots.txt file of your soccer forum? Probably to hide the problems that were not fixed in SMF 2.0.
It's probably true that most forum softwares suffer with duplicate content but why waiting for them to do something about it when it can be fixed easily here?

The developers aren't ignoring anything, point out the issues you see and they will be looked at.

I posted them in my last message and previous ones.
Duplicate content is a major issue and they come from the help pages and the login pages like the profile ones when guess access is disabled. This topic has been active for the last 4 months and I still don't see any changes done.
Edit
My forum receives much more visits by bots than by humans and before I setup rules in the robots.txt my VPS server was being shut down regularly because of the speed bots where visiting useless pages. I think this is a major issue too.

[Dannii]

Then make a mod to fix it.

[chadon]

 Here we go with the mods again. Even if I could write one, it wouldn't be the solution. SMF has problems and they should be fixed without the help of external mods that most forums wouldn't have or people knowing about them.

[metallica48423]

As the devs have already said... they know that there are things with SMF that make it SEO unfriendly.  As also already mentioned, there were some changes put into beta 4 that were intended to be the start of improving SEO.

The points being made are simply that SEO practices of software itself is only one part of SEO, which is very much so true.

I wish it were as simple as making a small change here and there, that'd be cool... but that change would also need to be tested live to see how it might affect the SEO of the thousands on thousands of other boards out there.  That testing takes time.  The research for those changes takes time.

[metallica48423]

If we need to have all topics linked from other websites to have our forums crawled, then SMF has a very big problem. Links pointing to the home page should be enough to have all important pages indexed if the useless ones were not crawled. It can be done with the sitemap mod and others but not with the default configuration of SMF.

Thats not what was said.  It isn't the end-all, but it is a major player in how google's crawling works.  Google originally ranked pages based on backlinks.  It is a lot more complex now, but a good portion of weighting works off backlinks... Also, how google does work and how google SHOULD work are two different things, unfortunately

crawling and indexing and ranking, though related, really are seperate things.  What i believe we're discussing here though is more the crawling aspect of things, in which case, yes, theres some work to be done in making sure crawlers catch the relevant pages.  I admit not having yet tried beta 4 (i have been wary of upgrading due to the template changes).  But I will see how it goes once I do upgrade

[青山 素子]

Why do you have so many lines in the robots.txt file of your soccer forum? Probably to hide the problems that were not fixed in SMF 2.0.

If you are talking about Dannii's site, it is running SMF 1.1. Also see my comments further down.

It's probably true that most forum softwares suffer with duplicate content but why waiting for them to do something about it when it can be fixed easily here?

I personally think you don't give Google enough credit. Three instances of the same text is duplication, thousands means it's part of a system. Much web software has similar text and interfaces, and if a lot of sites are using the software, there will be a lot of duplicated text. If you think Google and the other engines can't distinguish between actual duplication issues and common pages for software, you are mistaken.

The developers aren't ignoring anything, point out the issues you see and they will be looked at.

I posted them in my last message and previous ones.
Duplicate content is a major issue and they come from the help pages and the login pages like the profile ones when guess access is disabled. This topic has been active for the last 4 months and I still don't see any changes done.

See above. Also note that SMF 1.1 is only getting bug and security updates. Any new features that might cause a regression (and yes, many of your complaints refer to new features that would need adding) will not be put in a stable, released version. They are up for consideration in future versions where we can adequately test the effect before releasing it to everyone.

A decent amount of work has been put into the upcoming version 2.0 to assist search engines in what content to index. We have a few active beta testers who have put a lot of effort into making changes on their boards and studying how those changes affected what was indexed. As a result, many things have been cleaned up that will help a bit in indexing. One of the more interesting things out of the studies is that the noindex meta tag was not always obeyed. As a result, you sometimes have to resort to both a noindex tag and an entry in robots.txt to get a page out of the index and keep it out.

Here we go with the mods again. Even if I could write one, it wouldn't be the solution. SMF has problems and they should be fixed without the help of external mods that most forums wouldn't have or people knowing about them.

Being only half-informed on things and relying on conjecture for others does not indicate SMF has a problem. Rather, it indicates you have an incomplete understanding of things.

As for mods, the only way you'll get new features into older versions is via a mod. Development is frozen on the 1.0 and 1.1 branches - only bugfixes and security fixes are being done.

As for people not knowing about things, there seem to be a lot of people that know about SEO4SMF (although I don't like some of the stuff it does), so I am not sure why you think a mod to enhance what you feel are deficiencies in SEO will stay obscure. Right now, everyone seems to want SEO, but no one knows what it is, which means anyone who claims to be an expert is listened to and believed without pause. SEO is the current snake-oil of the internet, and everyone is buying.

If we need to have all topics linked from other websites to have our forums crawled, then SMF has a very big problem. Links pointing to the home page should be enough to have all important pages indexed if the useless ones were not crawled. It can be done with the sitemap mod and others but not with the default configuration of SMF.

Thats not what was said.  It isn't the end-all, but it is a major player in how google's crawling works.  Google originally ranked pages based on backlinks.  It is a lot more complex now, but a good portion of weighting works off backlinks... Also, how google does work and how google SHOULD work are two different things, unfortunately

Here is something in the style of a koan to think about that sums up that particular issue:


If a page is created on the Internet, but no one links it, does it actually exist?


Obviously the page exists, but think in the context of being indexed and placed in search results. Does it exist?

[chadon]

I wish it were as simple as making a small change here and there, that'd be cool...

It would take a few minutes to replace the text "News: Please assist us by completing our theme survey" on the top of all pages by something like "Bug fix" with a link to tips on how to set up your robots.txt file.
Now if you want to explain me why what I was talking about are not major issues, I would like to know about it.

crawling and indexing and ranking, though related, really are seperate things.

If there is problems with the crawling thing, then the two other things will be problems too

[chadon]

If you are talking about Dannii's site, it is running SMF 1.1. Also see my comments further down.

No, I was talking about Ben_S' website and it is running with SMF 2.0 Beta 4 like I said in my post

Being only half-informed on things and relying on conjecture for others does not indicate SMF has a problem. Rather, it indicates you have an incomplete understanding of things.

I know I have a lot to learn in this domain but if you want to quote me do it with the important things I said not what I could have said wrong.

Why don't you reply on this? Aren't they major issues?

I posted them in my last message and previous ones.
Duplicate content is a major issue and they come from the help pages and the login pages like the profile ones when guess access is disabled. This topic has been active for the last 4 months and I still don't see any changes done.
Edit
My forum receives much more visits by bots than by humans and before I setup rules in the robots.txt my VPS server was being shut down regularly because of the speed bots where visiting useless pages. I think this is a major issue too.

[chadon]

Other thing. I have duplicate pages indexed depending on topic numbers and I don't know where the links come from. How come the links below send to the same page content and not depending of the topic requested.
http://www.simplemachines.org/community/index.php?board=2.10
http://www.simplemachines.org/community/index.php?board=2.0

[Ben_S]

No, I was talking about Ben_S' website and it is running with SMF 2.0 Beta 4 like I said in my post

In the hope of keeping the bots from spidering a while load of pages for nothing, if it keeps them out of those pages, it saves me some bandwidth. Plus it exists from a time before 1.1 came about. IIR, it was me that suggested adding in the meta noindex tag.

Other thing. I have duplicate pages indexed depending on topic numbers and I don't know where the links come from. How come the links below send to the same page content and not depending of the topic requested.http://www.simplemachines.org/community/index.php?board=2.10
http://www.simplemachines.org/community/index.php?board=2.0

Presumably from users that have the post per page setting set to 10. Since you don't it works out what page you should be viewing to show roughly the same content as the user who used that link and perhaps shared it with you.

As you will notice, as 10 isn't the default here, SMF has helpfully added a meta noindex tag to the one that is effectively acting as a duplicate.

[chadon]

So what you are saying is that SMF 2.0 beta 4 has no SEO issues?
Too bad we have to wait forever to have this version out of beta. Liverpool FC Forum is nicely indexed by Google and it seems to have a lot of traffic. Can you tell me if the version has any bug? I'd rather hear it from someone who is not part of the simplemachines team