Do SMF posts appear in search engines? (Something to be concerned about)

Started by geezmo, September 06, 2006, 09:16:08 PM

Previous topic - Next topic

Stüldt Håjt

Quote from: eldʌkaː on January 03, 2007, 02:07:27 AM
Agreed, there is a problem. But can you define the problem and what causes it?

The biggest problem is definitely the default theme. Most are complaining about urls, but that is just a one thing in seo, still biggest issue for most users because everyone sees them. Most don't look "under the hood" where the problem lies. I'm not going into details what is wrong with the default theme, but that is something SMF team should start to improve asap.

This is the first thing where to start: <h1>forum name</h1><h2>topic name</h2><menu><posts>

Ben_S

The fact is, however, google has no problem indexing profile, it just stutters on actual topics. Yes, H1 tags would probably help with actual result positioning but the fact is topics just aren't being indexed well for whatever reason.

What is so different about topics and profiles, the only thing I can come up with is the amount of duplicate links to a topic (hopefully noindex will clear that one up) compaired with a single link to profiles.
Liverpool FC Forum with 14 million+ posts.

Stüldt Håjt

Quote from: Ben_S on January 03, 2007, 07:21:24 AM
What is so different about topics and profiles, the only thing I can come up with is the amount of duplicate links to a topic (hopefully noindex will clear that one up) compaired with a single link to profiles.

Probably the amount of links, yes, and probably the complexity of page structure. Spiders comes but it seems that they are having hard time getting results.

destalk

I've never seen any evidence that H tags help (or hinder) with search engine positioning or indexing.

I still think that the main issue with topics is less that there are duplicate links to them (Google will simply choose one URL over the other). I suspect that it has something to do with the content and structure of topics/threads.

Although I'm not technical enough to know whether the code structure is an issue, I commented earlier (or was it on another SE topic? There are about three on the go at the moment) that simple themes seem to work best. Forums seem to have so much repetitive content - by this I mean repetitive words and template structure - that I believe that Google sees many topics/threads as having duplicate content, or 'boilerplate' content. Although, so do Profiles for that matter, so I may be talking nonsense.   :-\

Having said that, I found that excluding Profiles from indexing has helped somewhat.

I also wonder whether the board list of links is seen by spiders as just an unimportant list of links, as opposed to an important content page. I know that when a thread is linked to from a static page on a web site, it is indexed much more quickly.

Ben_S

How else can topics be displayed, it's a (fairly) simple table, pretty much the same as all other forum software does it, if google can't handle that then google is seriously flawed IMHO.
Liverpool FC Forum with 14 million+ posts.

Dannii

The header tags give structure to web pages. Having a semantical default theme in a later version of SMF is something I definately want, and will probably give the biggest benefit to SEO of the various SEO ideas so far given.
"Never imagine yourself not to be otherwise than what it might appear to others that what you were or might have been was not otherwise than what you had been would have appeared to them to be otherwise."

destalk

Quote from: Ben_S on January 03, 2007, 07:49:52 AM
How else can topics be displayed, it's a (fairly) simple table, pretty much the same as all other forum software does it, if google can't handle that then google is seriously flawed IMHO.

I agree.

But I didn't mean that complexity of code was the issue. More a question of repetitive elements on the page. Some themes have so many 'template' elements on them that the actual 'unique' content can be hidden and perhaps appears to Google as automatically created 'boilerplate' pages? This is something that Google is currently making a big deal out of. Of course, I'm making an educated guess rather than having a lot of evidence. But this would make some sense as to why a simple theme may work better.

Of course SMF is no different to any other forum software in this area (or any other area for that matter). In my vB forums, I also tend to strip out as much of the unnecessary repetitive content as possible.

Something that appears to be coming up from a few people is that the boards of a forum are being indexed, but the threads are not (or are stuck in the supplemental index). I wonder why that might be? And why would profiles be so popular?

I can't claim to understand this very much.

Dannii

I think that's to be expected. Where are topics linked from? Their boards of course. But in busy forums those topics quickly get moved backwards and soon aren't shown on the first page of the board. The old links get lost, and new ones are formed, which isn't something I think would be good for SE rankings.
Profiles however get linked to from all over the forum, ever post + many other places. And those links don't change as often, so therefore they would rank higher I suppose.
"Never imagine yourself not to be otherwise than what it might appear to others that what you were or might have been was not otherwise than what you had been would have appeared to them to be otherwise."

Stüldt Håjt

Quote from: destalk on January 03, 2007, 08:05:44 AM
Something that appears to be coming up from a few people is that the boards of a forum are being indexed, but the threads are not (or are stuck in the supplemental index). I wonder why that might be? And why would profiles be so popular?

I can't claim to understand this very much.

Header tags would guide spiders to see important link from not so important links.

destalk

Quote from: Stüldt Håjt on January 03, 2007, 08:18:52 AM
Header tags would guide spiders to see important link from not so important links.

I don't believe that spiders see header tags as any more important than making links bold (assuming that they see that as important, which I doubt). In fact, putting links between header tags is quite unusual and might even be seen as an attempt to spam.

Ben_S

From what I've read, header tags (used correctly) can be quite an important factor, although I'm not sure it would help with actually getting topics archived.
Liverpool FC Forum with 14 million+ posts.

Dannii

Indeed, a semantic theme would help you get higher rankings for the pages that are being indexed, but it wouldn't help if the pages aren't even being visited.
"Never imagine yourself not to be otherwise than what it might appear to others that what you were or might have been was not otherwise than what you had been would have appeared to them to be otherwise."

Stüldt Håjt

Quote from: destalk on January 03, 2007, 08:43:24 AM
I don't believe that spiders see header tags as any more important than making links bold (assuming that they see that as important, which I doubt). In fact, putting links between header tags is quite unusual and might even be seen as an attempt to spam.

I don't think that's unusual. Actually I'll take a test and put some header tags to the default theme and see what happens.

Dannii

Headers shouldn't be used to given importance to a link, but to give structure to a page. However in certain situations it would be perfectly fine. If you had a header for each section, you might make it a link to itself so that you can easily bookmark the section.
"Never imagine yourself not to be otherwise than what it might appear to others that what you were or might have been was not otherwise than what you had been would have appeared to them to be otherwise."

destalk

I agree, using header tags correctly to structure the levels of information on a web page is a good idea. That's just good practice (although certainly not a requirement). But I have never read anything on an SEO forum, or reputable article, that supports the point of view that using header tags will have any bearing whatsoever on ranking a web site. At least not since around 1998 anyway. ;)

If it were that easy, spammers would have a field day and wouldn't bother wasting their time generating thousands of incoming links to their sites.

I wonder if we are in danger of grasping at straws here? For example, making changes to robots.txt files and then making assumptions about the 'results' after a few days or a few weeks. Or talking about meta tags and so on.

SMF is no worse at getting indexed by search engines than any other forum software or content management system Imo. If a web site is useful and has lots of incoming links, it will get indexed regardless of url format, meta tags, or whatever.

Forums (all of them) do have specific indexing problems because of the way they are structured (continually moving links, changing content and so on). In fact the only forum tool that I know of that doesn't appear to suffer from this problem is webmasterworld.com. And that is partially because it is so well known and linked to. But also because the forum has been written from the ground up to be SE friendly. It simulates a static directory structure, has nice urls and so on.

SMF developers have done all the right things with the noindex rules. In fact, these mirror what Google themselves recommend regarding duplicate links. I really think that there are only a couple of points that still really need addressing.

I think that eldʌkaː was probably right earlier on regarding the reasons why profiles are liked more than topics, so perhaps these are best disallowed in the robots file.

Other than that it may be worth trying to work out how the search engines get hold of the dynamic urls, when SE friendly urls are enabled.

But otherwise I would be concerned if major changes were made to the way that SMF forums are structured each time there was an upgrade, without good reason. Google gets suspicious about major changes to web sites and I have seen sites drop right down the rankings for a few months after a major redesign.

Phew, I'm exhausted with this subject now.  :P

青山 素子

Wow. A lot of conversation when I come back. At least this is all constructive.

I appreciate everybody who has been participating. Getting ideas formed is an important thing.

Quote from: destalk on January 03, 2007, 10:36:18 AM
I agree, using header tags correctly to structure the levels of information on a web page is a good idea. That's just good practice (although certainly not a requirement). But I have never read anything on an SEO forum, or reputable article, that supports the point of view that using header tags will have any bearing whatsoever on ranking a web site. At least not since around 1998 anyway. ;)

There is a difference between header tags and properly formatted code. Properly using header tags and other tagging is always helpful (if only to point the search engine to the content).


Quote from: destalk on January 03, 2007, 10:36:18 AM
But otherwise I would be concerned if major changes were made to the way that SMF forums are structured each time there was an upgrade, without good reason. Google gets suspicious about major changes to web sites and I have seen sites drop right down the rankings for a few months after a major redesign.

As far as I know, the developers always aim to keep compatibility and avoid changing the viewable side of things unless it is needed.
Motoko-chan
Director, Simple Machines

Note: Unless otherwise stated, my posts are not representative of any official position or opinion of Simple Machines.


Dannii

QuoteI agree, using header tags correctly to structure the levels of information on a web page is a good idea. That's just good practice (although certainly not a requirement). But I have never read anything on an SEO forum, or reputable article, that supports the point of view that using header tags will have any bearing whatsoever on ranking a web site. At least not since around 1998 anyway. ;)
http://www.sitepoint.com/article/most-common-seo-mistakes Look there it is at number 4 ;) And that article is less than 2 months old.

QuoteIf it were that easy, spammers would have a field day and wouldn't bother wasting their time generating thousands of incoming links to their sites.
Abuse of headers doesn't give any structure to a page any more than abuse of meta tags, or a title tag, or pretty much any abuse.

QuoteSMF is no worse at getting indexed by search engines than any other forum software or content management system Imo. If a web site is useful and has lots of incoming links, it will get indexed regardless of url format, meta tags, or whatever.
I agree, most people seem to forget that. Content is King.

QuoteOther than that it may be worth trying to work out how the search engines get hold of the dynamic urls, when SE friendly urls are enabled.
That would be good to know.. but I wouldn't be interested in investigating myself cause I've never used the inbuilt SE urls.
"Never imagine yourself not to be otherwise than what it might appear to others that what you were or might have been was not otherwise than what you had been would have appeared to them to be otherwise."

destalk

QuoteLook there it is at number 4  And that article is less than 2 months old.

Hmmmmm. But the key is that nowhere in the article does the author claim that the use of these tags has any effect on ranking, which was my point. Moreover, the fact that he points out that "correct heading hierarchies are among the most underutilized SEO tools" tends to reinforce the point that it is not really an SEO technique. In fact he seems to be talking about it in the same way that we are. I.E. that the correct use of H tags helps to build the structure of the site. From that point of view, it's certainly sensible and perhaps may add to accurate indexing.

QuoteThat would be good to know.. but I wouldn't be interested in investigating myself cause I've never used the inbuilt SE urls.

Fair enough. But perhaps it should be flagged as a somewhat 'broken' feature.

Stüldt Håjt

I finally made myself a sitemap that I gave to Yahoo and Google. It includes about 7000 urls and now Google and Yahoo both have over 5000 pages in their results.

Edit: And especially Yahoo have crawled my site like a maniac.

青山 素子

Quote from: Stüldt Håjt on January 04, 2007, 05:06:47 PM
Edit: And especially Yahoo have crawled my site like a maniac.

Yahoo has a habit of being an aggressive crawler, so much so as to cause many sites to have issues under the load of its spiders. If you see it is indexing your site, keep an eye on it.
Motoko-chan
Director, Simple Machines

Note: Unless otherwise stated, my posts are not representative of any official position or opinion of Simple Machines.


Advertisement: