Almost ready to give up on SMF

[dupont24]

One of the boards I run: http://forum.staticsubs.org/

I mentioned my increase in spidering in a different thread. I went from around 95 items on a Wednesday to 414 on a Saturday (about 110 pages added a day on average), to 600 on Monday,  then 681 the next day, where I am currently hovering around. The site has about 989 topics, with about 100 not viewable by any guests, making about 880 topics viewable by guests.

681 / 880 ~= 0.77386 * 100 ~= 77.4 percent.

And all this without any attempt at optimization, no sitemap, no dynamic meta tags (in fact, it is still set to the defaults).

As I said in that other thread:

With my site, we are actually competing against our old site (usually #1 on searches for the name). With the combination of moving to 1.1 final, adding more content to be indexed, and getting similar sites to link to our new site, things seem to be improving.

Also remember that Google is a moving target. They update their search algorithms somewhat often, and have in the past, dropped huge numbers of pages from their indexes. Moving to a different system might just gain pages in the index because they are new to the crawler, not because that software is necessarily better.

Not to be difficult but your forum says it has 14k posts....... google should be indexing all and its not on your forum as well......

Some seem to think I am bashing SMF.....  I am merely the messenger and some seem to want to shoot me...

Let me repeat it again.... SMF IS  IMO an easier and better BB than most i have used....I like it better and would have prferred to keep using it....

The non believers need to do some research ....The fellow that created the sitemaps.php for smf to help with the indexing ISSUE has not updated it in quite some time.....and I don't think he will be...he switched to Vb.

[青山 素子]

Posts are not the same thing as topics, and it is topics, not posts that are indexed (since the individual posts display on a topic page).

[dupont24]

Posts are not the same thing as topics, and it is topics, not posts that are indexed (since the individual posts display on a topic page).

Maybe thats what you are use to with smf....but that is not how most bb's get indexed.... Regardless your site is indexed not to bad.... but you are still missing tons....

Copy and paste out of some of your posts....some are indexed and many are not... try the same thing with different bb's and you will see a stark contrast...

If you believe your board is indexed as good or better than other bb's thats all that matters.

I believe and have seen the difference and it is to big to ignore....

[Toadmund]

In the last hour or two I have had two special guests on my forum, Mr.Googlebot and Mr.MSNbot, I find it funny they are considered guests!
However, before I updated to 1.1 I have never experienced the presence of their company, I am sure they will bring me bodies real soon!


So far 1.1 is awesome!

PS, just checked, some Yahoo is checking out my forum as we speak!

[Dannii]

I am running the out of box smf no special theme...nothing but the finished product.

I know that the theme is a problem, but that will be fixed in the next version I hope. However if you cared enough about search engines, you'd fix it yourself now.

[dupont24]

I am running the out of box smf no special theme...nothing but the finished product.

I know that the theme is a problem, but that will be fixed in the next version I hope. However if you cared enough about search engines, you'd fix it yourself now.

If its a problem and you know its a PROBLEM don't you think you should tell people that 10 or so posts back instead of this

I find it ironic that those who claim that SMF has bad SEO stuff don't seem to try very hard to optimise their forum.

So far you are the one that wants to shoot the messenger.....Do you think i am fabricating this?

Help would be nice......Whats the problem with the theme?.....and where do we fix it?

I used the duplicate content mod as well....to handle the multiple links.....so does that work?

I also have the archive installed with 2 links in the footer

BTW to those that have google landing on there site.... I have google there daily and slurp and msn and ask .......so for almost 5 weeks with them visiting daily don't you think its a little odd that indexing is minimal at best.....

I

[Dannii]

Its widely known that the current default theme has no semantical meaning. It would probably help quite a bit if it was. Then the spiders could use the page structure to find new pages, and they would understand the content more too.

[Toadmund]

Dupont24 said:

BTW to those that have google landing on there site.... I have google there daily and slurp and msn and ask .......so for almost 5 weeks with them visiting daily don't you think its a little odd that indexing is minimal at best.....

Well 1.1 has only been out for a week now, perhaps you have to give the search engines more time with the new version?

[Defiant]

The theme issue would explain alot for sure, using default here too

[青山 素子]

Not to be difficult but your forum says it has 14k posts....... google should be indexing all and its not on your forum as well......

All search engines index pages. As SMF doesn't have a feature to show individual posts on their own page, there is no way they could index individual posts. Considering the thread count and indexed count are fairly close, it seems that most of what can be indexed is.

The non believers need to do some research ....The fellow that created the sitemaps.php for smf to help with the indexing ISSUE has not updated it in quite some time.....and I don't think he will be...he switched to Vb.

Yes, it looks like he did. Do we know the exact reason why? Inferring that he did it over some SEO reason has no more credibility than saying he did it because (he thought) aliens told him to. Why? Because we don't know why.

[classixuk]

If I can just jump in on this thread please.
Hopefully I won't be shot as I too am a messenger that bears some bad news.

I must agree with dupont24 in that my SMF forum installation has been unsuccessful in getting indexed too. I moved over to SMF at the beginning of November and admittedly my site is not very big (only 2,500 posts), but not one of them is indexed by Google.

If you do a search for my site (hairdressingworld.com) ALL of my recent content (articles etc.) have been indexed, but none of the threads on my forum have been indexed bar the index page.
You will see lots of forum posts indexed by Google that are now supplemental, but they were all indexed while I was running my old forum software which was a basic flat-file perl forum.
I must admit that I am beginning to grow concerned about the suitability of using SMF for a site where the forum posts need to be indexed.

Each time the issue of poor indexing is brought up, people seem to get defensive here as if SMF is somehow being attacked and they must defend it. That really doesn't help.
People are telling you that with other forum softwares you can search for a sentence from a member's post and it will show up in Google as a result within a week or so of the original post being made. That's the truth.

For SMF to move forward as a serious contender to the likes of Vbulletin, IPB and even phpBB (I say that because phpBB gets indexed absolutely fine), then the developers need to "nail their colours firmly to their mast" so to speak.

Nobody in this thread has said SMF isn't a great board with great features. We all agree on that one. The problem highlighted by many, not just one or 2, is that the forum threads do not get indexed by the search engines in the same manner as every other bulletin board. That is a real problem.

I felt a bit sorry for dupont24 when I read a post slating him/her on their robots.txt file as if the shortcomings of SMF are his fault. Guys, I never once had to alter my robots.txt file to try and get my forum threads indexed with any other software. I don't think that some people on this thread realise that.

If you've only ever used SMF then you won't understand what it means to move from a forum that gets indexed within days to not seeing any threads indexed at all; you'll just presume that Google takes 6 months to index forums and even then, does not index more than say 50% of the content at best.

Well; that's not the case with the other forum softwares. You have new members joining up everyday because they found posts written last week on Google search engine results.

I think to solve this problem, the guys writing the software need to have a meeting of minds and decide whether they believe they do have a major problem with SMF and search engine indexing or not.
Then tell it to us straight.
As end-users we don't want to hear about "it could be this" or "it could be that", or even "have you done this to your robots file".

There can only be one of 2 answers:
1. "In our opinion we have a problem with the SMF code and this prevents certain search engines from indexing the content. We are working on this and should be able to provide a solution soon"
or 2. "In our opinion we do not have a problem with the SMF code and recommend you search the forum for SEO techniques"

The developers have not made it clear which camp they are standing in yet.

I believe that if you were to begin the conversation with the definitive answer, it would help a lot of webmasters decide which forum software they will feel comfortable running on their site.

To give you an idea of our frustration as end users, I went to the wordpress forums (seeing as they don't compete with SMF at all, and do not use any competing forum softwares) and found a post 6 days old. I pasted a sentence from that post into google and it showed up in the search results!
"I've uploaded files, setup mySQL, made changes to wp-config.php. Now when I try to install, I receive this message:"

I did the same with this forum, picking a post at random and got no search results!
"And you should be installing the SMF 1.1 final version"

Why can't SMF do what other forum softwares do?

[Kindred]

I'll say this again...  and I am not being defensive...   I don't understand your problems with being indexed.

I run 4 sites on SMF, all of them bridged to Joomla, which, by the statements of the people who complain, should be indexed even less than regular smf.
I do not have SEO turned on AT ALL.

My sites are nearly completely indexed, by thread.

As noted above, each individual POST is not indexed (and should not be, posts are duplicate content to the thread itself).

So, I guess I just do not understand what the problem is for all of you people who keep yelling that google, msn and/or yahoo are not indexing your sites.

[青山 素子]

The thing here, from all who are sharing their experience shows two things:

1. The majority of their threads are being indexed.
2. Almost nothing is being indexed.

The interesting thing is there have been no reports of any situation in-between (or I have missed them).

If you've only ever used SMF then you won't understand what it means to move from a forum that gets indexed within days to not seeing any threads indexed at all; you'll just presume that Google takes 6 months to index forums and even then, does not index more than say 50% of the content at best.

Oddly, after my upgrade from 1.1 RC3 to 1.1 final (and opening some more boards to guest viewing), I moved from just about 100 indexed items to over 400 in under two days, and just over 600 in an additional two. Calculating from my previous post puts this at about 77% of publicly viewable content.


The problem we are running into here is a few things.

1. We haven't settled on a common set of criteria for what good indexing is.
2. There seems to be a huge difference in indexing on various forums.

Now, dupont24 seems to feel that each and every post should be indexed. I won't disagree with him on that, although I will point out that since SMF doesn't have the built-in ability to put one post per page, that will never happen on a core SMF install. Indexing of threads is the most fine-grained you can mange without modifications.

What is troubling me the most is the differences in indexing behavior. My topics went up very quickly, while it seems other sites take forever. I do have my forum site set up on the Google Webmaster Tools to check things, so maybe that helps? I don't have a sitemap added though. The forum is also using some color variations of the Classic theme (not the newer Core). Perhaps that is also making a difference. Without guidance on how things are being processed (and let's not hold our breath waiting), we are forced to rely on conjecture.

No one here is saying that you are lying, or brushing you off. Your concerns are genuine. Most of us (perhaps not all) do, however, think there might be more causing your indexing problems than just using SMF. Certainly SMF can do better in indexing, but for a lot of people, it is doing really well currently. The problem now is to figure out what problems are SMF's and which are caused by outside forces.

[Toadmund]

I honestly think the issue now is confusion between the thread title, which is now being indexed in 1.1, and does show up in search engines, and the subject under the title which is not being indexed.

It almost forces one to describe in as few words as possible the exact subject matter in the thread title, so it can more exactly match the subject matter, which is useless if a thread goes off-topic.

[Defiant]

Its a shame you'd think all the clever people behind this forum could come up with something now instead of waiting for a new version.  This SMF forum has to be one of the best going but the bots thing is one of the most poor

[青山 素子]

I honestly think the issue now is confusion between the thread title, which is now being indexed in 1.1, and does show up in search engines, and the subject under the title which is not being indexed.

It almost forces one to describe in as few words as possible the exact subject matter in the thread title, so it can more exactly match the subject matter, which is useless if a thread goes off-topic.

The post text is being indexed, just not displayed (although you can do that by making dynamic meta description tags).

Its a shame you'd think all the clever people behind this forum could come up with something now instead of waiting for a new version.  This SMF forum has to be one of the best going but the bots thing is one of the most poor

Things are being worked on. Just because nothing is said doesn't mean work isn't getting done.

[Toadmund]

The post text is being indexed, just not displayed (although you can do that by making dynamic meta description tags).

How does one make dynamic meta tags, and will this make the thread subject matter show up in google? and not just the thread title?

This is how my google links look now:

Was the hippy movement a failure?
Discuss World Issues, Phenomenon, Ways to Reverse Disorder, Bicycles, and Heavy Metal! (Or Whatever else you can pound out!)***New Member Drive in Effect.
reversedisorder.com/forum/index.php?PHPSESSID=0b51a905d643adcf7536aa062e706098&topic=28.new - 35k - Supplemental Result - Cached - Similar pages

The thread title (underlined) is fine, what is not fine is that under every thread title listed I have the page description as written in my meta description tag, and not the thread title subject material.

I assume that when people work on SEO stuff it's hush-hush, the developer want's it to be secret, so as to SURPRISE the community.

Oh boy! Can't wait!

So far it's been good progress, it's just got to be tweaked a bit more. Something I am wondering though, in omitted results google is showing stuff like this:

Index of /forum/Themes/helios_multi/images/pod/red
... 02-Sep-2006 09:33 2k spacer.gif 02-Sep-2006 09:33 1k titlebg.gif 02-Sep-2006 09:33 1k. Apache/1.3.37 Server at www.reversedisorder.com Port 80.
www.reversedisorder.com/forum/Themes/helios_multi/images/pod/red/ - 13k - Supplemental Result - Cached - Similar pages

Is it really necessary, and would one consider it a waste of the bots time to index that, how do I tell the robot to ignore that stuff? Is that still what the problem is, bots are too busy and unprioritized?

[青山 素子]

The description tag is used if it is present, otherwise the page content is used. If you mess with the template, you should be able to get the first post's text as the description.

Turn off directory indexes (or put up a blank index page) if you don't want directory indexes being added. Search engines will index everything.

[Toadmund]

So, I should delete my meta description, the stuff between "".
Check.

However, I am unclear what you mean by this:

Turn off directory indexes (or put up a blank index page) if you don't want directory indexes being added.

[kegobeer]

You don't want a directory without an index.html or index.php file.  If you don't have that, a search engine can crawl the directory structure.