Simple Machines Community Forum

I've long been concerned about SMF posts not appearing in search engines. My SMF forum has been running for 6 months now and NOT EVEN ONE forum post is in Google. A lot of SMF defenders have, obviously, defended that the problem is not SMF's because SMF URLs are actually search engine-friendly, but the question is, do SMF posts really appear in search engines?

Let's compare posts from an SMF forum and a vBulletin forum:

http://www.simplemachines.org/community/index.php?topic=48256.0
Posted on September 5, 2005

QuoteMigrating Database from PHPBB2 to Simple Machines

Hello!

I *really* like this forums script, and since my World of Warcraft guild has been using PHPBB2 for a pretty long time now, I would really love to switch them to these forums but I don't want to loose all of the posts and member accounts.

Is there a way to migrate the database file from PhPBB to SimpleMachines so that people don't need to create new accounts, etc.? I know the structure is completely different and all, but I would really love to make the switch.

Please advise. Your help is greatly appreciated!

Here's a post on the same date in vBulletin.org, a site that uses vBulletin.

http://www.vbulletin.org/forum/showthread.php?t=95681
Posted on September 5, 2005

QuoteGetting an editor on moderated posts/threads in AdminCP

I asked this on vB.com and have been advised by Jake to ask here...   I have a few forum categories where my team gets to moderate the threads and posts made in them. However, quite often we need to edit these threads not just in terms of content, but in terms of the format of the post, and the fact that we feel they may not be appropriate to validate in the moderated forum but they would be fine in another forum.  It would be nice if there was a way of editing the format of a post/thread whilst within AdminCP Moderation area as well as ammending the forum category to where the post will show - instead of validating it first then editing it straight away.  Is there a way to accomplish this? How difficult would this be folks?  Thanks.

THE TEST: Search whether the threads will appear in Google.

Google Search for an SMF post: "Migrating Database from PHPBB2 to Simple Machines"
Search Results: NONE

Google Search for a vBulletin post: "Getting an editor on moderated posts/threads in AdminCP"
Search Results: 1

Try the same experiment in other posts and you will see that vBulletin posts appear in search engines but SMF posts sometimes DO NOT.

(SMF posts in 2003 and 2004 appear in Google but some latter posts in 2005 & 2006 do not which means something was changed in SMF that caused search engines not to properly spider SMF.)

I hope I'm wrong in this analysis. I like SMF a lot but if using it would make my site obsolete in the search engines, I'll have no other option but to use another forum software.

I'd like to know what others think.

I have a tiny site, and have lots of indexed topics: http://www.google.com.au/search?q=site:ftgforums.com&hl=en&lr=&start=0&sa=N

Have you tried the SMFSEO mod?

Being indexed as a site and having specific threads appear as SERP are two different things.

Let's take the example of your site. In your forum, you have this thread about the "Mind Static Device" posted on January 25, 2006: http://ftgforums.com/index.php?topic=1490.0

As a forum owner, you expect that whenever somebody searches for "Mind Static Device" in Google, that thread in your forum would come up as a search result right?

Now, let's try it in Google: http://www.google.com/search?hl=en&lr=&q=%22mind+static+device%22&btnG=Search

The "Mind Static Device" thread in your forum does not appear as a search result, and yet it was posted for more than 9 months already. (Your forum had a SERP for the Mind Static Device wiki page but we're talking about the thread itself that I pointed out above.)

That's my concern. When people make a search in Google, SMF posts don't usually come up as a result, as seen in the "Mind Static Device" thread example.

Isn't that something that us forum owners should be concerned about?

Yeah I have to say I've noticed this too.

On my site there are certain keywords that have been posted about that are fairly rare words and have been there for months. But they dont get picked up in a google search even if using the "site:" command.

As geezmo says it only seems to be SOME posts that don't get picked up.

Does someone know if this is a fixable problem?

I'm getting many support questions about how to make google index SMF files. A vBulletin website on the same server that has opened in the same time with same backlinks etc with a SMF forum, gets indexed very quickly on google. But the SMF doesn't get indexed.

www.turkvisit.com was the largest SMF forum of my country, (about 60k members and 500k posts) recently migrated to vBulletin, only due to not being indexed by google. He says he still wants to switch to SMF when the SEO mod is available for 1.1 line, or SMF itself finds a solution.

Good to know people are noticing this too. This is a serious issue for serious webmasters. You simply cannot afford to have a site that does not appear as a search result in Google.

Harzem, I'm thinking the same thing for my forum. If SMF really cannot solve this problem, I'd no doubt move to vBulletin. I'd rather pay a hefty price just to get more members and activity in my forum then just have earnings from ads to compensate for the cost rather than stick to a free forum which won't get me ad earnings because people searching for my site cannot find it in Google.

I still like SMF but as I said, this is a serious issue that should be addressed asap.

I can afford to not have a site that appears well in Google.

Quote from: eldʌkaː on September 07, 2006, 04:02:54 AM
I can afford to not have a site that appears well in Google.

I'm also personally never concerned about google, but there are webmasters that really concern. Also many forums are to give information about some issues and they want to be the solution when the issue is searched in google.

All I can say is that if you're not concerned with your site not ranking high in the search engines, it means you're merely a newbie webmaster or your site caters to only a few people. Ranking in search engines is a must if you want more traffic and more earnings.

I have noticed this for a long time.    I put my site up last January and have right at 10,000 posts.  I did a site search for my SMF forum and got 4, that's right, only 4 links to my forums.   And those weren't even posts, there were links to theme stuff.  (It's a site for dads, so it is important to get indexed by google and other search engines to help dads).   

I do want to say that I LOVE SMF.   I think it is a great piece of software, but this one bug/problem really has me thinking.   I want to continue using SMF because I love it that much.   Please help fix this problem..      Thanks for listening to me.

Soo. Instead of blaming SMF why don't you learn to be a webmaster and go and find what helps your site in Google.
First off make sure google has access to your site, and where it is allowed to crawl. Are you preventing Google from indexing your site?
Do you use a robots.txt file?

Go and checkout Googles webmaster tools and help yourself also.

https://www.google.com/webmasters/sitemaps/docs/en/about.html

I'm not saying I know any better. Or that your not doing it right. I just don't see why its SMF fault. I too would love to be up there but I fully expect I need to  have top content and the hits for it to be valid to be at the top in goggle.

I understand Poly.       My site is very well indexed by using the meta tags..  keywords and description.     I also see that all of the major search engines come to my site everyday, I even see the topics that they are viewing.    But, in the search engines they just aren't there..    I give it plenty of time before I check.   

I am not putting down SMF.  I LOVE SMF forums and I probably won't change software..   I just think that this needs to be addressed..  That's all.     Everything else in SMF is great and I couldn't be more happy.

Now my question is what part of SMF do you think is causing this to happen? How is acting differently than other forum software when it comes to search engines.

My site is well-indexed too. Google's spiderbot visits the site everyday, robots.txt file is set up properly, I use sitemaps, I've got decent traffic, my site's up for more than 6 months now, so what else needs to be done? It's not a problem with my site anymore, there's something in SMF that Google just doesn't like, that's why SMF posts normally don't appear as a SERP. Also, if this problem applies to almost everyone using SMF, the problem is not with the webmaster  or site anymore, it's in the forum software.

I too am not putting down SMF, but if it wants to stay on top of the competition, it needs to address this problem. I hate using phpbb but I more hate the fact that phpbb posts appear in Google but SMF posts do not! Aargh!

@vbgamer45, I already sent an email to Google's support. Hopefully, they'd reply and you can use that to tweak SMF's codes.

Quote from: geezmo on September 14, 2006, 01:36:16 PM
@vbgamer45, I already sent an email to Google's support. Hopefully, they'd reply and you can use that to tweak SMF's codes.

Thanks for doing that. Please inform me about their reply.

vbgamer45, I don't know the problem but it can be the individual message links that connect to the same page. Google doesn't like different links to the same content, but SMF has about 21 links to the same page.

site.com/?topic=123.0
site.com/?topic=123.msg1
site.com/?topic=123.msg2
site.com/?topic=123.msg3
site.com/?topic=123.msg4
site.com/?topic=123.msg5
...
site.com/?topic=123.msg19
site.com/?topic=123.msg20


These are all the same page, but different links. Google doesn't like it.
http://www.google.com/support/webmasters/bin/answer.py?answer=35769

Quote
Don't create multiple pages, subdomains, or domains with substantially duplicate content.

hmmm, I now do see what you mean, from the latest posts. Thanks for not jumping down my throat.

I have always just assumed its hard work getting listed often.  Like have a live site for a year or two before things get moving. I don't have a site map..its something I start to do but can't really be bothered with it, but I do think it is more important than keywords are now.

But you guys no more about software that if somethings not right, you will know it, I probably won't.
I have gone with the 'well theres a million forums out there and Im just a new one' attitude.

Watch this thread with interest.

Cheers

Ohh, and there is one thing that annoys me with peoples SMF sites and thats that you cannot view content until your a member. You practically can't go anywhere at a lot of SMF sites, I look at a lot to see what others are doing with theirs.

I know that Google don't like that for a start. You can have all the google listing you like, but if you can't read it, I assume they dump it. Same with Arcades, not even allowed to view them...places like that I don't hang around.
You must have noticed that..   >:(

Just a thought.

Quote from: HarzeM on September 14, 2006, 01:54:37 PM
SMF has about 21 links to the same page.

site.com/?topic=123.0
site.com/?topic=123.msg1
site.com/?topic=123.msg2
<>

vB does exactly the same thing.

Quote from: Polymath on September 14, 2006, 05:43:03 PM
Ohh, and there is one thing that annoys me with peoples SMF sites and thats that you cannot view content until your a member. You practically can't go anywhere at a lot of SMF sites, I look at a lot to see what others are doing with theirs.

I know that Google don't like that for a start. You can have all the google listing you like, but if you can't read it, I assume they dump it. Same with Arcades, not even allowed to view them...places like that I don't hang around.
You must have noticed that..   >:(

Just a thought.

Well an arcade hardly has useful unique content does it? And if you don't let guests browse your forums, then of course the pages won't appear in search engines.

Quote from: eldʌkaː on September 14, 2006, 09:49:18 PM


Well an arcade hardly has useful unique content does it? And if you don't let guests browse your forums, then of course the pages won't appear in search engines.

My point exactly

Don't know abiout you but Ive had hits from game searches for the game name.. and as for forums not appearing in search engines thats what I said, they won't appear. But if someone wants to appear I am informing them to remove the block on guests and not complain. I don't know, but even one board locked to guests may not be liked by google.

These are the links used in vB:

Look at this thread from vBulletin.com

http://www.vbulletin.com/forum/showthread.php?t=142051

The links of posts 1-4 are:

QuotePost 1: http://www.vbulletin.com/forum/showpost.php?p=884772&postcount=1
Post 2: http://www.vbulletin.com/forum/showpost.php?p=884784&postcount=2
Post 3: http://www.vbulletin.com/forum/showpost.php?p=884786&postcount=3
Post 4: http://www.vbulletin.com/forum/showpost.php?p=884798&postcount=4

The links use a variety of URLs (p=884772&postcount=1, p=884784&postcount=2, p=884786&postcount=3, p=884798&postcount=4) while SMF uses:

Quotesite.com/?topic=123.0
site.com/?topic=123.msg1
site.com/?topic=123.msg2

Google probably hates the variations in the URL for the same page: msg1, .msg2, .msg3 ....

Quote from: Ben_S on September 14, 2006, 08:09:58 PM

Quote from: HarzeM on September 14, 2006, 01:54:37 PM
SMF has about 21 links to the same page.

site.com/?topic=123.0
site.com/?topic=123.msg1
site.com/?topic=123.msg2
<>

vB does exactly the same thing.

No, at least I couldn't find any single duplicate link up to now.

Geezmo pointed that but slightly wrong. The duplicate links in vB are single post links that include different content. Try clicking on geezmo's links and you'll see what I mean.

At least we're going somewhere here. I hope the SMF coders are watching us here, cos this will definitely help solve the problem that's hindering SMF from being a forum that can beat vB. I hope Google would reply already to the question I sent them on why Google does not display SMF posts.

I agree. This is a real probleme. I have 2 boards, one is phpbb and the other is SMF. Phpbb is much more link by google.
for exemple:
- smf (361 483 messages) and only 3 250 présents in Google
- phpbb (372 601 messages) and only 54 211 présents in Google

Hello fellas, I made a modification that can really help you with this, the archive for SMF (http://www.webdigity.com/index.php/topic,2433.0.SMF+%26amp%3B+SEO+-+Archive+for+SMF.html).

You can sheck the results here : http://www.google.com/search?hl=en&q=site%3Awww.webdigity.com&btnG=Google+Search

Hi Nikolas, good to see you here. I've installed your Archive mod last week, let's see if this is really effective and when this would bring results.

In any case, I think this should not stop SMF developers from finding out why Google does not list SMF threads as search results. Much is really being lost when an SMF forum does not appear in Google.

I agree
i ran a phpnuke site  (yes i learned the hard way about it getting hacked also)
but all be dammed if i posted something the very next week it was in google no issues

then i switched my site over  *( after nuke was hacked) to smfand it ran for  a year or so
and the search engine stuff was never ever the same :(

although nuke got hacked i was slamming with hits and visitors...

I just re opened the site sunday with SMF again
there are few posts and it will grow again.. but i know the hits wont be the same

I run 2 sites now
handgunplace.com
and basslurereviews.com
both SMF

sample of what i mean also
http://www.google.com/search?hl=en&lr=&q=site%3Ahandgunplace.com&btnG=Search
>:(

Just discovered that Yahoo can properly index my forum and some of my forum posts appear as search results in Yahoo. Unfortunately, majority of net users use Google to search and that's where our forum posts fail to show up.

did anyone contacted google staff about it? what is theirs side of story ?

No reply to my email in Google after a month :( There's also a pending question in the Google Groups board about SMF posts not appearing in Google but no one still has replied to it...

I switched my bass site to Vb and vbseo and traffic has been great

Wait for some months and those vB posts will appear in Google. Not sure whether posts in your SMF forum will be there too by that time. :(

The only reason I can think of for topics not doing to well is the duplicate links to the same content, google supports wildcards in robots.txt so something like this may help.

User-agent: Googlebot
Disallow: /forum/*.msg*
Disallow: /forum/*sa=showPosts*
Disallow: /forum/*prev_next*
Disallow: /forum/*action=printpage*
Disallow: /forum/*action=recent*

Have you already tried that in your site, Ben_S?

Went to your Liverpool FC forum, congrats, that's a really huge forum. But still are you not concerned  that out of 2.3 million posts, only 1,000+ entries appear in Google search results, and most are not even threads or posts?

http://www.google.com/search?hl=en&q=site%3Aredandwhitekop.com%2Fforum&btnG=Google+Search

Google.com - site:redandwhitekop.com/forum

Results 1 - 10 of about 1,640 from redandwhitekop.com/forum for . (0.04 seconds)

Quote from: geezmo on October 26, 2006, 10:18:01 AM
Have you already tried that in your site, Ben_S?

I added it ~ 20 days ago, whether it helps or not will take a little while to show up.

The duplicate content issue is the only thing I can think of that would be causing topics not to index well, unless others have any thoughts on what may be causing it.

Quote from: HarzeM on September 14, 2006, 01:54:37 PM

Quote from: geezmo on September 14, 2006, 01:36:16 PM
@vbgamer45, I already sent an email to Google's support. Hopefully, they'd reply and you can use that to tweak SMF's codes.

Thanks for doing that. Please inform me about their reply.

vbgamer45, I don't know the problem but it can be the individual message links that connect to the same page. Google doesn't like different links to the same content, but SMF has about 21 links to the same page.

site.com/?topic=123.0
site.com/?topic=123.msg1
site.com/?topic=123.msg2
site.com/?topic=123.msg3
site.com/?topic=123.msg4
site.com/?topic=123.msg5
...
site.com/?topic=123.msg19
site.com/?topic=123.msg20


These are all the same page, but different links. Google doesn't like it.
http://www.google.com/support/webmasters/bin/answer.py?answer=35769

Quote
Don't create multiple pages, subdomains, or domains with substantially duplicate content.

I had also stated this in the first page.

vBulletin has links to posts exactly the same way, but not publicly. You only have those links in your emails. In regular surfing, these don't show up.

Maybe considerable for SMF ::)

Hopefully we'll see some good results soon. I'll add those codes to my robots.txt file too.

i hope we can find a solution to this soon.  i am currently redesigning my site to run SMF instead of phpBB.  i have a SEO for phpBB and every single one of my threads shows up in google.  its the word content on the threads that get me all my traffic, if i dont have google, i dont have traffic.

Oh dear, this is not cool. This google search for "Do SMF posts appear in search engines";

http://www.google.com/search?q=%22Do+SMF+posts+appear+in+search+engines%22 (http://www.google.com/search?q=%22Do+SMF+posts+appear+in+search+engines%22)

doesn't return any results*, and a search for "geezmo" returns only the profile and user statistics, no posts or threads.

:'(

*Update, now it does, one hit titled 'latest posts of dave'

Update two, this search;

http://www.google.com/search?&q=site%3Awww.simplemachines.org+geezmo&btnG=Search&meta= (http://www.google.com/search?&q=site%3Awww.simplemachines.org+geezmo&btnG=Search&meta=)

gives 400+ results, so there are posts making it to the serps, they just don't rank.

Google is funny. There are thousands of SMF forums using "Dilber MC" theme, and all have a small copyright text at the bottom right. When I searrch google for "Dilber MC", I don't get them. When searching via yahoo, I can see almost all of them. Google started to be too picky.

R-S_Doug, thanks buddy for the extra publicity by using me as example ;) Yup, I've mentioned that same problem in the first page of this thread. Very, very, very few posts and threads appear as a serp in Google, what appears normally are the profile and user statistics info in SMF.

HarzeM, I've posted the same observation in this thread. Some SMF posts appear in Yahoo but there's usually none or very few in Google. There's something in the Google search ranking engine that hates SMF posts. Unfortunately, more than 80% of internet users use Google to search so it's not that much value to appear in Yahoo or MSN or Altavista. I only need my SMF posts to rank high, or simply just appear as a serp, in Google, that's it.

Geezmo: I agree that SEO is a major webmaster responsibility. I think it ranks up somewhere under good content (#1), usability (#2), then acccesibility (#3), so maybe #4 ;)

If you have any suggestions on how to make SMF more SEO'd, we are all ears.

Hi groundup, I think right now we can only guess. Nobody knows how Google's search engine works exactly so we won't know why it dislikes SMF posts.

The other guys have made a few guesess, such as the duplicate content issue, and I just tried using the codes Ben_S gave for the robots.txt file. It will take months before we see any results.

I think one good example is webdigity's site. Posts in his SMF forum are appearing in Google because of his Archive mod and other modifications. Not sure though how long it took the forum for those to appear as SERPs. I've been using his archive mod for more than a month now. No results in Google yet, but I'm crossing my fingers.

Welp, I'm giving this a shot:
http://mods.simplemachines.org/index.php?mod=339 (http://mods.simplemachines.org/index.php?mod=339)
I hope it searches for the meat under the plastic wrap (topic catagories), at least when I 'view source', the meta tag content has the topic subject title (name of the meat).

ie Title: Deliver the Liver (which is in the meta tag)
Subject: Liver is awesome with onions on buttered bread! Liver has that velvity powdery yummyness whose flavour bursts out with onions, bread and butter, of course, some salt, pepper, garlic and soya sauce couldn't hurt!

Which I think is good (that it's in the meta tag, liver too).

Now, I just have to wait a few weeks to see what happens, knock on wood, or 'pound on keyboard' ;)

Here is what my archive looks like:
http://www.reversedisorder.com/forum/archive.php (http://www.reversedisorder.com/forum/archive.php)

Toadmund,
Please note that the archive mod at no 339 in the SMF mods list will not work with SEO friendly URLs, as it uses query strings. If you email Nikolas at webdigity, or register on his forum and pm him, he will send you the code for his archive mod, which will.
Doug.

I am doing er, have done just that R-S_Doug.
I hope my email made it, I had issues with thunderbird, think it's fixed now though.

I'm also new to SMF, and have been trying to find out why the forum wont show in Google search results. Its interesting that both Yahoo, and MSN seem to find it OK.

P.S. Iv'e also sent a request for the mod to Nikolas.

Quote from: geezmo on October 26, 2006, 10:18:01 AM
Google.com - site:redandwhitekop.com/forum

Results 1 - 10 of about 1,640 from redandwhitekop.com/forum for . (0.04 seconds)

Update, results initially dropped down to about 700 or so when the .msg pages etc were dropped but is now rising by about 300 results a day, currently on 3,580 so hopefully within a few months there will be a decent amount of topics listed.

SMF 1.1 Final will have a meta robots noindex tag on the .msg, print pages etc which should have the same effect as the robots.txt.

so, your editing of robots.txt file gives a results ben?

action=printpage

still indexing in SMF 1.1 Final

Guess we still need to add some of Ben_S suggestions to robots.txt

and i think that some wap version of smf are better index by google then version for internet browsers

Upgraded our live forum to 1.1 earlier today, and it seems Yahoo likes the change A LOT. If the insane amount of crawling results in better indexing remains to be seen though...

Yes like yahoo bot go crazy . in my forum there is 10 times more yahoo bots then before upgrading on 1.1.

Yahoo's bot just likes to spider sites. I've had it almost take down some sites before (spidering about 20 sites running heavy db-driven stuff on the same server = insane loads).

yes but so much yahoo bots at one time... it doesnt happen before, with 1.09 version...

I had over 100 Yahoo spiders last night and 80 now >_>

I get visitors from Yahoo, MSN, and Google.  However, most of my visitors come from Google for the query "forum posters".

I'm not sure how well this will format:


Google    4346      49.4 %    4352      48.2 %
MSN       1847      21 %      1888       20.9 %
Yahoo     1618     18.3 %    1642       18.1 %


The forum's been up and running (after a rather long period of downtime) for just about 2 months. The above is pulled from the AWStats script for the month of November - the only full month we've had with the new forum in use.

I took a trip to Google and specified the domain and then +reply to ensure it would only show me if it listed the subjects themselves:

http://www.google.com/search?hl=en&lr=&q=site%3Aforum.kgiii.info+%2Breply

(I'm pretty sure this isn't going to be nearly as helpful as people were hoping.)

Anyhow...

That listed 130 results which properly indicates the 130 topics (minus a few in a private section and a couple of new ones that haven't been indexed yet) that are available to it. Err... It has done this since I first installed SMF? The only changes that I've ever really made were to actually swap out the meta tags in there so that they represented the content better and, honestly, Google doesn't give any great weight to meta tags anyhow seeing as the entire online world decided to mis-use them.

I only have a PR of 4 the last time I looked, up from a 0 in the start of November.

I guess what I'm saying is that maybe you're trying too hard or doing something that you shouldn't be doing? I haven't done anything, there aren't any tips, tweaks, or major changes I've made. My results aren't spectacular but the forum was just re-created in October after having been down for quite a while so who knows what it will turn into?

Most of the traffic comes looking for stuff about IE7, Vista, or DVD/CD problems. Google traditionally ranks anything I ever touch really low while MSN/Live Search seems to love anything I do. I really don't have any great pearls of wisdom here - just that, well, I didn't make any great effort to get indexed or the likes.

Edited to fix up the code section for readability.

Hmm, just checked my stats on google webmaster, I now have around 700 pages listed, and am getting hits from many different google sites...Belgium, Canada, france, Germany, Hungary, India, Holland, Sweden UK, and the US. I made a sitmap using a tool that removes the sessionID crap from the url, I think that helped a lot. I also submitted the sitemap to Yahoo, and within a couple of days I have up to 67 spiders on my site. Googlebot visits more than once a day.

I am on 1.1, and decided to remove the SEO mod...because of the robots-noindex that was added in final. That had me worried at first, it looked like it would just tell all the robots not to index the page...I hope I am right. I also have the archive mod from Nikolas, with the link to it as the first link on the page...

Just checked yahoo, and they have indexed 291 pages (From the archive and the normal site), but I only submitted to them last week...

Quote from: Witte on December 12, 2006, 06:17:47 AM
I am on 1.1, and decided to remove the SEO mod...because of the robots-noindex that was added in final. That had me worried at first, it looked like it would just tell all the robots not to index the page...I hope I am right. I also have the archive mod from Nikolas, with the link to it as the first link on the page...

Actually, it does tell the search engine robots not to index the page, but only on pages that are using links that point to specific posts. This avoids the engine from indexing duplicate content which might have been causing Google to not index the forums. For main thread links and page links, things are normal.

More accurately, it tells the spider not to follow the link.

It's interesting reading this discussion. I have read similar discussions on vBulletin forums about how search engine unfriendly vBulletin is. In the scheme of things, SMF is pretty good out of the box. Although, I've noticed that some themes are less helpful than others when it comes to indexing and the use of a good robots.txt file is recommended.

QuoteSMF 1.1 Final will have a meta robots noindex tag on the .msg, print pages etc which should have the same effect as the robots.txt.

I wish this kind of stuff was optional. If there is one thing that Google does dislike, it is contantly changing 'behind the scenes' stuff. It has the problem of potentially looking like spam or 'artificial manipulation'. I have spent time getting my robots.txt file right and I'm not sure that I would like this to have to 'compete' with SMFs own set of noindex parameters. Is there a way to edit these noindex tags in SMF 1.1? Also, noindex works slightly differently from the way that a robots.txt disallow command file does.

Some thoughts on the duplicate links issue.

I'm no expert. But from what I've read, it seems that not getting pages indexed is rarely a duplicate link issue. Google simply decides to choose one set of content and relegates the rest to the supplemental index.

My main issue with SMF from a SE friendly and consistency point of view, is that even with SE friendly URLs enabled, it still gives out the dynamic urls to search engines. Using 301 redirects is the only way to keep consistency, but it's not an ideal solution.

Quote from: destalk on December 12, 2006, 02:02:13 PM
I'm no expert. But from what I've read, it seems that not getting pages indexed is rarely a duplicate link issue. Google simply decides to choose one set of content and relegates the rest to the supplemental index.

Perhaps not, but Ben_S actually did some trial on his rather large board with the changes and saw a huge increase in index count after making the change.

Quote from: destalk on December 12, 2006, 02:02:13 PM
My main issue with SMF from a SE friendly and consistency point of view, is that even with SE friendly URLs enabled, it still gives out the dynamic urls to search engines. Using 301 redirects is the only way to keep consistency, but it's not an ideal solution.

It shouldn't be doing that, but I don't have a public board that can do SEO links to play with right now and find the cause.

i just checked how many pages google indexed by my forum
http://www.google.com/search?q=site:farmaceuti.com+&hl=en&lr=&start=0&sa=N
the most of pages listed are page for login, registration and error pages who tells guests that they cannnot come in without registration....

You need to use a robots.txt to remove those pages.

Quote from: Ben_S on October 26, 2006, 08:49:00 AM
The only reason I can think of for topics not doing to well is the duplicate links to the same content, google supports wildcards in robots.txt so something like this may help.

Code Select Expand

User-agent: Googlebot
Disallow: /forum/*.msg*
Disallow: /forum/*sa=showPosts*
Disallow: /forum/*prev_next*
Disallow: /forum/*action=printpage*
Disallow: /forum/*action=recent*

I put this a 2 weeks ago in my forum's directory i hope it will help  ;)

Quote from: Motoko-chan on December 12, 2006, 04:27:53 PM

Quote from: destalk on December 12, 2006, 02:02:13 PM
I'm no expert. But from what I've read, it seems that not getting pages indexed is rarely a duplicate link issue. Google simply decides to choose one set of content and relegates the rest to the supplemental index.

Perhaps not, but Ben_S actually did some trial on his rather large board with the changes and saw a huge increase in index count after making the change.

Indeed. But it's impossible to be sure that one factor has affected the other. I had a forum that had only three pages indexed by Google for over a year. Suddenly, one day, it had several thousand pages indexed. Google is a funny beast. But I agree, it is best to try and reduce duplicate links as much as possible.

Quote from: destalk on December 12, 2006, 02:02:13 PM
My main issue with SMF from a SE friendly and consistency point of view, is that even with SE friendly URLs enabled, it still gives out the dynamic urls to search engines. Using 301 redirects is the only way to keep consistency, but it's not an ideal solution.

It shouldn't be doing that, but I don't have a public board that can do SEO links to play with right now and find the cause.

It is a strange one. But if you check the Google cache of SMF sites that use SE friendly URLS, and hover over the links, you will see that Google has indexed the original 'dynamic' version of the links. Interestingly, Google appears to index both sets of URL. Some search results appear as dynamic PHP urls and some as SE friendly. This is one reason why I leave the default urls on for any new forums that I start. I don't see any problem with the default SMF URLs generally, but it's easier to disallow using a robots.txt file if you use the SE URLs.

I originally thought it had something to do the the PHPSESSID issue. But that has been fixed since version RC2, so I wonder... I know that the drop down menus of SMF forums still use the dynamic URL, but I didn't think that search engines could index those.

If you don't want the meta noarchive tag, you can easilly remove it from the index.template.php file.

Thanks Ben_S.

I've checked, and destalk seems to be right - for some strange reason, my forum is indexed with a combination of the regular URLs and search-engine friendly URLs (although the majority of topics are indexed with the former), even though I have SE-friendly URLs enabled, and it displays correctly for members (with SE URLs exclusively).

I also thought that it had something to do with PHPSESSID, but apparently not as that's fixed but most topics are still listed in search engines with the regular URLs.

Anyone have any ideas why?

Quote from: Daniel Hofverberg on December 13, 2006, 07:46:05 AM
I've checked, and destalk seems to be right - for some strange reason, my forum is indexed with a combination of the regular URLs and search-engine friendly URLs (although the majority of topics are indexed with the former), even though I have SE-friendly URLs enabled, and it displays correctly for members (with SE URLs exclusively).

I also thought that it had something to do with PHPSESSID, but apparently not as that's fixed but most topics are still listed in search engines with the regular URLs.

Anyone have any ideas why?

How long have you had friendly URLs enabled? Google doesn't usually de-index a page unless it gets a 404, so if it indexed an old-style link, it will continue to keep it unless it has some reason to remove it.

Quote from: destalk on December 13, 2006, 06:21:05 AM
Indeed. But it's impossible to be sure that one factor has affected the other. I had a forum that had only three pages indexed by Google for over a year. Suddenly, one day, it had several thousand pages indexed. Google is a funny beast. But I agree, it is best to try and reduce duplicate links as much as possible.

This is why I hold that most SEO techniques are superstition, with the rest being mostly simple things that are just general good practice. Now, some of that superstition is based on behavior of certain search engines, but the problem then is that they often tweak the internals, nullifying the benefit of certain things (long before the use is dropped - friendly URLs anyone?).

My thought is to follow good practices, make use of tools and public hints from the engines (sitemaps are a great idea) and then sit back and see how things go. If your site is not being indexed well, something is wrong on the pages and you should figure out what it is and work on that.

In the latest case of SMF indexing, it was determined that multiple links to the same pages might be causing problems, as they seemed to fit to certain other behaviors that have been considered spammy. As such, these paths were marked to not index (a good behavior in general).

Quote from: Motoko-chan on December 13, 2006, 10:32:01 AM
How long have you had friendly URLs enabled? Google doesn't usually de-index a page unless it gets a 404, so if it indexed an old-style link, it will continue to keep it unless it has some reason to remove it.

I've had friendly URLs enabled pretty much since I first started the forum.

But as topics started long after I enabled friendly URLs are still listed with regular URLs (I e with index.php? etc), obviously something must be wrong that causes search engines to receive those URLs instead of the friendly ones.

Quote from: Daniel Hofverberg on December 13, 2006, 11:51:12 AM

Quote from: Motoko-chan on December 13, 2006, 10:32:01 AM
How long have you had friendly URLs enabled? Google doesn't usually de-index a page unless it gets a 404, so if it indexed an old-style link, it will continue to keep it unless it has some reason to remove it.

I've had friendly URLs enabled pretty much since I first started the forum.

But as topics started long after I enabled friendly URLs are still listed with regular URLs (I e with index.php? etc), obviously something must be wrong that causes search engines to receive those URLs instead of the friendly ones.

Yeah, that shouldn't be happening. I'm going to have to poke around when I get the time and see if I can replicate it to determine the cause.

QuoteThis is why I hold that most SEO techniques are superstition, with the rest being mostly simple things that are just general good practice.

Agreed. My view on this is that good SEO can be as much about about usability for human users, as for search engines. Having more than one URL pointing to the same place can be as confusing to humans when trying to add backlinks to a page, as it may be for search engines. Obviously, when developing an interactive site, that can be easier said than done.

QuoteIn the latest case of SMF indexing, it was determined that multiple links to the same pages might be causing problems, as they seemed to fit to certain other behaviors that have been considered spammy. As such, these paths were marked to not index (a good behavior in general).

Would you be able to list exactly what urls the noindex applies to? I'm not able to work that out, so that would be really helpful.

Also, if I want to rely on my robots.txt file instead, do I just stript out the line    <meta name="robots" content="noindex" />', ' from index.template.php. Or do I ned to stript out any of the other code?

Thanks.

Specifically, the previous and next topic links at the bottom of each page have that added, as well as any topic link that specifies a specific message.

It won't hurt to have both a robots.txt and the meta tag, but if you want to remove the tag, don't delete the whole line, just delete up to the "  ', '  " portion, leaving that intact. This is in all index.template.php files for 1.1, so if you are using a non-default theme, you will want to check it too.

Sorry, I'm a bit confused (it happens easily ;) ).

Quotedon't delete the whole line, just delete up to the "  ', '  " portion, leaving that intact

Could you please give me an example of what I should delete?

Thanks.

QuoteIt won't hurt to have both a robots.txt and the meta tag

True. The only thing is that I have had no issues with leaving the .msg messages for indexing, so far. Google simply chooses one version of the URL over the other and I'm quite a big believer in not fixing what isn't broken. If it all goes horribly wrong one day, I'll know what steps to take.   :-\

If I suddenly added the noindex, I might lose thousands of indexed URLs. Although it may only be temporary, I would not like to do this without implementing some kind of 301 redirect of the .msg urls to the 'root' urls. But that is far too complex for me.  :o

With the robots.txt I have simply excluded everything with 'action=' in it plus a couple of others, which does the trick.

Also, MSN and Yahoo/Slurp (http://www.ysearchblog.com/archives/000372.html) bots now accept wildcards, which makes everything so much easier (it used to only be Google).

If I want to disallow my 'login' page and my 'theme directory' pages from being indexed, what do I do?
Do I add this to the appropriate spot?
$context['robot_no_index'] = true;

If so, whereabouts do I insert this code?
For the second one I assume it's in '$themes_dir'?

destalk said:

QuoteWith the robots.txt I have simply excluded everything with 'action=' in it plus a couple of others, which does the trick.

More specifically, I would like to stop google and others from indexing 'actions', how do I go about doing that?

Toadmund, put *action=login* and /Themes/ in your robots.txt file.

# robots.txt generated at www.mcanerin.com
User-agent: *
Disallow: *action=login*
Disallow: /Themes/
Like this?
Then I stick it in my forum root?

Sorry to be so dense, but I am dense!

Yep, that should work fine.

Quote from: Toadmund on December 17, 2006, 10:14:02 PM

Code Select Expand

# robots.txt generated at www.mcanerin.com
User-agent: *
Disallow: *action=login*
Disallow: /Themes/
Like this?
Then I stick it in my forum root?

Sorry to be so dense, but I am dense!

You may want to specifiy the wildcard rules for each of the big three search eingines only, Google, Yahoo and MSN. They are the only ones that understand wildcards (*). Other smaller serach engines bots will just get confused or ignore wildcards, as widlcards are not part of the robots.txt standard.

Also, there have been issues with Googlebot defaulting to the rules specified in "User-agent: Googlebot", as oppopsed to User-agent: *

So it may be worth repeating any general rules for each specific bot.

So something like this may be more useful.

User-agent: *
Disallow: /Themes/
Disallow: /community/index.php?action=login

User-agent: Googlebot
Disallow: *action=login*
Disallow: /Themes/

User-agent: MSNBot
Disallow: *action=login*
Disallow: /Themes/

User-agent: Slurp
Disallow: *action=login*
Disallow: /Themes/



So I just put all that on notepad in a .txt file, and then shove it into my forum root.

Thanks! :D
I'll do it when I get up tomorrow.

Quote from: Toadmund on December 18, 2006, 12:52:53 AM
So I just put all that on notepad in a .txt file, and then shove it into my forum root.

Thanks! :D
I'll do it when I get up tomorrow.

Well... something like that. I was just giving an example. You may want to customise it for your particular forum. For example, if your forum is in a directory called community, then it would look something like

Disallow: /community/Themes/

You really want to get this kind of thing right for your site, otherwise you can end up excluding files that you don't want or including pages that you didn't want indexing.

A good way to check if your robots.txt file is properly configured is to get a Google site maps account. They have an analysis tool which will let you type in some urls to see if the robots rules that you have set are functioning properly.

I looked through this whole post and maybe I missed it, but it seemed to me that while we have mentioned google sitemaps, nobody said:

Quote"Go here and get the SMF plugin to build google sitemaps."

Maybe this is old news to everyone, but I didn't see it mentioned clearly here.

• Google has a technology to build sitemaps. Sitemaps are XML files that list every page you have, with options about how often they might change so the robot can come back to check.
• Many CMSs - WP and others - have plugins that build google sitemaps, and then ping google to suck them up.
• The information can be easily found on the google webmasters pages.
• If SMF doens't have a google sitemap builder, then it should - IMHO. An optional plugin that those of us concerned about page traffic can use to put every topic and reply into Google.
• Without a sitemap, Google does not search all the pages of a CMS site.

I found all this out by searching google when a client of mine wanted to assure that she had more traffic for the WordPress site that I built for her.

fwiw, and sorry if I've repeated things, but I didn't see it in the 6 pages.

ww

There is a new mod out called SEO4SMF (http://mods.simplemachines.org/index.php?mod=518) that is compatible with 1.1. It currently isn't approved for the official site yet, but will hopefully be soon.

QuoteWithout a sitemap, Google does not search all the pages of a CMS site.

This is not true. A sitemap just helps Google find your pages faster. If it thinks your site is interesting enough it will still index everything.

OK, so here is what I put into my forum root, root being 'forum':
User-agent: *
Disallow: /forum/Themes/
Disallow: /forum/index.php?action=login

User-agent: Googlebot
Disallow: *action=login*
Disallow: /forum/Themes/

User-agent: MSNBot
Disallow: *action=login*
Disallow: /forum/Themes/

User-agent: Slurp
Disallow: *action=login*
Disallow: /forum/Themes/

Is that fine?

And I would have to concur with windyweather, it asks for a sightmap (goog webmaster tools), but how does one make one?

You can use your forum's rss feeds. That's all I use.

You don't actually have to make a sitemap to use the Google robots.txt tool (in fact many people think don't like them). Just sign up for an account and type in various urls to check that the robots.txt file is working as it should (for Google anyway).

Quote from: Toadmund on December 18, 2006, 11:56:40 PM
OK, so here is what I put into my forum root, root being 'forum'

The robots.txt file needs to go into the web site root, not the root folder of the forum. I.E. www. yourdomain.com/index.txt

That's why I was being so clear as to where i put it, so someone may correct me.
I was wondering where exactly it goes.
Thanks!

For those interested in duplicate content issues. Here is a brand new blog post from the horses mouth;

Google's view of Duplicate Content (http://googlewebmastercentral.blogspot.com/2006/12/deftly-dealing-with-duplicate-content.html).

Now that 1.1 has been out for a while, anyone see Google indexing more of their pages?

Yes, but still inadequate, it only index's as far as the title of the post, therefore all that is indexed under the link is what I had in my meta tag. And since all the links in google are identical (all but title) they are relegated to the omitted results department.

I went and deleted my meta tag, and put in a robots.txt tag, now I wait......

Not sure if I posted in this thread or not, but I went from about 95 items indexed to just over 600 in under a week after moving from RC3 to final (I did open up some more boards as well, but the older setup should have still had over 100 threads). I just checked and see 813 items indexed now.

Couple weeks ago i have about 3000 links in google but now (https://www.simplemachines.org/community/proxy.php?request=http%3A%2F%2Fimg165.imageshack.us%2Fimg165%2F8148%2Fgooglefz3.th.jpg&hash=3aae30cc548b140c8561576fc9ed2fed44ee7924) (http://img165.imageshack.us/my.php?image=googlefz3.jpg)
its maybe because of robots.txt file or Nicolas' archive script , i don know... :wink:

I just got Nick's SMF archive script this morning.  Really cool guy.  Let's hope it works for me.

i put thid robots.txt file
User-agent: Googlebot
Disallow: /forum/*.msg*
Disallow: /forum/*sa=showPosts*
Disallow: /forum/*prev_next*
Disallow: /forum/*action=printpage*
Disallow: /forum/*action=recent*
User-agent: *
Disallow: /Themes/
Disallow: /community/index.php?action=login

User-agent: Googlebot
Disallow: *action=login*
Disallow: /Themes/

User-agent: MSNBot
Disallow: *action=login*
Disallow: /Themes/

User-agent: Slurp
Disallow: *action=login*
Disallow: /Themes/

and i got this result from google sitemap site:
(https://www.simplemachines.org/community/proxy.php?request=http%3A%2F%2Fimg227.imageshack.us%2Fimg227%2F8754%2Fgsimdt5.th.jpg&hash=0ceaafe555df58c8d10da6fdceaa5c6ccb625cde) (http://img227.imageshack.us/my.php?image=gsimdt5.jpg)
Why are there so many restricted pages?

Because these lines block them:
User-agent: Googlebot
Disallow: /forum/*.msg*
Disallow: /forum/*sa=showPosts*
Disallow: /forum/*prev_next*
Disallow: /forum/*action=printpage*
Disallow: /forum/*action=recent*

Thats good. They're supposed to be blocked.

yes, but lot of messages and posts which could be indexed are not because of robots.txt file.
i suppose that problme is this line "Disallow: /forum/*.msg*". what that line exactly do with the posts?

It blocks the links to individual posts to reduce duplicate content. If you're running 1.1 you can remove that line.

yes, im run 1.1.1. and i will remove because with that line google bot cant get to all posts in certain topics.

If you remove it, it will just be not indexed by the meta tag that tells it not to index it and you will be wasting the bandwidth on google bothering to visit it in the first place (assuming it doesn't bothered reading it because of the robots.txt rule - it may anyway).

The .msg links are just duplicated links that are already indexed by the proper link to it, e.g. index.php?topic=3424 and index.php?topic=3424.40 etc.

aha, ok, i thought that google doesnt indexed that post at all.
Thank u Ben_S.

I see in the default index.template that

Quote<meta name="robots" content="noindex" />

Is that good for SEO? Cuz I always thought it was good to index.

It's only in pages that are duplicate content.

Google Officially Address Duplicate Content for Forums (http://www.forumtrends.com/forum-seo/google-officially-addresses-duplicate-content-for-forums.htm)

Quote from: Isaac on December 28, 2006, 12:18:14 AM
Google Officially Address Duplicate Content for Forums (http://www.forumtrends.com/forum-seo/google-officially-addresses-duplicate-content-for-forums.htm)

Oddly, when one of our testers did something similar to the fix we added in 1.1 final their indexed count shot up. I had similar results when moving from 1.1 RC3 (before this) to the final 1.1 for one of my boards.

I tested my forum (SMF 1.1.1) with this tool: http://www.linkvendor.com/seo-tools/se-spider.html
And at the inbound links I see only such links like "index.php?PHPSESSID=9ed92b606c1c0053772a8f601d5919ea&board=42.0" or "index.php?PHPSESSID=c966a4ff67834db2d7be410517b56320&topic=1475.0"

Maybe is "PHPSESSID=something&" the problem!?  ???

For several months now, I've been using Nikolas's SMF Archive mod and have put the following in my robots.txt:

Quote
Disallow: /forum/index.php?referrerid*
Disallow: /forum/index.php?action=calendar
Disallow: /forum/index.php?action=profile*
Disallow: /forum/index.php?action=help
Disallow: /forum/index.php?action=search
Disallow: /forum/index.php?action=search*
Disallow: /forum/index.php?action=register
Disallow: /forum/index.php?action=login

I was satisfied to see that Google has indexed more than 10,000 entries from my forum, almost equivalent to the number of topics.

But surprise! I upgraded the forum from 1.1 RC3 to 1.1 and voila! the Google entries from my site have dropped to 200. I'm sure the cause was the upgrade because I never changed any settings in the forum or modified robots.txt and have kept the sitemaps intact for months now. Also, the drop in Google SERPs occured 3 days after I have upgraded the forum to 1.1.

Any ideas why this happened? One week after the upgrade, the Google SERPs for my forum is still 200.

Quote from: pushkin22 on December 31, 2006, 05:47:45 AM
I tested my forum (SMF 1.1.1) with this tool: http://www.linkvendor.com/seo-tools/se-spider.html
And at the inbound links I see only such links like "index.php?PHPSESSID=9ed92b606c1c0053772a8f601d5919ea&board=42.0" or "index.php?PHPSESSID=c966a4ff67834db2d7be410517b56320&topic=1475.0"

Maybe is "PHPSESSID=something&" the problem!?  ???

Since RC2 SMF will not show PHPSESSID urls to spiders. I guess that the SEO Tools spider is not recognised as a spider.  The best way to see what Google is seeing is to check the cached pages.

now, there is over 8000 pages of my forum indexed by google although i restricted near 5000 posts by editing robots.txt file. :)

Geezmo, my forum currently has 454 pages indexed in Google.  Yesterday, I changed my robots.txt file to what you posted above.  It'll be interesting to see the results.

If you have mkportal installed you have to turn off the "search engine friendly urls" in admin :(

I've just started using SMF, but have a tip for a good sitemap maker ... gsitecrawler.com

When you use it, use the Wizard to add your site by clicking on the "Add New Project" button ... it will step you through setting it up to crawl your site.

In the "Filter" section, you list any part of a URL you want to have excluded from the sitemap file.  "action=", "Themes", "sort=" and a few others are in mine ... like "login.php", my phpAdsNew directory, my cgi-bin directory, etc. 

The way I built my list of exclusions was to have the spider crawl my site and look at the URLS.  I wanted to get to the point where only URLs with the format "index.php?board=10.0" and "index.php?topic=23.0" were listed.

It will take a while to crawl your site because it pauses periodically to reduce the load on the server.  But once you have it set up, you simply "Recrawl" the site, review the URL list, and then have the program FTP the sitemap.xml file to your site and ping Google that it's there.  It also creates Yahoo style "urllist.txt" files as well.

How i missed this topic? Many of my friends sugges to use vB, because of this reason! Let's see what happens n next version! It will be good, project managers look here

Quote from: nitins60 on January 02, 2007, 03:05:40 AM
How i missed this topic? Many of my friends sugges to use vB, because of this reason! Let's see what happens n next version! It will be good, project managers look here

Are you asking the SMF Project Managers to read through this? If so, they are aware of it. Development doesn't stop and any good info here will be considered. (This is how some of the changes for SEO were added to 1.1 Final.)

Good to hear that. When I first posted this issue last year, a few "Simple Machines Heroes" criticized me and some other people for being too interested in having our sites appear in Google. They even said they don't care whether their sites appear in Google and it doesn't matter if their sites don't come up on the first pages of Google searches. Anyway that's been a long time, just glad that finally SMF has realized that search engine optimization is a must these days, unless of course you're not concerned with getting more customers or more hits.

Quote from: geezmo on January 02, 2007, 07:13:00 PM
Good to hear that. When I first posted this issue last year, a few "Simple Machines Heroes" criticized me and some other people for being too interested in having our sites appear in Google. They even said they don't care whether their sites appear in Google and it doesn't matter if their sites don't come up on the first pages of Google searches. Anyway that's been a long time, just glad that finally SMF has realized that search engine optimization is a must these days, unless of course you're not concerned with getting more customers or more hits.

The "Hero" part is just a postcount title. The thing to look for are the badges of team members.

SEO is one of those love/hate things. If we can do a bit to help and it doesn't affect the software's performance, it will likely be done. If it is something that will affect performance it will likely not be done. (things like auto-generation of sitemaps and stuff would affect performance, so that will usually be left for mods to take care of).

If you see anything that can improve SMF in any way, don't be afraid to mention it (even if it meets with criticism). The more participation and opinions that are given, the better SMF will be, and the team will also know where the community wants SMF to evolve.

Good to hear that Motoko-Chan. That kind of attitude will make SMF a very competitive software in the future. A lot of us SMF users are on the verge of moving to vB just because of this SEO issue but because it's now being addressed, I think we can stay on. Thanks again.

Anyway, I'd just like to ask again my concern. Do you know what caused my Google SERPs to decrease right after I upgraded to 1.1?

Quote from: geezmo on December 31, 2006, 02:43:09 PM
For several months now, I've been using Nikolas's SMF Archive mod and have put the following in my robots.txt:

Quote
Disallow: /forum/index.php?referrerid*
Disallow: /forum/index.php?action=calendar
Disallow: /forum/index.php?action=profile*
Disallow: /forum/index.php?action=help
Disallow: /forum/index.php?action=search
Disallow: /forum/index.php?action=search*
Disallow: /forum/index.php?action=register
Disallow: /forum/index.php?action=login

I was satisfied to see that Google has indexed more than 10,000 entries from my forum, almost equivalent to the number of topics.

But surprise! I upgraded the forum from 1.1 RC3 to 1.1 and voila! the Google entries from my site have dropped to 200. I'm sure the cause was the upgrade because I never changed any settings in the forum or modified robots.txt and have kept the sitemaps intact for months now. Also, the drop in Google SERPs occured 3 days after I have upgraded the forum to 1.1.

Any ideas why this happened? One week after the upgrade, the Google SERPs for my forum is still 200.

Quote from: geezmo on January 02, 2007, 07:34:24 PM
Good to hear that Motoko-Chan. That kind of attitude will make SMF a very competitive software in the future. A lot of us SMF users are on the verge of moving to vB just because of this SEO issue but because it's now being addressed, I think we can stay on. Thanks again.

I hear there are also topics on vB sucking at SEO on their boards (haven't checked personally), so my guess is the problems are shared across everything.

Quote from: geezmo on January 02, 2007, 07:34:24 PM
Anyway, I'd just like to ask again my concern. Do you know what caused my Google SERPs to decrease right after I upgraded to 1.1?

It might be related to the noindex we are adding to certain URLs to avoid duplicate content being indexed. Then again, it might be because Google felt like it, they don't exactly give info out like that. Has your count increased? You might want to look into their webmaster tools (http://www.google.com/webmasters/) and check what they have on your site through it, it can be very informative (including telling if they are having problems indexing pages).

I took a bit of this, and a bit of that just mentioned, and now I have this:
User-agent: *
Disallow: /forum/Themes/
Disallow: /forum/index.php?action=login
Disallow: /forum/index.php?referrerid*
Disallow: /forum/index.php?action=calendar
Disallow: /forum/index.php?action=profile*
Disallow: /forum/index.php?action=help
Disallow: /forum/index.php?action=search
Disallow: /forum/index.php?action=search*
Disallow: /forum/index.php?action=register

User-agent: Googlebot
Disallow: *action=login*
Disallow: /forum/Themes/

User-agent: MSNBot
Disallow: *action=login*
Disallow: /forum/Themes/

User-agent: Slurp
Disallow: *action=login*
Disallow: /forum/Themes/

That look OK?


From google webmaster tools:

QuoteHome page crawl:     
Googlebot last successfully accessed your home page on Dec 24, 2006.

THat's good 'cause just yesterday it said Sept. 9th was the last crawl date, finally updated a bit it seems.
But I have yet to see anything updated, no new posts and no evidence of robots.txt kicking in yet, C'mon google, speed 'er up!

Quote from: Motoko-chan on January 02, 2007, 07:47:25 PM
I hear there are also topics on vB sucking at SEO on their boards (haven't checked personally), so my guess is the problems are shared across everything.

Yes, there are SEO problems that have started to crop up in vB. This was caused by their constant upgrading of the forum features (and of course, changes in Google rules that we never get to hear of). I think it's a problem with a growing and improving forum software. But we should never trade it off with "search engine-ability" because most site owners still rely on organic searches for visitors.

What I look forward though in SMF is the ability to put in Google SERP the posts themselves. What I know is that only SMF thread titles are being indexed, not the actual posts yet. vB used to be very good in having each and every forum post indexed in Google.

Quote from: Motoko-chan on January 02, 2007, 07:47:25 PM
It might be related to the noindex we are adding to certain URLs to avoid duplicate content being indexed. Then again, it might be because Google felt like it, they don't exactly give info out like that. Has your count increased? You might want to look into their webmaster tools (http://www.google.com/webmasters/) and check what they have on your site through it, it can be very informative (including telling if they are having problems indexing pages).

That's what I'm thinking too but I can't pinpoint exactly what's wrong. Of course, Google won't bother to explain to me what's really going on. I have a Google Webmasters account but it doesn't say anything about the reasons for such in my account. It's still saying there that they are properly indexing my site. I'll wait for a week if there are changes, probably Google just got "surprised" with the 1.1 upgrade so it decided to hide some of my pages. Hopefully they will come back by next week.

The problem with SEO is that noone knows anything absolutely. It's very hard to make decisions about what to include as default when we don't know what all the effects will be.

QuoteWhat I know is that only SMF thread titles are being indexed, not the actual posts yet. vB used to be very good in having each and every forum post indexed in Google.

As to that.. I don't understand how you can have a topic indexed without it's posts?

I made longer post here (http://www.simplemachines.org/community/index.php?topic=127715.msg886775#msg886775) but I'll show you my robots.txt:

QuoteUser-agent: *
Disallow: /index.php?action
Disallow: /index.php?wap
Disallow: /index.php?wap2
Disallow: /index.php?imode
Disallow: /index.php?type=rss
Disallow: /index.php*msg
Disallow: /index.php*sort
Disallow: /index.php*prev_next

User-agent: Googlebot-Mobile
Allow: /index.php?wap
Allow: /index.php?wap2

User-agent: Slurp
Allow: /index.php?wap
Allow: /index.php?wap2

This one disables everything else except index, board and topic pages. And mobile versions are for Google's and Yahoo's mobile search.

You can submit your forum's mobile pages to yahoo here: http://search.yahoo.com/info/submit.html

Of course if you think your forum's users will use i-mode (http://en.wikipedia.org/wiki/I-mode) too, add it to slurp and googlebot-mobile.

Edit: And btw. using SMF's "search engine friendly urls" is useless. I had them since the beginning of my forum and most of google's results were without them.

Quote from: eldʌkaː on January 02, 2007, 09:45:38 PM
The problem with SEO is that noone knows anything absolutely. It's very hard to make decisions about what to include as default when we don't know what all the effects will be.

QuoteWhat I know is that only SMF thread titles are being indexed, not the actual posts yet. vB used to be very good in having each and every forum post indexed in Google.

As to that.. I don't understand how you can have a topic indexed without it's posts?

What I think the user meant is that each post counted as a separate item in the index. This is because vB has the "feature" to just show one post. SMF doesn't show that granular, so you can only index at the topic level. Of course, IMHO, indexing individual posts is pointless as you don't see the context when you see just those, and if the content on the page is indexed, you are fine anyway.

Quote from: Toadmund on January 02, 2007, 08:47:11 PM
From google webmaster tools:

QuoteHome page crawl:     
Googlebot last successfully accessed your home page on Dec 24, 2006.

THat's good 'cause just yesterday it said Sept. 9th was the last crawl date, finally updated a bit it seems.
But I have yet to see anything updated, no new posts and no evidence of robots.txt kicking in yet, C'mon google, speed 'er up!

Google doesn't crawl some sites all that often, especially if they are unpopular. Viewing their toos page on your site will help speed things up a bit (usually), as will putting their ads on your site (their bot must crawl the site to determine context).

If you don't like seeing their ads on your site, fake it by "testing". Use the following URL, replacing the italic items with what you want:

http://pagead2.googlesyndication.com/pagead/ads?format=728x90&client=ca&adtest=on&url=your site's main url

This previews what ads would look like, and if you do it once a week, it seems to influence Google to index your site a bit faster so the content will be more accurate.

Ok, I was about to reply here so I referred to my first post in this thread posted September 6, 2006. I saw the test I made during that time and tried to check whether there have been changes.

Try this yourself. I'm afraid this test ultimately proved that SMF has a big problem regarding search engines.

Here's the test again:

Quote from: geezmo on September 06, 2006, 09:16:08 PM

Let's compare posts from an SMF forum and a vBulletin forum:

http://www.simplemachines.org/community/index.php?topic=48256.0
Posted on September 5, 2005

QuoteMigrating Database from PHPBB2 to Simple Machines

Hello!

I *really* like this forums script, and since my World of Warcraft guild has been using PHPBB2 for a pretty long time now, I would really love to switch them to these forums but I don't want to loose all of the posts and member accounts.

Is there a way to migrate the database file from PhPBB to SimpleMachines so that people don't need to create new accounts, etc.? I know the structure is completely different and all, but I would really love to make the switch.

Please advise. Your help is greatly appreciated!

Here's a post on the same date in vBulletin.org, a site that uses vBulletin.

http://www.vbulletin.org/forum/showthread.php?t=95681
Posted on September 5, 2005

QuoteGetting an editor on moderated posts/threads in AdminCP

I asked this on vB.com and have been advised by Jake to ask here...   I have a few forum categories where my team gets to moderate the threads and posts made in them. However, quite often we need to edit these threads not just in terms of content, but in terms of the format of the post, and the fact that we feel they may not be appropriate to validate in the moderated forum but they would be fine in another forum.  It would be nice if there was a way of editing the format of a post/thread whilst within AdminCP Moderation area as well as ammending the forum category to where the post will show - instead of validating it first then editing it straight away.  Is there a way to accomplish this? How difficult would this be folks?  Thanks.

THE TEST: Search whether the threads will appear in Google.

Google Search for an SMF post: "Migrating Database from PHPBB2 to Simple Machines"
Search Results: NONE

Google Search for a vBulletin post: "Getting an editor on moderated posts/threads in AdminCP"
Search Results: 1

I tried this test again and STILL, the SMF post above doesn't show up in Google but the vB post has been there all along. Remember that the SMF post I mentioned was posted in this forum on September 5, 2005 -- that was more than a year ago!

SMF still has a long way to go before solving this search engine problem.

Agreed, there is a problem. But can you define the problem and what causes it?

Here's another issue. Try this in Google: "site:simplemachines.org/community" and you'll get 4,940 results. That's waaaay low compared to the 852,662 Posts in 111,432 Topics in the forum.

And if you click Page 10 of the search result, you only get to go to Page 4, the rest of the results have been omitted.

I'm only pointing this out because I thought the new version 1.1.1 has already addressed the search engine issue. It looks like the problem even got worse.

What's causing this? I have no idea. But perhaps to sort things out, the developers (with some help from SMF users) can make an intensive comparison of how vB and SMF works, the codes, the functions, etc. That might show why vB posts appear in Google while SMF posts experience otherwise.

If you do a search for vBulletin duplicate URL and indexing issues you will find the web is full of criticism of vBulletin. In fact, many people feel that vBulletin is the worst forum software for duplication issues. One thread I read on the subject pointed out that each vB thread has "at least 10 URLs that can access it".

I am not trying to 'excuse' SMF. The .msg urls are a problem from a duplicate url point of view, but moving to a different forum software is rarely an answer IMHO.

Most forum software has duplicate URL issues (as do most content management systems) because developers generally concentrate on making the software work fast, rather than concentrate on SEO. As a result we have to utilise noindex or robots.txt solutions. It's a shame, because this is rather like shutting the stable door after the horse has bolted.

It's also a shame that many webmasters/developers don't see this issue as important, because SEO is not just about search engines and web site promotion. It's also about good practice for real human users. Google's reasons for excluding dupicate URLs is because they don't want users to be bombarded with lots of different links to the same places. It's a good ideal and as webmasters we should be helping them with that.

QuoteAs to that.. I don't understand how you can have a topic indexed without it's posts?

This can happen if the topic is excluded by a robots.txt file or by noindex. It can also happen if Google thinks that URL is a duplicate. In these cases, Google may only index the title of the page.

QuoteI'm only pointing this out because I thought the new version 1.1.1 has already addressed the search engine issue. It looks like the problem even got worse.

It's also worth noting that it can take many months for Google to reindex and sort out its results when any changes have been made to a web site. In my experience, the bigger the change the longer it takes for Google to 'trust' the web site again.

I mentioned on another thread how a site I moved to vBulletin from SMF took over a year to be reindexed properly by Google.

The new SMF 1.1.1 has added noindex tags for all .msg urls. This means that a site that has thousands of URLs indexed with the .msg URLs, will suddenly be asking Google to exclude those URLs from its search results.

Once Google has dropped all those .msg urls, Google then has to make the effort to reindex those threads using the 'correct' root url. For example;

http://www.simplemachines.org/community/index.php?topic=112100.0

instead of;

http://www.simplemachines.org/community/index.php?topic=112100.msg718725

This is one reason that I think we should try to find a way to only apply the noindex rules to threads that were started *after* a forum upgrades to version 1.1. If anyone knows how to do this, it might seriously lessen the negative affect of upgrading to version 1.1.

I started a thread asking for help on how to do this here (http://www.simplemachines.org/community/index.php?topic=134918.0), but no-one seems to know how to do this.

Don't triple post please.

As to the .msg, you should have excluded them in your robots.txt file, so the 1.1 change of using a noindex tag wouldn't make a difference.

Quote from: eldʌkaː on January 03, 2007, 04:16:45 AM
Don't triple post please.

Is this a Simple Machines forum rule? I wasn't aware of this. I was always under the impression that it was good BB protocol, when addressing separate issues from different users, to deal with them in separate posts. My apologies if I have broken the rules.

Quote from: eldʌkaː on January 03, 2007, 04:16:45 AM
As to the .msg, you should have excluded them in your robots.txt file, so the 1.1 change of using a noindex tag wouldn't make a difference.

Well perhaps. But what should or shouldn't have done in the past doesn't really help with the current situation. It would have been great if SMF had had this feature from the beginning, but it didn't. That's life, we all live and learn. ;)

Most SMF forum owners have probably not even heard of robots.txt. And it seems clear from the many threads on this subject that many more are unclear as to how to even format the file properly.

That has led to a situation where most SMF forums will have had many of their threads with .msg urls indexed in search engines. These URLs will now all be dropped by search engines. Ideally, these will eventually be replaced by the root urls, but this can take a very long time and sometimes can cause other problems. Wouldn't it be a good idea if we could find a solution to this by finding a way to only apply the new noindex rules to newly created threads?

That should be possible.. try this:
Display.php
find:
// Create a previous next string if the selected theme has it as a selected option.
replace:
// Don't use noindex if an old topic
if ($topic < 1000)
$context['robot_no_index'] = false;

// Create a previous next string if the selected theme has it as a selected option.

Wow. Brilliant. Thanks eldʌkaː.

I'll give that a go. I assume that if ($topic < 1000)
is simply the topic number that you want the noindex rules to start working from?

Yep. Change it to your current newest topic ID prehaps.

Great thanks.

Quote from: eldʌkaː on January 03, 2007, 02:07:27 AM
Agreed, there is a problem. But can you define the problem and what causes it?

The biggest problem is definitely the default theme. Most are complaining about urls, but that is just a one thing in seo, still biggest issue for most users because everyone sees them. Most don't look "under the hood" where the problem lies. I'm not going into details what is wrong with the default theme, but that is something SMF team should start to improve asap.

This is the first thing where to start: <h1>forum name</h1><h2>topic name</h2><menu><posts>

The fact is, however, google has no problem indexing profile, it just stutters on actual topics. Yes, H1 tags would probably help with actual result positioning but the fact is topics just aren't being indexed well for whatever reason.

What is so different about topics and profiles, the only thing I can come up with is the amount of duplicate links to a topic (hopefully noindex will clear that one up) compaired with a single link to profiles.

Quote from: Ben_S on January 03, 2007, 07:21:24 AM
What is so different about topics and profiles, the only thing I can come up with is the amount of duplicate links to a topic (hopefully noindex will clear that one up) compaired with a single link to profiles.

Probably the amount of links, yes, and probably the complexity of page structure. Spiders comes but it seems that they are having hard time getting results.

I've never seen any evidence that H tags help (or hinder) with search engine positioning or indexing.

I still think that the main issue with topics is less that there are duplicate links to them (Google will simply choose one URL over the other). I suspect that it has something to do with the content and structure of topics/threads.

Although I'm not technical enough to know whether the code structure is an issue, I commented earlier (or was it on another SE topic? There are about three on the go at the moment) that simple themes seem to work best. Forums seem to have so much repetitive content - by this I mean repetitive words and template structure - that I believe that Google sees many topics/threads as having duplicate content, or 'boilerplate' content. Although, so do Profiles for that matter, so I may be talking nonsense.   :-\

Having said that, I found that excluding Profiles from indexing has helped somewhat.

I also wonder whether the board list of links is seen by spiders as just an unimportant list of links, as opposed to an important content page. I know that when a thread is linked to from a static page on a web site, it is indexed much more quickly.

How else can topics be displayed, it's a (fairly) simple table, pretty much the same as all other forum software does it, if google can't handle that then google is seriously flawed IMHO.

The header tags give structure to web pages. Having a semantical default theme in a later version of SMF is something I definately want, and will probably give the biggest benefit to SEO of the various SEO ideas so far given.

Quote from: Ben_S on January 03, 2007, 07:49:52 AM
How else can topics be displayed, it's a (fairly) simple table, pretty much the same as all other forum software does it, if google can't handle that then google is seriously flawed IMHO.

I agree.

But I didn't mean that complexity of code was the issue. More a question of repetitive elements on the page. Some themes have so many 'template' elements on them that the actual 'unique' content can be hidden and perhaps appears to Google as automatically created 'boilerplate' pages? This is something that Google is currently making a big deal out of. Of course, I'm making an educated guess rather than having a lot of evidence. But this would make some sense as to why a simple theme may work better.

Of course SMF is no different to any other forum software in this area (or any other area for that matter). In my vB forums, I also tend to strip out as much of the unnecessary repetitive content as possible.

Something that appears to be coming up from a few people is that the boards of a forum are being indexed, but the threads are not (or are stuck in the supplemental index). I wonder why that might be? And why would profiles be so popular?

I can't claim to understand this very much.

I think that's to be expected. Where are topics linked from? Their boards of course. But in busy forums those topics quickly get moved backwards and soon aren't shown on the first page of the board. The old links get lost, and new ones are formed, which isn't something I think would be good for SE rankings.
Profiles however get linked to from all over the forum, ever post + many other places. And those links don't change as often, so therefore they would rank higher I suppose.

Quote from: destalk on January 03, 2007, 08:05:44 AM
Something that appears to be coming up from a few people is that the boards of a forum are being indexed, but the threads are not (or are stuck in the supplemental index). I wonder why that might be? And why would profiles be so popular?

I can't claim to understand this very much.

Header tags would guide spiders to see important link from not so important links.

Quote from: Stüldt Håjt on January 03, 2007, 08:18:52 AM
Header tags would guide spiders to see important link from not so important links.

I don't believe that spiders see header tags as any more important than making links bold (assuming that they see that as important, which I doubt). In fact, putting links between header tags is quite unusual and might even be seen as an attempt to spam.

From what I've read, header tags (used correctly) can be quite an important factor, although I'm not sure it would help with actually getting topics archived.

Indeed, a semantic theme would help you get higher rankings for the pages that are being indexed, but it wouldn't help if the pages aren't even being visited.

Quote from: destalk on January 03, 2007, 08:43:24 AM
I don't believe that spiders see header tags as any more important than making links bold (assuming that they see that as important, which I doubt). In fact, putting links between header tags is quite unusual and might even be seen as an attempt to spam.

I don't think that's unusual. Actually I'll take a test and put some header tags to the default theme and see what happens.

Headers shouldn't be used to given importance to a link, but to give structure to a page. However in certain situations it would be perfectly fine. If you had a header for each section, you might make it a link to itself so that you can easily bookmark the section.

I agree, using header tags correctly to structure the levels of information on a web page is a good idea. That's just good practice (although certainly not a requirement). But I have never read anything on an SEO forum, or reputable article, that supports the point of view that using header tags will have any bearing whatsoever on ranking a web site. At least not since around 1998 anyway. ;)

If it were that easy, spammers would have a field day and wouldn't bother wasting their time generating thousands of incoming links to their sites.

I wonder if we are in danger of grasping at straws here? For example, making changes to robots.txt files and then making assumptions about the 'results' after a few days or a few weeks. Or talking about meta tags and so on.

SMF is no worse at getting indexed by search engines than any other forum software or content management system Imo. If a web site is useful and has lots of incoming links, it will get indexed regardless of url format, meta tags, or whatever.

Forums (all of them) do have specific indexing problems because of the way they are structured (continually moving links, changing content and so on). In fact the only forum tool that I know of that doesn't appear to suffer from this problem is webmasterworld.com. And that is partially because it is so well known and linked to. But also because the forum has been written from the ground up to be SE friendly. It simulates a static directory structure, has nice urls and so on.

SMF developers have done all the right things with the noindex rules. In fact, these mirror what Google themselves recommend regarding duplicate links. I really think that there are only a couple of points that still really need addressing.

I think that eldʌkaː was probably right earlier on regarding the reasons why profiles are liked more than topics, so perhaps these are best disallowed in the robots file.

Other than that it may be worth trying to work out how the search engines get hold of the dynamic urls, when SE friendly urls are enabled.

But otherwise I would be concerned if major changes were made to the way that SMF forums are structured each time there was an upgrade, without good reason. Google gets suspicious about major changes to web sites and I have seen sites drop right down the rankings for a few months after a major redesign.

Phew, I'm exhausted with this subject now.  :P

Wow. A lot of conversation when I come back. At least this is all constructive.

I appreciate everybody who has been participating. Getting ideas formed is an important thing.

Quote from: destalk on January 03, 2007, 10:36:18 AM
I agree, using header tags correctly to structure the levels of information on a web page is a good idea. That's just good practice (although certainly not a requirement). But I have never read anything on an SEO forum, or reputable article, that supports the point of view that using header tags will have any bearing whatsoever on ranking a web site. At least not since around 1998 anyway. ;)

There is a difference between header tags and properly formatted code. Properly using header tags and other tagging is always helpful (if only to point the search engine to the content).

Quote from: destalk on January 03, 2007, 10:36:18 AM
But otherwise I would be concerned if major changes were made to the way that SMF forums are structured each time there was an upgrade, without good reason. Google gets suspicious about major changes to web sites and I have seen sites drop right down the rankings for a few months after a major redesign.

As far as I know, the developers always aim to keep compatibility and avoid changing the viewable side of things unless it is needed.

QuoteI agree, using header tags correctly to structure the levels of information on a web page is a good idea. That's just good practice (although certainly not a requirement). But I have never read anything on an SEO forum, or reputable article, that supports the point of view that using header tags will have any bearing whatsoever on ranking a web site. At least not since around 1998 anyway. ;)

http://www.sitepoint.com/article/most-common-seo-mistakes Look there it is at number 4 ;) And that article is less than 2 months old.

QuoteIf it were that easy, spammers would have a field day and wouldn't bother wasting their time generating thousands of incoming links to their sites.

Abuse of headers doesn't give any structure to a page any more than abuse of meta tags, or a title tag, or pretty much any abuse.

QuoteSMF is no worse at getting indexed by search engines than any other forum software or content management system Imo. If a web site is useful and has lots of incoming links, it will get indexed regardless of url format, meta tags, or whatever.

I agree, most people seem to forget that. Content is King.

QuoteOther than that it may be worth trying to work out how the search engines get hold of the dynamic urls, when SE friendly urls are enabled.

That would be good to know.. but I wouldn't be interested in investigating myself cause I've never used the inbuilt SE urls.

QuoteLook there it is at number 4  And that article is less than 2 months old.

Hmmmmm. But the key is that nowhere in the article does the author claim that the use of these tags has any effect on ranking, which was my point. Moreover, the fact that he points out that "correct heading hierarchies are among the most underutilized SEO tools" tends to reinforce the point that it is not really an SEO technique. In fact he seems to be talking about it in the same way that we are. I.E. that the correct use of H tags helps to build the structure of the site. From that point of view, it's certainly sensible and perhaps may add to accurate indexing.

QuoteThat would be good to know.. but I wouldn't be interested in investigating myself cause I've never used the inbuilt SE urls.

Fair enough. But perhaps it should be flagged as a somewhat 'broken' feature.

I finally made myself a sitemap that I gave to Yahoo and Google. It includes about 7000 urls and now Google and Yahoo both have over 5000 pages in their results.

Edit: And especially Yahoo have crawled my site like a maniac.

Quote from: Stüldt Håjt on January 04, 2007, 05:06:47 PM
Edit: And especially Yahoo have crawled my site like a maniac.

Yahoo has a habit of being an aggressive crawler, so much so as to cause many sites to have issues under the load of its spiders. If you see it is indexing your site, keep an eye on it.

The following links google has errors for on my site:
I removed my domain name as to not advertise my site hear aka keeping to the TOS

http://www.?.com/forum/Themes/Musiconica/images      4xx error   [?]      Dec 31, 2006
http://www.?.com/forum/Themes/Musiconica/images/    403 (Forbidden)   [?]    Dec 31, 2006
http://www.?.com/forum/Themes/default/images    4xx error   [?]    Dec 25, 2006
http://www.?.com/forum/Themes/default/images/    403 (Forbidden)   [?]    Dec 25, 2006
http://?/forum/...=054db1cc3c13475812063886fda7f4c6;sound    408 (Request timeout)   [?]    Dec 20, 2006

Google is trying to get to themes images, thats not surprising as I removed the /Themes/ entry from my robots.txt file, what is surprising is that there is an entry with ;sound at the end.

Meta tags do seem to improve results, more of my pages from another of my sites got indexed after I had a script automagically insert my title of that page at the beginning of that pages meta tag. I also noticed that the search engine seems to spend a certain amount of time on my pages, when the time ends it seems to leave my site indexing with only what it got during that time period. Also its going through phpsession id somehow? can i disallow those. If you guys could give a completed suggested robots.txt file that we could use, of course with a disclaimer saying that you don't know what it will do to other software on your site.

Thanks and you guys rock.

time = money, if your site doesn't = results then google is going to spend less time there imo

QuoteMeta tags do seem to improve results, more of my pages from another of my sites got indexed after I had a script automagically insert my title of that page at the beginning of that pages meta tag.

how do u done this?

I didnt do it on the forum...

but on regular pages you can have something along the lines of

<meta <?php echo $metainfo; ?>>

You can set the title's using variables or whatever you want or if you generate the whole page via php its even easier... Even pass the title name in the post server variable, but I wouldn't do that...

edit: I noticed the mod Seo4SMF claims to be able to adjust your meta tags etc.

How to Submit/Create SiteMap? i don't have basic knowledge also n it!

To submit a site map to google, go to http://www.google.com/accounts/ go to Webmaster Tools, and it will be in there. Google excepts several types of site maps, the easiest is a bare format listing of your files as plain txt separated by line returns.

https://www.google.com/webmasters/tools/siteoverview (https://www.google.com/webmasters/tools/siteoverview)

You will have to prove to Google you own your site, through a few easy steps while there, not to mention create or have a google account.

How does one create a sitemap? What is a sitemap (I know what, but more detail)?
I have been wondering that myself, and how, or in what way is it helpfull, is it even necessary? Or does it just assist the robot as to where to go?

I just use my forum's RSS feeds and that works fine.
Sitemaps are used to let the spiders know where you have pages at. They don't impact on page rankings, and chances are that they would find them all eventually anyways.

http://www.google.com/support/webmasters/bin/answer.py?answer=40318 (http://www.google.com/support/webmasters/bin/answer.py?answer=40318)
My forum's RSS feed is too short to use as a sitemap, it will get the spider crawling so I guess that would be fine.

You can make it longer.. I use a length of 100 in mine.

wonderful, time to go read the manual lol ... I have been meaning to read it... really  ;)

I am guessing its probably something simple like ;count=100;type=...

I always thought that live feed is only feeding latest posts with msg urls, at least my feed is.

Yep. The RSS feed does seem to feed the .msg posts, as well as any other url with RSS capabilities.

index.php?action=.xml;sa=news;limit=10;type=rss2 doesn't use msg urls for the link (it does for the guid though).

Just found this new article about fixing Google duplicate content & URL issues (http://www.webmasterworld.com/apache/3208525.htm). It is not the easiest read, but it is written by an authority on the subject and may be very useful for those interested in the subject. It probably requires free registration to webmasterworld.

And this is another interesting general discussion about a similar topic.
http://www.webmasterworld.com/google/3110528.htm

Of course, as ever, much of this is speculation rather than 'fact'. ;)

Copy paste for those who don't have access to webmasterworld!

Here's my sitemap: http://www.simplemachines.org/community/index.php?topic=139825.0

Quote from: Stüldt Håjt on January 05, 2007, 08:41:29 AM
Copy paste for those who don't have access to webmasterworld!

They are very long threads. I don't think it would be helpful to copy and paste. Registration is free.

Not to mention the possible copyright violation. Register or don't check it out. ;)

Quote from: destalk on January 05, 2007, 09:09:14 AM
They are very long threads. I don't think it would be helpful to copy and paste. Registration is free.

I thought registeration to webmasterworld costs. Well give us some kind of summary.

Quote from: Stüldt Håjt on January 05, 2007, 02:19:11 PM

Quote from: destalk on January 05, 2007, 09:09:14 AM
They are very long threads. I don't think it would be helpful to copy and paste. Registration is free.

I thought registeration to webmasterworld costs. Well give us some kind of summary.

I thought it did as well, but I can see the full thread right now. Go read while it is open.

Now that's cool, I can read ww.

Registration is free. They just have an opening page that makes it look as if you have to pay, because they put that large subscription page there as a home page.

If you look at the top left (next to their logo) there is a "register" link. That allows you to register for free. The paid subscription just allows you to access a couple of members only board.

Quote from: eldʌkaː on January 03, 2007, 06:17:02 AM
That should be possible.. try this:
Display.php
find:

Code Select Expand

// Create a previous next string if the selected theme has it as a selected option.
replace:

Code Select Expand

// Don't use noindex if an old topic
if ($topic < 1000)
$context['robot_no_index'] = false;

// Create a previous next string if the selected theme has it as a selected option.

Just to confirm that this seems to work perfectly, so far. I have tested it on a brand new installation of 1.1.1 and a custom theme.

Thanks again. :)

But the way, is there a definitive list of the exact urls that the noindex tag applies to?

Thanks.

The code is in Display.php (look for "robot_no_index").

If there is a specific message ID in the url as in 1234.msg4321 a noindex meta will be set. Likewise for links that are prev or next links.

Thanks

Hi again, vB and IPB are very well indexed by Google. Also, you can notice that these forum propose a lightweitgh version of topics. Google easyly indexes lightweight versions of forum, because, this is just text with links. And these lightweight versions of these forums can corresponds to the print page of SMF. So why SimpleMachines doesn't add fonctionnalities to this printpage or add a ligthweight version of the forum ? Just an idea ;)

I think that what these archive versions of SMF are supposed to be like, however, it's no different IMO.

http://www.reversedisorder.com/forum/archive.php (http://www.reversedisorder.com/forum/archive.php)

Anyone else have an opinion on archivers?

Quote from: Toadmund on January 10, 2007, 12:22:37 AM
I think that what these archive versions of SMF are supposed to be like, however, it's no different IMO.

http://www.reversedisorder.com/forum/archive.php (http://www.reversedisorder.com/forum/archive.php)

Anyone else have an opinion on archivers?

- Is this a new mod ?
- With this archive, are all your topics indexed by google and other bots ?
- What do you put in robots.txt to avoid duplicates ?

I observed few things, Google bot visit my forum every day, where MSN/YAHOO @one week! I have noticed, Msn sends bots at a time 5-10, they crawl my site like anything, MSN bot even printing the topics, where Google never does! Any idea?

It's vbgamers archiver, only index's the post title, just like regular SMF. Perhaps a version 1.1.1 would be better?
I still can't seem to find it yet (the mod and/or updates to it), I'm only half done my 1st cup of coffee though.

PS, my forum has been completely desolate for the past couple weeks, where is everyone? What does that tell ya?! I need to find something better.

PS, my robots.txt is in this thread, not sure how it's going to be yet, I must wait.

Within "Basic Features"
there is an an option -  "Search engine friendly URLs
Apache only!"

What does this do?

andy

In Basic Features I found an "Search engine friendly URLs
Apache only!"

Perhaps this would go some way?

andy

yes, thats it.

Yes, thats it. That fixes the problem, lets all go home and have tea?
sorry about the multiple posts. getting confused in my old age

Hi, the real problem is that SMF don't produce Search-Engine-Friendly URL's even when the option "Within "Basic Features"
"Search engine friendly URLs
Apache only!" is activated.
Only (and of course a good one) solution is the mod called: SEO4SMF -  since today it also supports bridged Joomla-SMF: http://www.webmasterstalks.com (http://www.webmasterstalks.com)
Enjoy!

It would be nice if our every post apear on every SE's.

It's not the only one, mine works pretty well too ;) http://www.simplemachines.org/community/index.php?topic=126917.0

Quote from: eldʌkaː on January 16, 2007, 08:34:24 AM
It's not the only one, mine works pretty well too ;) http://www.simplemachines.org/community/index.php?topic=126917.0

Ok looks also good.... maybe you two can connect together and develop a very good mod? Hpw about that?

eldʌka, is your pretty URL thing almost done?

I really don't feel like risking an iffy install process, and the pretty URL thing on the webmaster talk forum is not yet updated to 1.1.1.

So basically I wait....again :-\

Can't wait till this SE problem is fixed, getting tired of it!

Thanks for working on the problem though!

Quoteand the pretty URL thing on the webmaster talk forum is not yet updated to 1.1.1.

absolute wrong!!!!

Quoteeldʌka, is your pretty URL thing almost done?

There's still a lot I'd like to add to it, but it's in a stable working condition. I already use it on two of my live forums.

is SEO4SMF v 0.2.4 the latest one?

If so It says this in the install:

QuoteRequirements
A working SMF 1.1RC3 or SMF 1.1.
Apache with mod_rewrite enabled.

And this:

Quote
!!
Error in Package Installation
At least one error was encountered during a test installation of this package. It is strongly recommended that you do not continue with installation unless you know what you are doing, and have made a backup very recently. This error may be caused by a conflict between the package you're trying to install and another package you have already installed, an error in the package, a package which requires another package that you don't have installed yet, or a package designed for another version of SMF.

Here are my mods installed:

QuoteMod Name     Version     
1.    SMF Archive    1.1    [ Uninstall ]
2.    SMF 1.1.1 Update Package    1.1.1    [ Uninstall ]
3.    Topic description    1.0    [ Uninstall ]

[ Delete Mod List ]

PS, sorry for seeming rude, but I was pressed for time, didn't mean to be a dick :-[

Here is the error in the SMF package manager:

Quote10.     Execute Modification     ./Sources/TPortal.php     Test failed

This file is blank in my FTP, I don't have Tiny Portal installed.

Just typed site:www.simplemachines.org/community/ into Google.

It shows

QuoteResults 1 - 7 of about 22,400 from www.simplemachines.org/community.

However;

QuoteIn order to show you the most relevant results, we have omitted some entries very similar to the 7 already displayed.
If you like, you can repeat the search with the omitted results included.

Hmmmmmmm. But yet most of the omitted pages are not in the supplemental index. Strange.

Without using the topic name for the url, what methods could be done to increase SEO? How can we make it so that all message indexes and all topics are crawled and will be shown in Google?

Quote from: groundup on January 30, 2007, 03:40:27 PM
Without using the topic name for the url, what methods could be done to increase SEO? How can we make it so that all message indexes and all topics are crawled and will be shown in Google?

non "fixed" keyword meta tags, set them to post title or leave them out.
title tag moved up to near top are standard seo things

Both are important in google, although keywords only if same on all pages as they don't like it. Title near top very important, as is informative description tags.

Quote from: groundup on January 30, 2007, 03:40:27 PM
Without using the topic name for the url, what methods could be done to increase SEO? How can we make it so that all message indexes and all topics are crawled and will be shown in Google?

Topic name for url is irrelevent IMO. As is keyword metatags, Google ignores them.

I think one of the clues as to why Google is only seeing 7 'dissimilar' pages, can be seen in the google descriptions in the search results.

For some reason it is picking up the 'boilerplate' text on the pages as primary content. For example;

QuoteWelcome, Guest. Please login or register. Did you miss your activation email? Have you tried looking at the Simple Machines Online Manual for your answer? ..

I don't know why, but perhaps it is a layout issue with some themes (including the default one), which makes this content seem the most important to Google.

I also think it is crucial to disallow search engines from crawling user profiles, print pages and different language templates.

Thousands of user profiles is simply seen as duplicate boilerplate pages. Why would anyone want to find a user profile anyway? What possible use is it to anyone, until they have actually signed up as a member of a forum?

One of my robots.txt rules is

User-agent: Googlebot
Disallow: /forums/*action=


Which seems to deal with most of the 'non-user friendly' content part of the forum.

And it may be worth disallowing this in robots.txt, rather than the noindex rules, as this way search engines won't even look at the page, let alone index them.

My 2 cents. Your mileage may vary. ;)

site:domain.com is not a search! It is only a listing. It shows the welcome text because that's the first text on the page. If you're not searching for anything, then you can't expect to get good results. If you actually search for something (http://www.google.com.au/search?&q=site%3Asimplemachines.org%2Fcommunity+package+manager) you'll both get more results, and meaningful excerpts.

Quote from: destalk on January 30, 2007, 08:49:59 PM
Topic name for url is irrelevent IMO. As is keyword metatags, Google ignores them.

They don't provide much weight, but might help push you up to the next slot if you are hanging on the edge. That is about all it does, and the results aren't really worth the effort.

Quote from: destalk on January 30, 2007, 08:49:59 PM
I think one of the clues as to why Google is only seeing 7 'dissimilar' pages, can be seen in the google descriptions in the search results.

For some reason it is picking up the 'boilerplate' text on the pages as primary content. For example;

QuoteWelcome, Guest. Please login or register. Did you miss your activation email? Have you tried looking at the Simple Machines Online Manual for your answer? ..

I don't know why, but perhaps it is a layout issue with some themes (including the default one), which makes this content seem the most important to Google.

If you don't have a meta description tag, Google picks the first text on the page to display. I am not sure how much this influences the ranking though.

Quote from: vagrant on January 30, 2007, 08:12:15 PM
title tag moved up to near top are standard seo things

Both are important in google, although keywords only if same on all pages as they don't like it. Title near top very important, as is informative description tags.

I really don't see what advantage the title tag has in placement in the header block.

The meta description can be changed per the post. I think that would be a good idea. I was actually looking at that.

Google is the worst search engine of the big three in my opinion.  I can't find any of my forum posts in Google search results, either, but I can find them in MSN and Yahoo.

Most likely, if Google indexes your forum, which it won't do until your website is of a certain age or has a certain number of incoming inks, it will put all the forum posts in the "Supplemental Index".  If something is in the supplemental index, odds are it will never be returned in the search results.

My experience:  Google has some of my forum posts indexed, MSN has more, and Yahoo has most of all.  My forum pages are in Google's supplemental index.  If I search all three search engines for a phrase that is contained in an forum post indexed in all three search engines, it will be found in Yahoo and MSN but not in Google.

I do NOT think the problem with Google is SMF.  It is Google.  Google relies heavily on incoming links.  Unless your forum pages (posts) have incoming links, Google isn't going to index them or if it does index them they are going into Supplemental.  Yahoo and MSN rely far less on incoming links than Google does.

Good luck getting thousands of incoming links to your forum posts so Google will like you.

 

Quote from: geezmo on September 07, 2006, 01:48:45 AM
Being indexed as a site and having specific threads appear as SERP are two different things.

Let's take the example of your site. In your forum, you have this thread about the "Mind Static Device" posted on January 25, 2006: http://ftgforums.com/index.php?topic=1490.0

As a forum owner, you expect that whenever somebody searches for "Mind Static Device" in Google, that thread in your forum would come up as a search result right?

Now, let's try it in Google: http://www.google.com/search?hl=en&lr=&q=%22mind+static+device%22&btnG=Search

The "Mind Static Device" thread in your forum does not appear as a search result, and yet it was posted for more than 9 months already. (Your forum had a SERP for the Mind Static Device wiki page but we're talking about the thread itself that I pointed out above.)

That's my concern. When people make a search in Google, SMF posts don't usually come up as a result, as seen in the "Mind Static Device" thread example.

Isn't that something that us forum owners should be concerned about?

It isn't the forum software that is responsible for the experience you describe above it is the inner-workings of Google.

Google is a sh*t search engine! 

Look, if all your posts to your forum had a bunch of incoming links from high Pagerank pages then all your posts would be returned in the Google results.  No incoming links means that in Google's eyes it is of no value and they won't return it in their search results.

That's all their is to it.  Google relies less on relevance and more on incoming links than other search engines.

By the way, I've tried several times to modify the post immediately above that I made but SMF keeps giving me a "database error".

Also, submit buttons die after SMF returns an error and gives you the link to go "back" to the page you were at before the error occurred.

That's how Google works, and I like it that way. If something has a lot of incoming links, its obviously worth being linked to, and is worth me reading it. And if something isn't being linked to, why does it deserve to be ranked well?

Quote from: eldʌkaː on January 31, 2007, 05:40:33 AM
That's how Google works, and I like it that way. If something has a lot of incoming links, its obviously worth being linked to, and is worth me reading it. And if something isn't being linked to, why does it deserve to be ranked well?

It deserves to be ranked well if it is relevant to the search.  What is irrelevant is the number of incoming links because not everything is going to be widely linked to.

For example, let's say you are the president of your graduating high school class and are responsible for planning class reunions.  Let's say 6 months prior to the reunion you put up a webpage about the plans for the reunion, when and where it is to be held, cost, RSVP information, etc.

How many incoming links do you think you are going to get???  Probably zero.  Most webmasters didn't go to your highschool and don't give a rip about your reunion, right?

But is your information not worthy of being found by the classmate searching Google for information about upcoming class reunions???  Because if you don't have incoming links, Google isn't going to treat your informative webpage favorably--possibly not even indexing it and if it does index it, it may not return it in the results.

Do you really think that you are going to get hundreds of incoming links to your class reunion page?  Or even two?

The fact that you don't have incoming links does NOT mean that your information is of no value to anyone.

Consider this:  if something has a lot of incoming links it could possibly be because the website owner purchased a lot of incoming links, got into link exchanges with other webmasters (which is NOT indicative of worthiness of being linked to as they are designed to influence search engines only), or even set up a bunch of websites to link to each other so they all appear to have lots of incoming links.

As you were saying...?

A straight count of incoming links isn't enough to get high Google rankings, because it takes into account where the links come from etc.
Now how is it going to be able to determine what's relevant to your search? When there are billions of web pages you can't have people determining individually what is relevant to each topic. Google spends a huge amount of money on their engineers to try and make each search return the most relevant results.
In your example of a school reunion website, you should be able to get incoming links from the school's webpage, your own, some past students blogs perhaps, and maybe even a local newspaper or two. All of these should be very relevant incoming links, especially if they talk about the school a lot.
If someone purchases lots of incoming links, they're very unlikely to be relevant. The page with the link on it won't have any similar context or content to the linked-to page. Again, Google spends a huge amount of money to make sure these schemes don't work.

Quote from: eldʌkaː on January 31, 2007, 06:06:09 AM
A straight count of incoming links isn't enough to get high Google rankings, because it takes into account where the links come from etc.

What happens to your webpage in Google's eyes then if it has 0 incoming links?

Quote from: eldʌkaː on January 31, 2007, 06:06:09 AM
Now how is it going to be able to determine what's relevant to your search?

Easily:  by the search terms entered into the search box!

Search term example: St. Louis South High School Reunion

I'd assume that any webpage about St. Louis South class reunions would be relevant to those search terms regardless of how many incoming links there were.  Right?  Any page with those keywords either in that exact phrase or close to it would be relevant, right?  The Class of 1992 St. Louis South High School reunion page would be relevant, as would be the 1957 St. Louis High School reunion, etc.

But, if there are no incoming links, Google is probably not going to return it in its results despite the fact that it is totally relevant to the search.

Quote from: eldʌkaː on January 31, 2007, 06:06:09 AM
When there are billions of web pages you can't have people determining individually what is relevant to each topic. Google spends a huge amount of money on their engineers to try and make each search return the most relevant results.

You just said Google spends a lot of money.  They could hire tens of thousands of people to evaluate search results, remove spam, etc.  But they don't need to.  They need to rely on what is contained on the webpage and not on how many incoming links it has.

The fact that Google spends a lot of money doesn't mean they are the best.  Plenty of Linux users will argue that Windows is not the best because Microsoft spends a lot of money on it.  (Ask around here!)  Microsoft reportedly spent $6 billion developing Vista.  It doesn't mean anything other than they spent a lot of money. 

Quote from: eldʌkaː on January 31, 2007, 06:06:09 AM
In your example of a school reunion website, you should be able to get incoming links from the school's webpage, your own, some past students blogs perhaps, and maybe even a local newspaper or two.

Blogs?  What value are blogs?  Most are very low PR.  Google values incoming links largely based on the pagerank of the page providing the incoming link. 

I just checked the website of the high school I went to and there are absolutely no links of any kind on it to any class reunion websites.  I don't know if school districts pay webmasters to just sit around and wait for link requests to come in.  I kind of doubt it.

Do you link to every website everytime someone sends you an email asking you for a link?

Quote from: eldʌkaː on January 31, 2007, 06:06:09 AM
All of these should be very relevant incoming links, especially if they talk about the school a lot.

Relevant?  Perhaps--if those incoming links exist (remember, the hypothetical webpage is going up only 6 months prior to the reunion.  Not a lot of time to get incoming links, is it?).  But as I mentioned above the incoming link is valued largely based on the pagerank of the page containing the link.  Low PR page linking to something means the link is of little or no value.  If that wasn't the case, people would just set up dozens of free websites on Freewebs.com and instantly have a bunch of incoming links to look popular.

Quote from: eldʌkaː on January 31, 2007, 06:06:09 AM
If someone purchases lots of incoming links, they're very unlikely to be relevant.
The page with the link on it won't have any similar context or content to the linked-to page. Again, Google spends a huge amount of money to make sure these schemes don't work.

What if the content on the linking page is relevant?  For example, a new porn website buys links from existing porn websites.

I know you know that there is an entire industry that does nothing but buy and sell links to make Google think they are popular.  I know that you also know that webmasters contact other webmasters trying to get links in an attempt to influence Google. 

What you said about Google makes me think of this analogy:  If a girl doesn't get asked out on a lot of dates, then she musn't be worth dating.  Yet nothing could be further from the truth.

The fact that a website has a lot of links does not mean it is of the most value to the searcher.  What is of value is the information contained on those webpages and incoming links--particularly if they were purchased or acquired via link exchanges--does not mean the content is valuable.

What about small town high schools?  What about high schools that no longer exist?  Neither are likely to get many, if any, links to reunion webpages.  But that doesn't mean the information is irrelevant.  It doesn't mean Google shouldn't return it because it has no links.

As a quick test I randomly chose an SMF post and seached Google and Yahoo using a phrase from that SMF post.

http://www.simplemachines.org/community/index.php?topic=133553.0

"add a horizontal Google ad in the area circled in green"

Yahoo returned the post, Google did not.

Then I searched Yahoo and Google for: SMF+add+horizontal+google+ad.  Inarguably, Yahoo's results are superior.

http://search.yahoo.com/search;_ylt=A0geurrBgcBFlSIB6oRXNyoA?p=SMF+add+horizontal+google+ad&ei=UTF-8&fr=sfp&x=wrt

http://www.google.com/search?num=100&hl=en&q=SMF+add+horizontal+google+ad&btnG=Search

Also, Yahoo shows 322,000 simplemachines.org pages indexed.  Google has only 106,000 (and the bulk of those are probably in Supplemental and will never be returned in search results).   

Quote from: eldʌkaː on January 30, 2007, 09:16:32 PM
site:domain.com is not a search! It is only a listing. It shows the welcome text because that's the first text on the page. If you're not searching for anything, then you can't expect to get good results. If you actually search for something (http://www.google.com.au/search?&q=site%3Asimplemachines.org%2Fcommunity+package+manager) you'll both get more results, and meaningful excerpts.

Indeed. I never claimed it was search. But it's one useful way (of many) to see how a search engine sees the content structure of a web site. Having only 7 pages come up in this way is very unusual, as is the lack of picking up a unique description.

If we push all zealotry and BS aside - I get better results from MSN/Live. Much better actually. Google has reached the point where, I am afraid, they are too large. Their ideals of doing no evil have long since been overshadowed by their desire to make more money.

Having done no marketing with the link in my signature Google now has me down to just 54 links. It was up to about 1000 just a month ago and yet it ranks just as high I suppose. MSN lists about 400 page and doesn't have session IDs in the links. I do a search on Google for:

"I just noticed we don't have a hardware section?"

Google? Nothing returned...

First (and only) listings on MSN?

Two results returned - both pointing to the appropriate pages.

A search for registerfly theft (without quotes) points to my blog in MSN first, in Google? Not on the first page. In fact, it has sites that have a lower PR listed higher. I am boarderline convinced that Google WANTS you to rank poorly - that way you want to buy ads. After all, with 21000 inbound links it is a bit unusual to rank just a PR4 still but, then again, I don't bother with tags of any kind or even optimization. (I get too much traffic there as it is.)

To get back to the topic at hand, SMF topics appear in search engines. They do, slowly they appear. Good SEO is an on-going process and not something that is done overnight but rather something that is maintained like a diet.

Is it possible to delete the comma in the search engine friendly URLs?

For example, make
http://www.domain.com/index.php/board,52.0.html or http://www.domain.com/index.php/topic,1852.0.html
to
http://www.domain.com/index.php/board52.0.html
and http://www.domain.com/index.php/topic1852.0.html

Quote from: KGIII on January 31, 2007, 10:28:59 AM
If we push all zealotry and BS aside - I get better results from MSN/Live. Much better actually. Google has reached the point where, I am afraid, they are too large. Their ideals of doing no evil have long since been overshadowed by their desire to make more money.

I use MSN/Live when I can't find anything in Yahoo or Google.  And sometimes I actually find what I'm looking for in MSN/Live when I couldn't find it in Yahoo or Google.

Sometimes I search for names of people I know, went to school with, or worked with when I have insomnia.  Using Live/MSN I found that a former coworker of mine and her husband opened a franchise and declared bankruptcy.  I didn't find that interesting info in Yahoo or Google.

No search engine will ever be 100% perfect.  However, Google isn't even trying to be perfect--especially when they are purposely not showing results that they index because there aren't enough incoming links.

Quote from: KGIII
I do a search on Google for:

"I just noticed we don't have a hardware section?"

Google? Nothing returned...

First (and only) listings on MSN?

Two results returned - both pointing to the appropriate pages.

It's not in Yahoo, either.

Quote from: KGIII
A search for registerfly theft (without quotes) points to my blog in MSN first, in Google? Not on the first page. In fact, it has sites that have a lower PR listed higher.

In Yahoo the Simplemachines post you made about the Registerfly theft is in position #1 and your blog is in position #4.  blog.kgiii.info/2007/01/15/the-registerfly-sage

So in your case, Live and Yahoo are definitely superior to Google.

Quote from: KGIII
I am boarderline convinced that Google WANTS you to rank poorly - that way you want to buy ads.

That is a popular conspiracy theory.  Look at it this way:  it used to be easy to get indexed in Google and returned in the search results.  Now it is very difficult.  What changed?  They went public.  They now have to generate huge profits to justify their astronomical stock price.  

Quote from: pushkin22 on February 01, 2007, 04:22:09 AM
Is it possible to delete the comma in the search engine friendly URLs?

It won't change anything because search engines don't get SEF URLs (I've reported that to devs long time ago). Turn off cookies in a web browser and check it. Problems with SEO are probably caused not by a form of URLs, but by a enormous amount of diffrent links produced by SMF.

Quote from: motumbo on February 01, 2007, 11:42:07 AM
I am boarderline convinced that Google WANTS you to rank poorly - that way you want to buy ads.

That is a popular conspiracy theory.  Look at it this way:  it used to be easy to get indexed in Google and returned in the search results.  Now it is very difficult.  What changed?  They went public.  They now have to generate huge profits to justify their astronomical stock price. 
[/quote]

I am not personally big on theories without evidence. One thing I have noticed lately is that I keep seeing ads for Google's products listed in their AdSense ads which, as near as I can tell, is a giant conflict of interest if not entirely immoral as well. I was a fan of Google until they started doing everything that was against what they were when they started. Now I mostly get junk when I search there.

Google is, for better or for worse, THE search engine.  They dominate the mindshare of web surfers.  They got that way because they were highly innovative in the results they provided.  Because of that, everyone and their dog is trying to get their web sites to the top of google's SERP.  Google has had to respond by being very selective about what they index, what they don't, what yields placement improvement and what consitutes someone trying to game them.

Be very clear - Google is one of the largest advertising firms.  They also have a search engine.  And an email service.  And a suite of on-line office tools.  And collaboration, development, imaging, payment, financial, online advertising and community tools.  They fully intend to become the Internet.  That's not a conspiracy theory - they want to become to the internet what Microsoft is to the desktop.  They are full of incredibly smart people, and I believe they are quickly obtaining their goal.

But, it's important to understand that the search engine is just a component of google, and one that doesn't make a lot of money at that, directly at least.   

Whatever it maybe, my site is submitted to all search engines throug a custom made script, it submitted to all SE. Even Google indexed my site! Problem s, it's not indexing ONLY forum topics/related things. All other pages which are not related to FORUM are listed

Care to share your custom made script? PM me?

Anyway, how the hell can one's site ever get popular if google doesn't list it adequately enough?
Word of mouth, other forums i guess?

It's like the proverbial saying: "How can I get the job experience that they want, if I can't get a job doing that job?"

Word of mouth is always the best way to get popular.

Quote from: eldʌkaː on February 02, 2007, 01:46:22 AM
Word of mouth is always the best way to get popular.

Amen. I guess they call it organic now. Go figure?

And btw, incoming links is sort of like the internets' version of word of mouth. Spam is the internets' version of viral marketing. BOOO@Spam!

Quote from: eldʌkaː on February 02, 2007, 02:00:52 AM
And btw, incoming links is sort of like the internets' version of word of mouth. Spam is the internets' version of viral marketing. BOOO@Spam!

Bah... It is the chit chat section. ;)

Boo is installing a billion and ten lines of code manually for a killer mod. ;) Thanks again and worth all the frustration and even hand-typing some I missed only a comma.

I am getting better I tell ya!

Quote from: eldʌkaː on February 02, 2007, 02:00:52 AM
And btw, incoming links is sort of like the internets' version of word of mouth. Spam is the internets' version of viral marketing. BOOO@Spam!

No, incoming links are not like word of mouth.  Incoming links, these days, is typically the result of one webmaster linking to another webmaster, often reciprocally, in hopes of making Google think they are more popular than they are.

True word of mouth is one web surfer telling another web surfer about a cool website, not one web publisher asking another web publisher for links.  Most web surfers are not publishers.  Most people looking for information want the information they are looking for and are not the least bit interested if the website has two thousand incoming links or zero.

If you go to the library to do research for a college paper, do you want books and resources that give you the information you need to do your task or do you want the library to only stock what is popular with other library patrons?

Google heavily weighs incoming links.  Google tells webmasters to get more incoming links to get into its search engine.  Google is working against itself by advising webmasters to engage in behavior designed to do nothing but trick Google.     

Quote
Google heavily weighs incoming links.  Google tells webmasters to get more incoming links to get into its search engine.  Google is working against itself by advising webmasters to engage in behavior designed to do nothing but trick Google.    

Google uses incoming links to establish authority on a subject.  In the early days, you could get away with using link farms to get your name on the map, but that's just not so any longer.  The pagerank of the site linking weighs very heavily.  If you have 2 or 3 PR >5 sites linking to you, that's going to do more for your SERP than hundreds of PR < 2 sites.

People are constantly finding ways to "game the system" with google.  And yes, there are quick ways to get to the top 3, but you won't stay there.  SEO forums are FILLED with comments like: "I did XXX and went from page 10000 for my keyword to #3 in 2 weeks, but now my site has disappeared!  please help!".  It's like the story of the bull and the turkey.  BS can get you to the top, but it won't keep you there.

The current popular scams now are:

multi-tiered link farms, where I link to you, you link to someone else, they link back to me, all as part of an arrangement.  It's hard to correlate that kind of activity, but I believe google has recently started to catch on.

Writing articles.  This started out (IMO) as a fairly honest thing years ago where people could get their (and their site's) name out by writing authoritative articles.  Nowadays, you get people writing about how to shave their dogs, how to get your oil changed at jiffy lube, and crazy stuff like that.  There's still some good stuff there, but google will ultimately be forced to penalized the whole process, IMO.

Domain farms.  Domains are cheap.  I've done this.  I own over 50 domains personally, though not out of the intention to game google, but in the misguided belief that I will do something with them all.  But, I CAN get a new site indexed and part of the SERP in under a day, which is impressive, IMO.  I'm not that sophisticated, so I can imagine that someone who is determined and makes this their livelyhood could likely leverage this technique well.

Save any fundamental problems with the software that is causing google to unfairly penalize the site (seems like it MAY be the case with SMF), doing the things that make your site a REALLY useful resource for your keyword targets is the only road to long term success.  Good content, active site, no spam, etc, and a bit of marketing will spur other quality sites to link to you.   

Quote from: confusion on February 02, 2007, 06:04:33 AM

Quote
Google heavily weighs incoming links.  Google tells webmasters to get more incoming links to get into its search engine.  Google is working against itself by advising webmasters to engage in behavior designed to do nothing but trick Google.     

Google uses incoming links to establish authority on a subject.  In the early days, you could get away with using link farms to get your name on the map, but that's just not so any longer. 

Nothing could be further from the truth.  I know of a few websites that have set up tons of websites all linking to the main site. 

I found an interesting Google group called "Google Groups: AJAX World" while searching Yahoo.  So, I searched Yahoo and Google using the name of the Google group and this is what I found:

Yahoo has the Google Group in position 1.

Google:  Not in top 100.

Google does have a French Ajax World Google Group around position 15.  But it isn't the group we are looking for.

Yahoo does a better job finding Google Groups than Google does!!!

Quote
Nothing could be further from the truth.  I know of a few websites that have set up tons of websites all linking to the main site. 

Absolutely, it's still done.  It doesn't have the effect it used to on SERP/PR status, though.  If you are not careful, it can get you sandboxed.

Quote
I found an interesting Google group called "Google Groups: AJAX World" while searching Yahoo.  So, I searched Yahoo and Google using the name of the Google group and this is what I found:

Yahoo has the Google Group in position 1.

Google:  Not in top 100.

Google does have a French Ajax World Google Group around position 15.  But it isn't the group we are looking for.

Yahoo does a better job finding Google Groups than Google does!!!

My comments were not a plug for the accuracy of google as a search engine.  I believe google has had to implement so many crazy algorithms to avoid spammers that you are far better off searching yahoo or msn if you really need to find something.  The fact remains, though, that despite it's troubles, it is THE search engine people use, and as such, it's what site owners care about.

Take for instance my syslog site.  On google, I am #3, on MSN I am #4 and on yahoo I am #2.  An analysis of the web server logs over the past 4 years should >92% of search engine referrals come from google and it's pretty steady month-to-month still.

So, yes, it is easier to get your site on "my keyword" in the top 5 on msn or yahoo, and it is also true that people looking for information on "my keyword" would be better off searching in msn or yahoo, but few people actually do.   

Quote from: confusion on February 02, 2007, 07:36:53 AM
Absolutely, it's still done.  It doesn't have the effect it used to on SERP/PR status, though.  If you are not careful, it can get you sandboxed.

You don't get sandboxed, you start out sandboxed, once you're out you're out for good (unless domain changes owner etc) you can however get removed/penalized heavily by overdoing stuff.

One thing I noticed though is that all pages have the same keywords, thus increasing the changes of pages being in supplemental hell.

Quote from: NoFear on February 02, 2007, 10:22:20 AM

One thing I noticed though is that all pages have the same keywords, thus increasing the changes of pages being in supplemental hell.

My understanding from reading Matt Cutts blog is that one of the primary determinants of whether or not your pages end up in supplemental is the number of incoming links to the page (another determinant is the number of parameters in your query strings). 

Most of my pages are in Supplemental.  Matt Cutts claims that being in Supplemental isn't bad.  But my experience has been that I can't find my Supplemental pages in the search results.  If my page is indexed and in Supplemental, I can't find it even if I search for a phrase in quotes on the page. 

Google's Supplemental Index is bad news.   

Quote from: confusion on February 02, 2007, 07:36:53 AM
My comments were not a plug for the accuracy of google as a search engine.  I believe google has had to implement so many crazy algorithms to avoid spammers that you are far better off searching yahoo or msn if you really need to find something.  The fact remains, though, that despite it's troubles, it is THE search engine people use, and as such, it's what site owners care about.

I'm glad to hear that you are a sensible person.  You are correct that Google is THE search engine.  It is used by close to 50% of web searchers.

However, does Google need to be THE search engine?  Would it be possible for enough webmasters to get together and encourage our users to ditch Google in favor of Yahoo or Live because Google has so many problems, puts pages in Supplemental Hell, doesn't put pages in SERPS for lack of incoming links no matter how relevant the page is, etc?

The nice thing about the internet is that the little guy can have a big voice.

Quote from: confusion
Take for instance my syslog site.  On google, I am #3, on MSN I am #4 and on yahoo I am #2.  An analysis of the web server logs over the past 4 years should >92% of search engine referrals come from google and it's pretty steady month-to-month still.

Google's marketshare isn't that high!  Does everybody find you using the same keywords?  Perhaps certain demographics slant toward Google.

Quote from: confusion
So, yes, it is easier to get your site on "my keyword" in the top 5 on msn or yahoo, and it is also true that people looking for information on "my keyword" would be better off searching in msn or yahoo, but few people actually do.   

For now...

There often exists a commonality in what is good for two different parties.  It is best for search users and webmasters alike that searches be done not with Google.  At the very least, I'd hope that a bunch of webmasters publically stating their case about Google would ecourage them to get their act together and go back to what they used to be.

I selected no indexation by google when I installed my forum.

Reading here (http://www.simplemachines.org/community/index.php?topic=141181.msg934086#msg934086), I think this may have been a good idea.

My site's main articles are indexed by Google, so it's not a problem that the forum isn't if spambots rely on Google indexation for their databases.

I haven't had a single spam registration since changing from phpBB2 to SMF.  :D

Quote from: geezmo on October 26, 2006, 10:18:01 AM
Have you already tried that in your site, Ben_S?

Went to your Liverpool FC forum, congrats, that's a really huge forum. But still are you not concerned  that out of 2.3 million posts, only 1,000+ entries appear in Google search results, and most are not even threads or posts?

http://www.google.com/search?hl=en&q=site%3Aredandwhitekop.com%2Fforum&btnG=Google+Search

Google.com - site:redandwhitekop.com/forum

Results 1 - 10 of about 1,640 from redandwhitekop.com/forum for . (0.04 seconds)

maybe you should try next time as :

http://www.google.com/search?q=site:redandwhitekop.com&hl=en&start=70&sa=N

not /forum  

Quote from: geezmo on September 13, 2006, 01:40:11 PM
All I can say is that if you're not concerned with your site not ranking high in the search engines, it means you're merely a newbie webmaster or your site caters to only a few people. Ranking in search engines is a must if you want more traffic and more earnings.

Only if your earnings are ad-based. I won't/don't run ads.

@Slamlander, again it's all about organic search. If searchers can't find your site via a search engine, there's a big problem with your site. Of course, you can say you can use Adwords, but for those who can't afford to advertise right now, organic referral is one, if not the only, way visitors can discover the site.

@majo, the /forum suffix was used in that search because we were only testing how many pages in the forum (thus /forum) appear as SERP. Using "http://www.google.com/search?q=site:redandwhitekop.com&hl=en&start=70&sa=N" will give you results such as articles, blog pages, etc. which are not the subject of this forum test. Again using the /forum site search, only 6,700 results appear in Google considering the fact that more than 80,000 topics and 2,000,000 posts are in the forum.

I found one site www.scriptsocket.com! There is a free sitemap generator! It will be useful for who want to submit sitemap to google

I personally use GSiteCrawler (http://gsitecrawler.com/) for making my sitemaps. It has some nice reporting options that are useful in many occasions.

Quote from: geezmo on October 26, 2006, 10:18:01 AM
Have you already tried that in your site, Ben_S?

Went to your Liverpool FC forum, congrats, that's a really huge forum. But still are you not concerned  that out of 2.3 million posts, only 1,000+ entries appear in Google search results, and most are not even threads or posts?

http://www.google.com/search?hl=en&q=site%3Aredandwhitekop.com%2Fforum&btnG=Google+Search

Google.com - site:redandwhitekop.com/forum

Results 1 - 10 of about 1,640 from redandwhitekop.com/forum for . (0.04 seconds)

Back on the climb since adding profiles to robots.txt

http://www.google.com/search?num=100&hl=en&safe=off&c2coff=1&client=firefox-a&rls=org.mozilla%3Aen-US%3Aofficial&hs=0Xm&q=site%3Aredandwhitekop.com%2Fforum&btnG=Search
Results 1 - 100 of about 30,900 from redandwhitekop.com/forum. (0.07 seconds)

Certainly scope for improvement but so far it's going up every day.

Ranking well for terms such as torrent help (result 3) despite not being a site remotely about torrents, which is nice.

Indeed. I found that excluding profiles was pretty crucial. Much more important than excluding .msg urls IMHO. In fact I still don't exclude .msg urls. It speeds up indexing and Google seems happy and capable of resolving this to the 'root' url most of the time. I tried leaving the noindex code from SMF 1.1 for a few weeks and got a sharp reduction in indexed files and visitor traffic.

Could be a coincidence of course. You do one thing, Google does this, you do another thing... drives you barmy in the end. ;) Best to concentrate on content..

My site has more than 1200+ index at google.com now! Last week or two weeks ago was only 300+...I am very happy and hopefully google will index my site more and more soon! ;D

site:pentaxworld.com (http://www.google.com/search?q=site%3Apentaxworld.com&ie=utf-8&oe=utf-8&aq=t&rls=org.mozilla:en-US:official&client=firefox-a)

My site is only 2 months old, with Total Posts: 4918. Of course google is not indexing all posts/threads but at least I can see some threads being indexed now.

I dont use SEO, just the default SMF URL type. If they can index SMF site, I believe they should be able to index mine as well :)

Quote from: nitins60 on February 20, 2007, 11:19:24 PM
I found one site www.scriptsocket.com! There is a free sitemap generator! It will be useful for who want to submit sitemap to google

no, this generator doesn't index all pages of forum

Many people have commented about issues with the SMF forum software and Google. In the past there do seem to have some indexing issues. Things I have noticed people express concerns about include;

- threads being indexed with duplicate content due to the .msg url system.
- profiles being given undue priority in indexing whilst threads remaining unindexed.
- threads not being indexed at a level below the main boards.
- threads being put into Google's supplemental index.

Well I checked the latest SMF forum listings in Google today and they are impressive, regardless as to how you look at it;

site:www.simplemachines.org/community

http://www.google.com/search?hl=en&q=site%3Awww.simplemachines.org%2Fcommunity&btnG=Search

and

site:www.simplemachines.org

http://www.google.com/search?hl=en&q=site%3Awww.simplemachines.org&btnG=Search

Some points in particular that I noticed have changed in these results;

1. The majority of the first few search results are ordinary threads, not member profiles, which always used to be a problem in Google. This is especially interesting as the simplemachines.org web site does not exclude member profiles via robots.txt. Be interesting to know if these changes are due to any kind page structure change of the SMF development edition, or just Google getting better at indexing the site?

2. Many of those threads are indexed at the 'root' of the url, rather than the .msg version. Which is a testament to the implementation of of the noindex feature of the new version of SMF. Still some way to go, but that's just a matter of waiting for Google to get around to update its index.

3. There are very few pages in Google's supplemental index - the bulk are in the main Google index. A sign that Google sees each topic as having 'unique' content. I.E. No duplicate content issues.

4. There are no new PHPSESSID indexed URLs. Which shows that other changes to the forum software a while ago are working well.

It seems clear that even the default SMF installation is being spidered quite well by Google.

I have to agree with that :)

I actually get completely different results. 7 on /community and 14 on the site root. 110,000 displayed in the supplemental results. Might just be my Google server.

I get the same as groundup.

Quote110,000 displayed in the supplemental results

How strange. That's what I got last month.

List of Google data centres if anyone is curious.

http://www.vaughns-1-pagers.com/internet/google-data-centers.htm

And it is true, on some data centres I also get only 7 results for site:simplemachines.org/community

The rest are not supplemental though. You just need to click on more results. Which does kind of indicate a 'similar content' issue.

But perhaps we still need to watch this space for a while. Let's hope it's going in the right direction.

I believe with more and more people using SMF, google will change their index structure on SMF forum and make it better index.

Search engines need to be smart in indexing (showing up good search results), so that people will like or love it more. If not, then there might be another Oogle coming out soon... :)

I am not sure about vbullitin, but smf comes with no form of SEO on the sites. If someone wants to be ranked high then they would need to have some SEO done to their site. If any one needs it, I can do SEO. If you would like more info please e-mail me at GPGrieco(at)msn(dot)com

QuoteI believe with more and more people using SMF, google will change their index structure on SMF forum and make it better index.

I highly doubt that they will change their indexes for the benefit of one group of people or one software.

I don't see that happening

Quote from: GPGrieco on April 09, 2007, 10:40:00 PMI am not sure about vbullitin, but smf comes with no form of SEO on the sites. If someone wants to be ranked high then they would need to have some SEO done to their site. If any one needs it, I can do SEO. If you would like more info please e-mail me at GPGrieco(at)msn(dot)com

That's not true. SMF now uses a meta noindex rule very effectively, which is probably one of the most important SEO practices.

Quote from: GPGrieco on April 09, 2007, 10:40:00 PM
I am not sure about vbullitin, but smf comes with no form of SEO on the sites. If someone wants to be ranked high then they would need to have some SEO done to their site. If any one needs it, I can do SEO. If you would like more info please e-mail me at GPGrieco(at)msn(dot)com

Why don't you just share your knowledge with all of us?

Ok, I'm going to resurect this thread as I can't find anything more relevent.

Currently running SMF 1.1.4
Posts- 48070
Threads - 4819
Forum age - 1 year

Most of the forum is private (to protect the tech content, long story).

My main issue is that the forums aren't showing up on Google search when the 2 most important keywords are used for a search - KLR650 & forums.

Yahoo doesn't see it either, though MSN sees it as the 4th on the results.

I really don't care if Google indexes the threads of the forums, what I really want is the main forum index to show up when a basic search is done.

It's the 2nd largest KLR650 forum-

http://klrworld.com/forums/

Any help/recommendations are appreciated ! :D

I'll also add that I have seen up to 130 spiders crawling at once, and usually at least 5 are there at all times.

hmmmmmmmmmm important

Do you have adequate backlinks (other sites linking to you)?  If not, you will not be indexed very highly in google's eyes.

By your restrictions, you are also limiting what can be indexed of your page, this can and probably will affect your indexing and position as well.  I would reccommend NOT using the disallow guests option.  Instead disallow them entry into the boards you don't want them to see by removing the board access from the guests group.

Using an SEO mod would have almost no effect for you since you are effectively blocking the content.  Without unique content you will not be indexed very highly.  Having a page doesn't guarantee good indexing ;).

I have quite a few sites that link the forums main index page.

Like I said earlier, it's really not that important to me if the threads are indexed.

What is important is that the forum shows up when "klr650 forum" (or similar) is searched in Google. Google isn't even showing it as a result & it should.

As far as the guests go, the thing that makes this forum atypical in this community is it's contemt. It's setup like a maintenance manual with dedicated reference sections. One of my mods had a bunch of his material stolen from an open forum (anyone could see the content) and it was being sold on ebay without his consent.

By restricting access to registered members only, they are bound by the user agreement which states that they cannot publish or use any of the content without the permission of the Admin or the author.

If you haven't already, get your site set up in Google's Webmaster Tools (http://www.google.com/webmasters/tools/). You can check on reports about how your site is being indexed along with any problems being encountered when it crawls.

(I just did a random text by searching based on post text in one of my forums and the posts came up at the top of returned results.)

already done, and a sitemap submitted-

Hi klra,

Now that after seeing your post, I check my site which is http://pentaxworld.com (http://pentaxworld.com) for search keyword "pentax forum" and after more than 10 pages in Google, I still can't find it...

If you got a solution, I would love to know!

yeah, we gotta figure this out-

Quote from: klra on November 03, 2007, 11:28:39 AM
already done, and a sitemap submitted-

Great. Check that the sitemap is being pulled (and is accurate) and that pages are being included in the index. Also see what errors are being found, if any. Also look to get incoming links with the text including search terms you want. Search engine re-ranking takes time (especially Google - it can take months to see results of work).

Quote from: khoking on November 03, 2007, 11:29:53 AM
Hi klra,

Now that after seeing your post, I check my site which is http://pentaxworld.com (http://pentaxworld.com) for search keyword "pentax forum" and after more than 10 pages in Google, I still can't find it...

If you got a solution, I would love to know!

I see a few things here. If you search for "PentaxWorld", your site is #1 on the results. If you, however, search "Pentax Community" then your site isn't on the first page. You are up against what looks like some well-linked and established sites. PentaxForums is PR4 and looks to be very established. Likewise, you have a site with that term as the domain (which counts quite a bit).

Searching for "Pentax Forum" gives us two sites with that term in their domain names. You also have dpreview.net with a PR of 6, photo.net with a PR of 7, another dpreview page with a PR of 5, etc. You are fighting in an established territory, your ranking will increase slowly. Work on building backlinks.


I'd suggest you remove the "welcome" page and go to the community index directly. The alt text for the image you must click doesn't have anything to do with the image or link, and that is probably hurting you since search engines only have that to go by. Put an introduction text on the community index explaining what your site is about while including the terms you want to rank for. Don't stuff the terms though, that will hurt you. Make everything as natural-sounding as possible.

Also, I'd update your forum title to include more of the terms you want to rank for. Perhaps "Pentax World - A community forum for Pentax users". Title text counts for a non-trivial amount of score, after all.

QuoteSearch engine re-ranking takes time (especially Google - it can take months to see results of work).

Well, I think that this is beyond that, because my Joomla pages (KLRWorld.com) index overnight & this forum has been around for over a year.

Many of the same keywords.

Now if I type in KLRWorld or KLRWorld.com Forums, they both pop up as the 1st result.

i creat sitemap.xml using sitemap mod
i submitted in google webmaster page
it says :
Sitemap statistics:
Total URLs: 5076
Indexed URLs: 1

after months i have 1 indexed URLs only :-(

Are there any things listed on the problem report? That type of thing usually indicates that their crawler is having troubles accessing the site.

Do you block guests from accessing the board? That will keep the search engines from accessing things as well.

Quote from: Motoko-chan on January 26, 2008, 06:49:08 PM
Are there any things listed on the problem report? That type of thing usually indicates that their crawler is having troubles accessing the site.

Do you block guests from accessing the board? That will keep the search engines from accessing things as well.

no !
guests have permission to see topics & posts
you can see my forum at : *removed for illegal content*
& sitemap : *removed for illegal content*/sitemap.xml

what can i do ?
please explain.

i created robots.txt & submitted it to google yesterday

why profile links is fast to submitt but topics & post is very slow ?

sorry for bad english
thanks

I removed your links as they pointed to a forum that dealt with illegal activities.

Have you checked the "Web crawl errors" section on Google's webmaster tools site?

I can't find "Web crawl errors" on google's webmaster tools
but i See "Set crawl rate" & "Crawl stats"

It's the bottom box on the overview page. You can also find more details in Diagnostics > Web Crawl

i have no error !
google found 400 URLs last week
it's better
i think robots.txt solved this problam

ok, but what with duplicate title  tags and short and duplicate meta tags???
for duplicate title tags i have these urls :
/forum/index.php?topic=9665.0;imode
/forum/index.php?topic=9665.100;imode
/forum/index.php?topic=9665.15;imode
/forum/index.php?topic=9665.30;imode
/forum/index.php?topic=9665.5;imode
/forum/index.php?topic=9665.95;imode

i'm confused !
google automatically indexed my urls
but on google webmaster page i have this results :
Sitemap statistics:
Total URLs: 5085
Indexed URLs: 1

It says that for one of my sites as well, but when doing a "site:" search, I see a lot of results for topics.

Quote from: Motoko-chan on January 28, 2008, 06:46:48 PMIt says that for one of my sites as well, but when doing a "site:" search, I see a lot of results for topics.

One thing which concerns me a bit regarding how search engines index SMF is that topics spanning over multiple pages often don't appear to be indexed properly. Take this thread for example. If you search Google for a phrase found on each page, you'll see that most of the pages in the last half of the thread are indexed, while only a few pages from the thread's first half seems to be indexed. Another example can be seen using phrases from this randomly selected multi-page topic, OMG Package manager = ignoring. (http://www.simplemachines.org/community/index.php?topic=127722.0), where only the second page (of two) is indexed by Google. The same behaviour has also been seen using Yahoo or Live Search.

Just to add here..
I switched from phpBB to SMF becz of SEO.. no to mention, now my SMF forum tops loads of keywords in google and yahoo.

I like SMF in every way, but I'm sure that it has a problem with search engines, and when I use the search function within my own site some things don't show up the first time, sometimes I have to search it 3 times for it to show up. I'm not sure if thats related to the search engines tho. Just wanted to bring it up to see if they are related or not.

This is what I did by searching around on the forum. Two days later I was spidered and Im spidered nearly every day. Sometimes multiple times a day with google.

http://www.simplemachines.org/community/index.php?topic=214583.msg1384123#msg1384123

Other search engines seem to do their thing on their own.

Quote from: arya6000 on January 29, 2008, 06:36:02 PM
I like SMF in every way, but I'm sure that it has a problem with search engines, and when I use the search function within my own site some things don't show up the first time, sometimes I have to search it 3 times for it to show up. I'm not sure if thats related to the search engines tho. Just wanted to bring it up to see if they are related or not.

You are talking about SEARCH FEATURE in SMF and thats different thing.
Here we are actually discussing how SMF is treated by Search engines like Yahoo and google etc.

Quote from: Ricky. on January 29, 2008, 02:02:21 AM
Just to add here..
I switched from phpBB to SMF becz of SEO.. no to mention, now my SMF forum tops loads of keywords in google and yahoo.

Could You explain a little more ?

smf have few things like "noindex" directive for duplicate pages and good mods available.. u can work upon it and make it good seo..
Though now phpbb3 also features similar stuff also..

Quote from: destalk on January 03, 2007, 10:36:18 AM

Other than that it may be worth trying to work out how the search engines get hold of the dynamic urls, when SE friendly urls are enabled.

I've wondered the same. Could the search engines be getting a hold of these dynamic URLs by users of gmail, yahoo, or hotmail, who receive them in their email notifications? That's the only way I think they could be getting them.

__________________
Kubuntu Forum (http://www.kubuntuway.net/)

This thread has just confirmed what I have been thinking for so long. Even SPAMMy rehashed content show above my forum posts  for the same title/keyword search.

I have every possible seo mod available keyword-URL's. H1 tags, brilliant meta data (unique/rellevant), backlinks, unique content etc etc.. whilst a VB or PHBB forum has nothing but is listed first page on Google for the title/keywords.

Such a shame.

Quote from: Low on April 25, 2009, 10:57:29 AM
This thread has just confirmed what I have been thinking for so long. Even SPAMMy rehashed content show above my forum posts  for the same title/keyword search.

Well, your topics do have the appearance of being similar to other spam content, so you'll need to fight harder to rank for your area simply because of the huge glut of stuff out there that has been around for a much longer time and are all interlinked.

Quote from: Low on April 25, 2009, 10:57:29 AM
I have every possible seo mod available keyword-URL's.

This has been shown to have virtual no impact on ranking. In fact, Google recommends not rewriting URLs. Also, if you have made that move recently, you will be penalized along with all the old index and credibility being gone. When you switch URL schemes, it is like starting over completely.

Quote from: Low on April 25, 2009, 10:57:29 AM
H1 tags

If you are tossing them in willy-nilly, you could actually confuse the search engine spiders to the structure of your content. This would hurt indexing of your content.

Quote from: Low on April 25, 2009, 10:57:29 AM
brilliant meta data (unique/rellevant)

That would have been very helpful ten years ago. Google has never trusted meta tags and all the other major search engines stopped using them in ranking calculations over 6 years ago. As a bonus, your competitors can easily look at your site and determine the keywords most important to you - giving them a competitive advantage.

Quote from: Low on April 25, 2009, 10:57:29 AM
backlinks

That can be helpful if you get linked with good quality link text and if the sites themselves have content in the same area as you. Doing a quick look, Google only sees two links to your site and Yahoo! sees 185.

Quote from: Low on April 25, 2009, 10:57:29 AM
whilst a VB or PHBB forum has nothing but is listed first page on Google for the title/keywords.

How long have those forums been around? Age is rewarded here as established sites are given greater credibility by default. A one to two year lead is a small advantage. How old are their domain names? Once again, older is better. Also, domains that expire more than a year away seem to have a small advantage. Do those domain names have words similar to search terms in them? Having a domain named similar to search terms does give you a boost. How many backlinks do you see for those sites? A larger number of backlinks and better quality ones also provide an advantage.

I think some of your problems are related to using tactics that no longer work and are potentially damaging (random header tags around content that isn't a header, for example). Another part of your problem is residing in an area that is heavy with spam - you could be hitting a filter and not knowing it simply because of your content area.

Also, you really need to update to the latest SMF. The version you are running has some security flaws that have been patched.

Quote from: Motoko-chan on April 25, 2009, 02:22:54 PM

Quote from: Low on April 25, 2009, 10:57:29 AM
This thread has just confirmed what I have been thinking for so long. Even SPAMMy rehashed content show above my forum posts  for the same title/keyword search.

Well, your topics do have the appearance of being similar to other spam content, so you'll need to fight harder to rank for your area simply because of the huge glut of stuff out there that has been around for a much longer time and are all interlinked.

Quote from: Low on April 25, 2009, 10:57:29 AM
I have every possible seo mod available keyword-URL's.

This has been shown to have virtual no impact on ranking. In fact, Google recommends not rewriting URLs. Also, if you have made that move recently, you will be penalized along with all the old index and credibility being gone. When you switch URL schemes, it is like starting over completely.

Quote from: Low on April 25, 2009, 10:57:29 AM
H1 tags

If you are tossing them in willy-nilly, you could actually confuse the search engine spiders to the structure of your content. This would hurt indexing of your content.

Quote from: Low on April 25, 2009, 10:57:29 AM
brilliant meta data (unique/rellevant)

That would have been very helpful ten years ago. Google has never trusted meta tags and all the other major search engines stopped using them in ranking calculations over 6 years ago. As a bonus, your competitors can easily look at your site and determine the keywords most important to you - giving them a competitive advantage.

Quote from: Low on April 25, 2009, 10:57:29 AM
backlinks

That can be helpful if you get linked with good quality link text and if the sites themselves have content in the same area as you. Doing a quick look, Google only sees two links to your site and Yahoo! sees 185.

Quote from: Low on April 25, 2009, 10:57:29 AM
whilst a VB or PHBB forum has nothing but is listed first page on Google for the title/keywords.

How long have those forums been around? Age is rewarded here as established sites are given greater credibility by default. A one to two year lead is a small advantage. How old are their domain names? Once again, older is better. Also, domains that expire more than a year away seem to have a small advantage. Do those domain names have words similar to search terms in them? Having a domain named similar to search terms does give you a boost. How many backlinks do you see for those sites? A larger number of backlinks and better quality ones also provide an advantage.

I think some of your problems are related to using tactics that no longer work and are potentially damaging (random header tags around content that isn't a header, for example). Another part of your problem is residing in an area that is heavy with spam - you could be hitting a filter and not knowing it simply because of your content area.

Also, you really need to update to the latest SMF. The version you are running has some security flaws that have been patched.

I understand you defending SMF so gracefully but alas you are way off imho.

For one you have NO idea as to what domain I'm referring to. Second.. are you seriously saying meta data has no effect on ranking or was you joking? lol.

As for backlinks, I have over 800 for the site YOU are referring to according to Google WMT ;)

Also, who says I use random h1/header tags when all headers are for the topic content.. relevant!

I tried to update but like usual it says forbidden.

Again.. Motoko-chan, I understand you defending SMF against this problem and replying with what I consider ok reasons, but 15 pages of proof must say something right?

My reply was not a rant, more like reverse shycology in that I can't wait for the day SMF becomes perfect in every way.

Anyhoo, I love SMF and will stick with it forever!

Edit:

I realize my post below might be a bit harsh, but I've just grown a bit tired of seeing so many people encourage bad optimization practices that either do nothing but waste time and resources, or are potentially harmful.

That does not mean that I feel SMF is perfect. There are likely lots of things that could be done better. However, no amount of "you need to do better" will help. What is needed are actual citations of what can be done and why it is helpful. That kind of thing is what put in the noindex tag for duplicate content so late in the 1.1 cycle. For the 2.0 series, one of our beta testers and later team members did an in-depth study on what could be done to make SMF better to index. They provided statistics, empirical evidence, and showed that those things were actually beneficial.

If you feel SMF is having indexing troubles, please investigate why and tell us. If you know a technique we could be using, tell us how we can implement it and where it is shown to be good by outside independent studies.



And now on to the original content, so you can understand the comments in my previous post.

Quote from: Low on April 25, 2009, 02:49:18 PM
For one you have NO idea as to what domain I'm referring to.

Since you did not provide one, I went with the site in your profile (yoebo). If that was not the site, I apologize.

Quote from: Low on April 25, 2009, 02:49:18 PM
Second.. are you seriously saying meta data has no effect on ranking or was you joking? lol.

I'm serious. Monika Henziger (http://research.google.com/people/monika/), Research Director at Google stated as much: "Currently we don't trust metadata because we are afraid of being manipulated." That was published in the Journal of Internet Cataloging in Volume 5(1) in 2002. SearchEngineWatch tags the last major search engine supporting meta keywords for ranking as AltaVista dropping that in July 2002. (Death of a Meta Tag (http://searchenginewatch.com/2165061)).

Most people who follow SEO practices closely can tell you this, just like I did. Even SEOmoz on their Search Engine Ranking Factors list places the importance at 1.2 out of 5, with the comments from experts indicating that it's barely worth your time.

Quote from: Low on April 25, 2009, 02:49:18 PM
As for backlinks, I have over 800 for the site YOU are referring to according to Google WMT ;)

That's good. Google's public facilities are a little constrained for various reasons. However, the huge ratio difference (2:800+) in private to public count indicates to me that very few of those links carry any weight. In other words, they are doing very very little to help your ranking.

Quote from: Low on April 25, 2009, 02:49:18 PM
Also, who says I use random h1/header tags when all headers are for the topic content.. relevant!

But do they make a proper outline? Remember that header tags (h1, h2, h3, ...) are for headers. Remember that search engine indexing doesn't quite have the spacial reasoning of a human. Tags help provide hints on how the content is structured. If you give those hints, make sure they make sense.

As a practice, open the source from a random page and go through your header tags from top down. Create an outline based on these tags. Does that outline make sense? If it does not, then you are probably confusing the search engines. In my opinion, it is better to offer no outline guidance than to provide confusing guidance.

Quote from: Low on April 25, 2009, 02:49:18 PM
I tried to update but like usual it says forbidden.

That sounds like a hosting issue, possibly an issue with mod_security. You really should contact your hosting provider.

Quote from: Low on April 25, 2009, 02:49:18 PM
Again.. Motoko-chan, I understand you defending SMF against this problem and replying with what I consider excuses, but 15 pages of proof must say something right?

Point to the proof in these pages. Most of it is arguing over keywords in URLs, meta tags, and other factors that even the true experts in the field say is of low use. Show me one good post that has proper citations and shows something SMF should be doing.

Quote from: Low on April 25, 2009, 02:49:18 PM
Anyhoo, I love SMF and will stick with it forever!

Glad to hear it. You might also be glad to hear that our upcoming 2.0 release has had many changes to help improve indexing of topics and posts.

Also, if you want a quick guide of good optimizations/changes to make to help get indexed better, that SEOmoz article is quite good (even if it was back in 2007).  SEOmoz: Google Search Engine Ranking Factors (http://www.seomoz.org/article/search-ranking-factors)