SMF and Google Indexing in Eleven Easy Steps

[cripwalk]

I have read numerous posts about forum posts not being indexed in Google's cache and wanted to share my experience.  Also keep in mind that my forum is exactly one month old as of today and a good chunk of my posts are now indexed and I currently do not have any SEO mods installed.  The following is the steps that I took by gathering information from all over these forums.  Some of the below probably had nothing to do with it but I wanted to breakdown everything just in case.  Your mileage may vary.

1. Changed my meta description in index.template.php to a description that reflects my site. This was more for an accurate description when my pages get indexed and probably has nothing to do with getting indexed quicker.

2. Put some keywords in the Title of my site.  For example in my case instead of "Hybrid Jungle" I made it "Hybrid Jungle - Hybrid Car Forum"

3. Changed the title to show my forum name on all pages.  Referenced the following thread:
http://www.simplemachines.org/community/index.php?topic=34221.0

4. Changed my meta keywords to reflect my site. I don't think this helps with Google, but it may help with other search engines.  In addition to static keywords, I also am using dynamic meta keywords using the following method:
http://gazeek.com/content/view/81/31/ [nofollow]

5. Submitted my forum to all the major search engines (Probably not necessary)

6. To help build backlinks I used a directory submission service to submit my forum to high PR sites. In addition I manually submitted my site to some relevant directories.

This is the point I started to notice spiders hanging out on my forum.  After about a week I noticed pages were starting to show in Google's index, however none of them were posts.   Mostly profile, login and help pages.  I waited this out for a while and then determined I needed a robots.txt file in the root of my domain to block the spiders from crawling these pages.

7. Added the following to my robots.txt file which was referenced from these threads:
http://www.simplemachines.org/community/index.php?topic=90763
http://www.simplemachines.org/community/index.php?topic=24726

Koodi [Valitse] Laajenna

User-Agent: *
Disallow: /index.php?action=search*
Disallow: /index.php?action=calendar*
Disallow: /index.php?action=login*
Disallow: /index.php?action=register*
Disallow: /index.php?action=profile*
Disallow: /index.php?action=stats*
Disallow: /index.php?action=help*
Disallow: /index.php?action=printpage*
Disallow: /index.php?action=post*
Disallow: /index.php?action=who*
Disallow: /Themes/


8. Wrote a few articles relevant to my site and submitted them to ezinearticles and isnare.  These articles contained backlinks to my site in the about author sections. 

9. Added "Most Recent Posts" to forum index.  My thinking was at this point Google was spending a lot of time only on the index, so maybe it would follow these links to posts?

By this point there were just a few posts indexed but most were not. 

10. I installed the Social Bookmarks mod and submitted a few informative posts to Digg.
http://custom.simplemachines.org/mods/index.php?mod=967

11.  Installed the SMF Sitemap mod and submitted the link to my sitemap to google and yahoo.
http://custom.simplemachines.org/mods/index.php?mod=755

Bam, next thing I know lots of my posts were being indexed.  At certain times of the day I was getting over 100 google and yahoo bots on my forum.  At this point a good portion of my site has been indexed on Google and it seems like the google and yahoo spiders have a very steady presence. 

If you are still waiting for your posts to be indexed hopefully some of the things in this post will help you.  I know it can be really frustrating, but don't give up hope because it should eventually happen if you are determined. 

[Ben_S]

If using wildcards, you should specify what search engines they are for, not all robots support wildcards.

[kizer]

11.  Installed the SMF Sitemap mod and submitted the link to my sitemap to google and yahoo.
http://custom.simplemachines.org/mods/index.php?mod=755

Thats all I did and Google came very strong. Others followed, but I really wanted google.

[HaGG]

Thank you

[dejiman]

11.  Installed the SMF Sitemap mod and submitted the link to my sitemap to google and yahoo.
http://custom.simplemachines.org/mods/index.php?mod=755

Thats all I did and Google came very strong. Others followed, but I really wanted google.

Please where is the link? I installed the SMF SITEMAP but I didn't see any link associated with it so i created a manual sitemap. Please How do I see the smf sitemap link?  www.dejimanaire.com

[dejiman]

Is the SMF SITEMAP  www.example.com/sitemap.php?

Please I have submitted my site map to google and i have a steady robots.txt file to stop some places i don't want google to index but google those not index other parts of my site.

please check this on my forum xml's links to confirm

May be am right?
www.dejimanaire.com/sitemap.xml

http://dejimanaire.com/index.php?type=rss;action=.xml