Meta Tags and Search Optimization

[MooseBoy]

I've spent a couple of hours reading various posts on these subject without fuly resolving my question.

1] Will search engine spiders index all my pages on the forum by default?
I've changed the keywords in index.template.php to ones relevant to my site but don't these other 'noindex' tags stop spiders going further?

   <meta name="description" content="', $context['page_title'], '" />', empty($context['robot_no_index']) ? '' : '
   <meta name="robots" content="noindex" />', '

2] I've read that 1.1.1 is more spider friendly but am concerned about my site not being fully indexed in the past

3] Is there anything else I can do to ensure better spidering and listing

thanks

Andy

[jdegerstrom]

I was searching this topic without finding the answer, so I worked it out and decided it may help others to reply to your (old) post. The robots META in the default template needs a simple edit to activate it on each page.

Here is the answer. Please see next reply in this thread for an IMPORTANT correction.

YOUR DEFAULT TEMPLATE CODE

<meta name="description" content="', $context['page_title'], '" />', empty($context['robot_no_index']) ? '' : '
   <meta name="robots" content="noindex" />', '

CHANGE TO THIS

<meta name="description" content="', $context['page_title'], '" />', ($context['robot_no_index']) ? '' : '
   <meta name="robots" content="index, follow" />', '
   
In the default template the word "empty" in effect states "do not show the robots meta on pages", so you need to remove "empty" from your template.&nbsp; On the next line of the default template the robots content "noindex" is irrelevant because it does not display on your pages.

Therefore, remove "empty" and change the "noindex" to "index, follow" to have all your pages crawled by Google, etc.

Jim

[metallica48423]

you really REALLY do not want to remove that noindex..

it is only on pages where there are duplicate content issues.

I'll give an example:
http://www.simplemachines.org/community/index.php?topic=145697 -- this topic -- the meta noindex tag is NOT there.
http://www.simplemachines.org/community/index.php?topic=145697.msg1118500#msg1118500 -- a post in this topic... it IS in there

Notice, both pages, though the url is different is exactly the same?

[jdegerstrom]

Thanks for the fast (great) response.

For a project that I just created in SMF the private forum is for members only and the owner does not want any pages indexed.

By removal of "empty" and changing the robots meta to "noindex, nofollow" each page has the noindex, nofollow tag in the code for every post, reply, and main pages like home and help files, etc.

I emailed the OP to advise my advice for his site was wrong.  Is there a downside that I have overlooked in my application?

Jim

[metallica48423]

thats different though than what's being asked, though it is a great solution for your issue.  I just didn't want him/her to remove or change that code and then find he, well, no longer has indexing in google . 

Thing about that is, that you do not and to add follow code to all pages either, because of the duplicate content.  The way SMF's topic display works, is that each post has a url, but it also displays all posts when you click one, you can scroll up and down.  So each post in a thread has at least 1 URL to it.  If a topic has an original post and a reply, the same exact page has 2 url's that go to it.  If it has >= 20 replies, it has up to 20 URL's PER PAGE that access it.  Toss in the ;all pages, and the other functionality that is passed through the URL... and you have... what google sees as a LOT of duplicate content.  That is why the noindex tag was introduced to give each topic and each page of it one possible indexed URL, which is the shortest possible URL also.

To answer the Topic Creator's question though, look at my prior response, and then look at the page source of both pages

[jdegerstrom]

Thank you for the feedback, metallica48423.

The nofollow solution I used worked for me in the private members only forum which was my first ever, but got carried over to my second project that I do want indexed.  Your solution and explanation was perfect. 

After correcting the template on the 2nd project my crawler found the "lost" forum pages and the nofollow is exactly where it belongs.

Your advice and thorough response to both situations is greatly appreciated.