Looking for noindex mod or system

wolverine09 · April 29, 2009, 10:47:12 PM

Hi everyone -

Is there a noindex mod for SMF? I'd like to be able to directly tell Google not to index pages. From what I understand this works better than a robots.txt command list (which I already have implemented).

karlbenson · April 30, 2009, 10:30:23 AM

Robots.txt are CRAWLER directives NOT indexing directives.
(which means robots.txt only stops the spider from visiting your pages, it does NOT remove existing pages from Google, and the link may still be indexed in google as a url only [without ever being visited, so it will be a url without a title or description or cached version]).

SMF noindexes alot of duplicate content / junk pages by setting a $context['robots_no_index'] to true.
When this is set, smf the meta tag <meta name="robots" content="noindex" />

Note 1: if you block pages with robots.txt, search engines will never visit the page, and so if there is also a meta noindex, then the spider will never see it.
For this reason using <meta tag> (or the x-robots equivalent) is the BEST and ONLY way to keep duplicate content pages out of the search engines.

Note 2: SMF 2.x noindex alot more duplicate content pages than SMF 1.1.x. And SMF 2.0 RC2 when released will contain support for Canonical tag. (to pass pagerank from duplicate content pages to the preferred page)

Note 3: SMF does not noindex ;wap, ;wap2, and ;imode since those modes run through Wireless.template.php instead of index.template.php. If your wanting to noindex those pages.
Edit Themes/default/wireless.template.php
FIND 3 occurances of <head>
And add after each (before the closing </head> naturally)

Code Select

<meta name="robots" content="noindex" />

---
Generally I'd recommend settings $context['robots_no_index'] = true; for pages that you don't want index.
Although you might want to post more specifics of what pages/where your attempting to noindex.

wolverine09 · May 03, 2009, 09:05:21 AM

Regular expression -

Thanks for the information.

bjraines · May 13, 2009, 09:41:53 AM

In continuing this discussion is it possible to add this meta tag to paginated pages?

青山素子 · May 13, 2009, 11:19:17 PM

It's possible, but then only the first page of any topic would be indexed.

James--- · May 08, 2012, 08:05:58 PM

Hello,

One question about placing the noindex code:

I want to noindex the "shows stats" and "show contributes" pages which are linked from every members profilepage, because they are causing duplicate content (they contain the same content as in the topics).

I know from another member here in the community that I have to edit stats.template.php and display.template.php in my Themes directory, but where exactly do i have to place the noindex metatag: <meta name="robots" content="noindex" />

Because I can't see the <head> tags.

Arantor · May 08, 2012, 08:24:30 PM

You could just disallow access to profiles to guests.

James--- · May 08, 2012, 08:50:14 PM

That's a simple solution but we want guest to be able to see members profile (without registration)
and also let them index by Google.

Arantor · May 08, 2012, 08:55:06 PM

Out of interest, why? Especially as you want to block most of what's in a profile for guests from being indexed...

James--- · May 08, 2012, 09:09:56 PM

Guests can see important information of active members or visit there websites directly if the feel their posts
helped them to find solutions and want to find more about the members.

Second, if Google index the profile, Google users can find them in search results. This can be useful
for a whole variety of reasons such as: if people want to buy a product from an active forum member with a
active profile, they can check their reputation and also check their forumposts and see what they've talked about.

Also, we add an option for our members to add some description on the profile page (just in normal text) where
they can show who they are and what they do. This adds more content to the profilepages and more text for Googlebot to find.

QuoteEspecially as you want to block most of what's in a profile for guests from being indexed...

The content in the statistics and contributes pages of a member is useful for visitors,
but not for bots, as they see the same content as on the forum itself (in the topics). This is duplicate content.
(same content accessible via multiple URLs)

Arantor · May 08, 2012, 09:13:09 PM

QuoteGuests can see important information of active members or visit there websites directly if the feel their posts
helped them to find solutions and want to find more about the members.

Their websites are shown next to posts or in signatures, whichever users prefer. Don't need to go to the profile for that.

QuoteSecond, if Google index the profile, Google users can find them in search results. This can be useful
for a whole variety of reasons such as: if people want to buy a product from an active forum member with a
active profile, they can check their reputation and also check their forumposts and see what they've talked about.

I'm not sure that's really a good measure, no matter what your site is. If someone is already looking to buy, they're going to be a member (and thus have access to profiles) but I can't see it being a big issue for someone to use Google to find that person. Odds are they'll be long drowned in a sea of irrelevance.

In any case, again, if you want people to be able to look around, they'll register and do it that way.

QuoteAlso, we add an option for our members to add some description on the profile page (just in normal text) where
they can show who they are and what they do. This adds more content to the profilepages and more text for Googlebot to find.

That still requires people to put something in and again it's almost certainly going to be lost in a sea of irrelevance, it's just not what people will generally search for.

James--- · May 08, 2012, 09:31:46 PM

Okay. You have your standpoint, I have mine. I respect your opinion, but that's not going to change anything.
I'm looking for some help to place the noindex meta tag at the appropriate places. The question stays the same actually:

Where do I have to put this noindex metatag in the stats and display files?
After which code?

Arantor · May 08, 2012, 09:57:55 PM

That's the problem, it doesn't work like that. The noindex has to be added in the master template - which applies *everywhere*.

You haven't said what version of SMF you're using and whether you have any other mods that add new areas to the profile area or not.

Kindred · May 08, 2012, 09:58:44 PM

Well, the best way to get your question answered would be to post in the correct board, first of all. This is not a comment nor feedback on SMF, so why did you post here?

Second of all. If you are using SMF 2.0.2, the system uses meta tags to define the base urls and thus will recognize and not penalize for duplicate content.

Arantor · May 08, 2012, 10:03:45 PM

Actually, it will penalise you for duplicate content here because the canonical tag doesn't point back to the threads from the posted content. Mind you, there will be penalties for the entire profile area anyway because so much of it is generalised boiler plate.

Would still be easier to simply hide those aspects from guests then it would to fart about with getting noindexes in place.

maestrosite.ru · May 12, 2012, 08:00:04 AM

Please, show examples urls for:

Quote from: James--- on May 08, 2012, 08:05:58 PM
"shows stats" and "show contributes" pages

James--- · May 15, 2012, 06:52:05 AM

http://www.geldverdienenopinternetforum.nl/index.php?action=profile;area=showposts;u=7

http://www.geldverdienenopinternetforum.nl/index.php?action=profile;area=statistics;u=7

I'm still looking for a way to add noindex to those pages (see links above) because they generate duplicate content for search engines.

Please see screenshot.

If you click on the 2 links on a members profile page, you notice that the same content appears as in the topics itself.

QuoteWell, the best way to get your question answered would be to post in the correct board, first of all. This is not a comment nor feedback on SMF, so why did you post here?

Sorry. Didn't notice that, because the topic was already there. Can you move the topic for me?

maestrosite.ru · May 15, 2012, 07:36:17 AM

1. Install http://custom.simplemachines.org/mods/index.php?mod=3315 for real canonical
2. In Profile.php add

Code Select


$context['robots_no_index'] = true;

to bottom of function ModifyProfile. Note: also you may add conditions for seperate pages in "Profile" action:

Code Select


if( _select_areas_ )
    $context['robots_no_index'] = true;

3. Add to robots.txt:

Code Select

Disallow: /*action=

James--- · May 15, 2012, 08:39:57 AM

Hi maestrosite.ru,

I downloaded and installed your rel=canonical mod.

I have a question:

The goal is to let search engines index the profile pages, but don't index the 2 links (in the screenshot),
only the show "show statistics" and "show posts" have to be noindexed. So you said:

QuoteNote: also you may add conditions for seperate pages in "Profile" action:

And then you gave me a code. But I can't find if( _select_areas_ ) in the profile.php template.
Can you further explain this?

maestrosite.ru · May 15, 2012, 08:54:32 AM

No need to find, a condition to be add =) For example:

Code Select

	if( in_array($profile_include_data['current_area'], array('showposts', 'statistics')) )
		$context['robots_no_index'] = true;

Or add bool flag in $profile_areas, and check this flag.

News:

Looking for noindex mod or system