Looking for noindex mod or system

wolverine09 · April 29, 2009, 10:47:12 PM

Hi everyone -

Is there a noindex mod for SMF? I'd like to be able to directly tell Google not to index pages. From what I understand this works better than a robots.txt command list (which I already have implemented).

karlbenson · April 30, 2009, 10:30:23 AM

Robots.txt are CRAWLER directives NOT indexing directives.
(which means robots.txt only stops the spider from visiting your pages, it does NOT remove existing pages from Google, and the link may still be indexed in google as a url only [without ever being visited, so it will be a url without a title or description or cached version]).

SMF noindexes alot of duplicate content / junk pages by setting a $context['robots_no_index'] to true.
When this is set, smf the meta tag <meta name="robots" content="noindex" />

Note 1: if you block pages with robots.txt, search engines will never visit the page, and so if there is also a meta noindex, then the spider will never see it.
For this reason using <meta tag> (or the x-robots equivalent) is the BEST and ONLY way to keep duplicate content pages out of the search engines.

Note 2: SMF 2.x noindex alot more duplicate content pages than SMF 1.1.x. And SMF 2.0 RC2 when released will contain support for Canonical tag. (to pass pagerank from duplicate content pages to the preferred page)

Note 3: SMF does not noindex ;wap, ;wap2, and ;imode since those modes run through Wireless.template.php instead of index.template.php. If your wanting to noindex those pages.
Edit Themes/default/wireless.template.php
FIND 3 occurances of <head>
And add after each (before the closing </head> naturally)

Code Select

<meta name="robots" content="noindex" />

---
Generally I'd recommend settings $context['robots_no_index'] = true; for pages that you don't want index.
Although you might want to post more specifics of what pages/where your attempting to noindex.

wolverine09 · May 03, 2009, 09:05:21 AM

Regular expression -

Thanks for the information.

bjraines · May 13, 2009, 09:41:53 AM

In continuing this discussion is it possible to add this meta tag to paginated pages?

青山素子 · May 13, 2009, 11:19:17 PM

It's possible, but then only the first page of any topic would be indexed.

James--- · May 08, 2012, 08:05:58 PM

Hello,

One question about placing the noindex code:

I want to noindex the "shows stats" and "show contributes" pages which are linked from every members profilepage, because they are causing duplicate content (they contain the same content as in the topics).

I know from another member here in the community that I have to edit stats.template.php and display.template.php in my Themes directory, but where exactly do i have to place the noindex metatag: <meta name="robots" content="noindex" />

Because I can't see the <head> tags.

Arantor · May 08, 2012, 08:24:30 PM

You could just disallow access to profiles to guests.

James--- · May 08, 2012, 08:50:14 PM

That's a simple solution but we want guest to be able to see members profile (without registration)
and also let them index by Google.

Arantor · May 08, 2012, 08:55:06 PM

Out of interest, why? Especially as you want to block most of what's in a profile for guests from being indexed...

James--- · May 08, 2012, 09:09:56 PM

Guests can see important information of active members or visit there websites directly if the feel their posts
helped them to find solutions and want to find more about the members.

Second, if Google index the profile, Google users can find them in search results. This can be useful
for a whole variety of reasons such as: if people want to buy a product from an active forum member with a
active profile, they can check their reputation and also check their forumposts and see what they've talked about.

Also, we add an option for our members to add some description on the profile page (just in normal text) where
they can show who they are and what they do. This adds more content to the profilepages and more text for Googlebot to find.

QuoteEspecially as you want to block most of what's in a profile for guests from being indexed...

The content in the statistics and contributes pages of a member is useful for visitors,
but not for bots, as they see the same content as on the forum itself (in the topics). This is duplicate content.
(same content accessible via multiple URLs)

Arantor · May 08, 2012, 09:13:09 PM

QuoteGuests can see important information of active members or visit there websites directly if the feel their posts
helped them to find solutions and want to find more about the members.

Their websites are shown next to posts or in signatures, whichever users prefer. Don't need to go to the profile for that.

QuoteSecond, if Google index the profile, Google users can find them in search results. This can be useful
for a whole variety of reasons such as: if people want to buy a product from an active forum member with a
active profile, they can check their reputation and also check their forumposts and see what they've talked about.

I'm not sure that's really a good measure, no matter what your site is. If someone is already looking to buy, they're going to be a member (and thus have access to profiles) but I can't see it being a big issue for someone to use Google to find that person. Odds are they'll be long drowned in a sea of irrelevance.

In any case, again, if you want people to be able to look around, they'll register and do it that way.

QuoteAlso, we add an option for our members to add some description on the profile page (just in normal text) where
they can show who they are and what they do. This adds more content to the profilepages and more text for Googlebot to find.

That still requires people to put something in and again it's almost certainly going to be lost in a sea of irrelevance, it's just not what people will generally search for.

James--- · May 08, 2012, 09:31:46 PM

Okay. You have your standpoint, I have mine. I respect your opinion, but that's not going to change anything.
I'm looking for some help to place the noindex meta tag at the appropriate places. The question stays the same actually:

Where do I have to put this noindex metatag in the stats and display files?
After which code?

Arantor · May 08, 2012, 09:57:55 PM

That's the problem, it doesn't work like that. The noindex has to be added in the master template - which applies *everywhere*.

You haven't said what version of SMF you're using and whether you have any other mods that add new areas to the profile area or not.

Kindred · May 08, 2012, 09:58:44 PM

Well, the best way to get your question answered would be to post in the correct board, first of all. This is not a comment nor feedback on SMF, so why did you post here?

Second of all. If you are using SMF 2.0.2, the system uses meta tags to define the base urls and thus will recognize and not penalize for duplicate content.

Arantor · May 08, 2012, 10:03:45 PM

Actually, it will penalise you for duplicate content here because the canonical tag doesn't point back to the threads from the posted content. Mind you, there will be penalties for the entire profile area anyway because so much of it is generalised boiler plate.

Would still be easier to simply hide those aspects from guests then it would to fart about with getting noindexes in place.

maestrosite.ru · May 12, 2012, 08:00:04 AM

Please, show examples urls for:

Quote from: James--- on May 08, 2012, 08:05:58 PM
"shows stats" and "show contributes" pages

James--- · May 15, 2012, 06:52:05 AM

http://www.geldverdienenopinternetforum.nl/index.php?action=profile;area=showposts;u=7

http://www.geldverdienenopinternetforum.nl/index.php?action=profile;area=statistics;u=7

I'm still looking for a way to add noindex to those pages (see links above) because they generate duplicate content for search engines.

Please see screenshot.

If you click on the 2 links on a members profile page, you notice that the same content appears as in the topics itself.

QuoteWell, the best way to get your question answered would be to post in the correct board, first of all. This is not a comment nor feedback on SMF, so why did you post here?

Sorry. Didn't notice that, because the topic was already there. Can you move the topic for me?

maestrosite.ru · May 15, 2012, 07:36:17 AM

1. Install http://custom.simplemachines.org/mods/index.php?mod=3315 for real canonical
2. In Profile.php add

Code Select


$context['robots_no_index'] = true;

to bottom of function ModifyProfile. Note: also you may add conditions for seperate pages in "Profile" action:

Code Select


if( _select_areas_ )
    $context['robots_no_index'] = true;

3. Add to robots.txt:

Code Select

Disallow: /*action=

James--- · May 15, 2012, 08:39:57 AM

Hi maestrosite.ru,

I downloaded and installed your rel=canonical mod.

I have a question:

The goal is to let search engines index the profile pages, but don't index the 2 links (in the screenshot),
only the show "show statistics" and "show posts" have to be noindexed. So you said:

QuoteNote: also you may add conditions for seperate pages in "Profile" action:

And then you gave me a code. But I can't find if( _select_areas_ ) in the profile.php template.
Can you further explain this?

maestrosite.ru · May 15, 2012, 08:54:32 AM

No need to find, a condition to be add =) For example:

Code Select

	if( in_array($profile_include_data['current_area'], array('showposts', 'statistics')) )
		$context['robots_no_index'] = true;

Or add bool flag in $profile_areas, and check this flag.

Kindred · May 15, 2012, 11:41:24 AM

Quote from: maestrosite.ru on May 15, 2012, 07:36:17 AM
3. Add to robots.txt:
Code Select Expand
Disallow: /*action=

really? disallow all ACTION urls? that's a bad idea....

maestrosite.ru · May 15, 2012, 12:01:36 PM

Quote from: Kindred on May 15, 2012, 11:41:24 AM
Quote from: maestrosite.ru on May 15, 2012, 07:36:17 AM
3. Add to robots.txt:
Code Select Expand
Disallow: /*action=

really? disallow all ACTION urls? that's a bad idea....

Can you show which "action"-pages should be allowed?
Of course in this case you can add specific "action" and "area", and don't disallow all actions

Kindred · May 15, 2012, 12:13:06 PM

action=mlist, action=calendar, action=profile, action=gallery, action=downloads.... etc etc and so forth.

maestrosite.ru · May 15, 2012, 01:17:14 PM

Quote from: Kindred on May 15, 2012, 12:13:06 PM
action=mlist, action=calendar, action=profile, action=gallery, action=downloads.... etc etc and so forth.

If you really need, then:

Code Select


Allow: /*action=mlist
Allow: /*action=calendar
Allow: /*action=profile
Allow: /*action=gallery
Allow: /*action=downloads....   etc etc and so forth.

But what information will be of interest to search engines, for example, on the pages: action=mlist, action=calendar, etc?

If you installed a mod that adds a new pages (new actions), of course, these pages do not need to lock.

upd For example:

Code Select


Allow: /*action=sitemap

Kindred · May 15, 2012, 02:02:10 PM

in other words... ban all actions in order to block a single one - and then re-allow each other action individually? that's just plain silly.

I certainly want the search engines to track my calendar/list of events.
tracking my memberslist (if it's available to the public) is also important, in case someone is searching for a user by name.
tracking users' profiles is also important (again, if its available to the public) for the same reason.

as I said... your idea of disallowing action is just silly.

and search engines will never, ever, get to the modify profile page anyway, so what is the point to adding a noindex to that function?

maestrosite.ru · May 15, 2012, 02:36:59 PM

Quote from: Kindred on May 15, 2012, 02:02:10 PM
in other words... ban all actions in order to block a single one - and then re-allow each other action individually? that's just plain silly.

I certainly want the search engines to track my calendar/list of events.
tracking my memberslist (if it's available to the public) is also important, in case someone is searching for a user by name.
tracking users' profiles is also important (again, if its available to the public) for the same reason.

as I said... your idea of disallowing action is just silly.

Your opinion is clear.
But I certainly want the search engines to track topics only. (And sitemap, but it's a different story)

Quote from: Kindred on May 15, 2012, 02:02:10 PM
and search engines will never, ever, get to the modify profile page anyway, so what is the point to adding a noindex to that function?

As described above, pages such as ?action=profile;area=showposts;, ?action=profile;area=statistics; contains duplicate content. Therefore search engines can get profiles pages. let's see that on this occasion we will answer Google, whether he sees the profiles pages?
http://www.google.com/search?q=inurl%3Aaction%3Dprofile%3Barea%3Dshowposts&btnG=%D0%9F%D0%BE%D0%B8%D1%81%D0%BA&oe=utf-8&client=ubuntu&channel=fs
Google said:

QuoteРезультатов: примерно 9 170 000

Isn't it? (You can have different numbers.)

This is stupid? Perhaps...

Kindred · May 15, 2012, 02:47:24 PM

you didn't read....

Quote from: Kindred on May 15, 2012, 02:02:10 PM
and search engines will never, ever, get to the modify profile page anyway,

you said to put the no index code into the function ModifyProfile

also... *YOU* want the search engines to only search topics... I don't think most of us would agree with that for our forums... but most people who are asking are not coders, so they'd take a look at your code and implement it as you suggest, never realizing that they have now stopped the search engines from getting to anything that is not a topic. (in my case between 30% and 50% of the site is not topics)

maestrosite.ru · May 15, 2012, 03:26:42 PM

Quote from: Kindred on May 15, 2012, 02:47:24 PM
you didn't read....
Quote from: Kindred on May 15, 2012, 02:02:10 PM
and search engines will never, ever, get to the modify profile page anyway,
you said to put the no index code into the function ModifyProfile

As we know, function ModifyProfile is entry point for pages action=profile:

Code Select

$ grep "'profile'" index.php
		'profile' => array('Profile.php', 'ModifyProfile'),

Quote from: Kindred on May 15, 2012, 02:47:24 PM
also... *YOU* want the search engines to only search topics... I don't think most of us would agree with that for our forums... but most people who are asking are not coders, so they'd take a look at your code and implement it as you suggest, never realizing that they have now stopped the search engines from getting to anything that is not a topic. (in my case between 30% and 50% of the site is not topics)

1. As you wrote

Quote from: Kindred on May 15, 2012, 02:47:24 PM
you didn't read....

Quote from: maestrosite.ru on May 15, 2012, 01:17:14 PM
If you installed a mod that adds a new pages (new actions), of course, these pages do not need to lock.

James-- forum has no such mods
2. Search engines see sitemap and get real pages - can again ask google. (And James-- forum has sitemap)
3. How much of the required (in search) information contains no-topic-pages? (30-50% pages or content )?

Kindred · May 15, 2012, 03:42:11 PM

Sorry about the modifyprofile thing... I was thinking of the file profile-modify.php

but, apparently you were not paying attention to james either, since he clearly said that he wants the search engines to see and index the profile pages....

Quote from: James--- on May 08, 2012, 08:50:14 PM
That's a simple solution but we want guest to be able to see members profile (without registration)
and also let them index by Google.

as for my site... 30-50% of the site is used for calendar events, galleries, articles, static pages and wiki. (among other things), but the entire site is handled through SMF - and I want ALL of that cataloged by the search engines.

maestrosite.ru · May 15, 2012, 04:15:18 PM

Quote from: Kindred on May 15, 2012, 03:42:11 PM
but, apparently you were not paying attention to james either, since he clearly said that he wants the search engines to see and index the profile pages....

Quote from: James--- on May 08, 2012, 08:50:14 PM
That's a simple solution but we want guest to be able to see members profile (without registration)
and also let them index by Google.

Yes, this item had to write more. For "onpage" noindex i said above http://www.simplemachines.org/community/index.php?topic=307449.msg3334879#msg3334879
but about robots.txt rules after your post http://www.simplemachines.org/community/index.php?topic=307449.msg3335120#msg3335120 Of course, I had to clarify at once.

Quote from: Kindred on May 15, 2012, 03:42:11 PM
as for my site... 30-50% of the site is used for calendar events, galleries, articles, static pages and wiki. (among other things), but the entire site is handled through SMF - and I want ALL of that cataloged by the search engines.

gallery, wiki, etc - is not default functions, therefore, for these pages requires Allow-rules

Kindred · May 15, 2012, 04:25:35 PM

only if I added that silly "disallow" in the first place... my argument is that - in your case, that disallow might make sense, but in the case of 99% of the rest of the forums out there, it would not.

News:

Looking for noindex mod or system