Spider Board & Topic Access

Started by Kermit, December 22, 2008, 10:16:08 AM

Previous topic - Next topic

Kermit

Link to Mod

Spider Board & Topic Access

Version: 1.0 Date 2008-12-5
Compatibility: SMF 1.1.7 , SMF 2.0 Beta 4





When this option is enabled,then the selected spiders are be able to browse the site and the others guests will be
redirected to the login page,when they try to enter a board or a topic.It makes sense,when somebody wants to restrict guest access of the site topics,
but it's not a good thing for the spiders,who brings hits & new members to our site.This little modification would let us to allow spiders to browse our site,when we don't want to let other guests to browse the site



How can you activate it ?

Admin -> Features and Options -> Enable spider access (SMF 1.1.7)

Admin -> Modifications -> Enable spider access (SMF 2.0 Beta 4)

Greetz
Duncan85


Ps: Search engines such as Google, may ban users' site for using this mod, as giving more access to bots than guests is against their TOS
My Mods
Please don't PM/mail me for support,unless i invite you
Formerly known as Duncan85
Quote
"Two things are infinite: the universe and human stupidity; and I'm not sure about the the universe."

A. Einstein

karlbenson

#1
Using this mod will get you banned (aka Sandboxed) from Google and most other search engines.

It is EXPRESSLY against Googles Terms & Conditions that you must NOT show content to 'spider' guests which you don't show to human guests.  also known as Cloaking.
http://www.google.com/support/webmasters/bin/answer.py?hl=en&answer=35769

The risk of getting caught results in temporary or permanent ban.  So I would not recommend using this mod if you want to be in search engines. (Its not worth the risk).  Search engines do check for cloaking and sites do regularly get banned for doing it.

Its ALL or NOTHING. If you want to be indexed, that same content must be showable to guests (without the need for them to register)

Alternative solutions: Limit no. of page views per day per guest. (without differentiating between spiders and human guests).

Kermit

Quote from: regularexpression on December 22, 2008, 10:48:15 AM
Using this mod will get you banned (aka Sandboxed) from Google and most other search engines.

It is EXPRESSLY against Googles Terms & Conditions that you must NOT show content to 'spider' guests which you don't show to human guests.  also known as Cloaking.
http://www.google.com/support/webmasters/bin/answer.py?hl=en&answer=35769

The risk of getting caught results in temporary or permanent ban.  So I would not recommend using this mod if you want to be in search engines. (Its not worth the risk).  Search engines do check for cloaking and sites do regularly get banned for doing it.

Its ALL or NOTHING. If you want to be indexed, that same content must be showable to guests (without the need for them to register)

Alternative solutions: Limit no. of page views per day per guest. (without differentiating between spiders and human guests).

I've already added a warning message related to that issue to readme file of installation package...
My Mods
Please don't PM/mail me for support,unless i invite you
Formerly known as Duncan85
Quote
"Two things are infinite: the universe and human stupidity; and I'm not sure about the the universe."

A. Einstein

Sabre™

This looked pretty enticing till I read this
"Search engines such as Google, may ban users' site for using this mod, as giving more access to bots than guests is against their TOS"
lol

Good idea, but not really worth the risk to me :)

Keep up the good work though mate :)
Do NOT give admin and/or ftp details to just anybody, see if they are trust worthy first!!  Do your homework ;)


DeviDarkL

Great Mod ;)

Portuguese_pt translation:
<file name="$languagedir/Modifications.portuguese_pt.php" error="skip">
   <operation>
   <search position="end" />
   <add><![CDATA[
$txt['enable_spiderAccess'] = 'Autorizar acesso dos Spiders';
$txt['enabled_spider'] = 'Os spiders que deveriam ter acesso ao site <br /><i> (Como um exemplo,para os spiders <b><font color="red">google e yahoo</font></b> deve-se escrever google,yahoo) ';

]]></add>
   </operation>
</file>
<file name="$languagedir/Modifications.portuguese_pt-utf8.php" error="skip">
   <operation>
   <search position="end" />
   <add><![CDATA[
$txt['enable_spiderAccess'] = 'Autorizar acesso dos Spiders';
$txt['enabled_spider'] = 'Os spiders que deveriam ter acesso ao site <br /><i> (Como um exemplo,para os spiders <b><font color="red">google e yahoo</font></b> deve-se escrever google,yahoo) ';

]]></add>
   </operation>
</file>

Sudhakar Arjunan

Good Mod.

Earlier everyone requested this mod.

And Author have clearly updated that this may also block the access to search engines.
Working on New Mods & Themes for SMF... Will update soon... My Blog page
My Smf forum : Discuss ITAcumens :: My SMF Forum

Astro1

Who's used this and is still indexed?

Snogmarr

Hmm I  have guest access to my boards and I lost it, so I did a bit of detective work and narrowed it down to this mod?
Install this and when guests click on a board it brings up the register template,, de install it and guests can enter the boards.

Kermit

Quote from: xnathansxdadx on March 17, 2010, 07:34:06 PM
Hmm I  have guest access to my boards and I lost it, so I did a bit of detective work and narrowed it down to this mod?
Install this and when guests click on a board it brings up the register template,, de install it and guests can enter the boards.


The purpose of this mod is that, to restrict access for guests and allow for the spiders
My Mods
Please don't PM/mail me for support,unless i invite you
Formerly known as Duncan85
Quote
"Two things are infinite: the universe and human stupidity; and I'm not sure about the the universe."

A. Einstein

Arantor

But it's limiting even on boards that should be visible to guests...

Snogmarr

But the spiders are accessing the boards without this mod

as we speak

Guest      05:08:21 AM     Viewing Who's Online.
Guest    02:36:02 AM    Viewing the board index of .
Guest    01:31:38 AM    Unknown Action
Guest    12:12:32 AM    Viewing the board index of .
Guest    March 17, 2010, 10:14:57 PM    Viewing the board index of .
Guest    March 17, 2010, 07:42:34 PM    Viewing the board Test Board.
Guest    March 17, 2010, 06:28:25 PM    Viewing the topic FAC-ADMIN.

so my boards are being spidered
my only guest board is the test board, i can tell their activity because i have 130 errors in my error log from the spiders trying to access chat, affiliates etc.

Arantor

That's proof of absolutely NOTHING.

Who's Online doesn't check whether the user listed actually has permission to see those things. It just logs that an account is trying to do that.

Snogmarr

It does when I check those IP addresses back to google, bing, yahoo, alexa, etc or is that saying somebody from the google alexa yahoo page etc has accessed my site?

Arantor

It's saying that someone from that IP address has tried to access those areas, nothing more.

Snogmarr

Well thats me told then I will shut up now!


MaXiForum.cz

Hello,
is there another possibility to allow access to search engines but no for guests?
Is this mod a big risk?
Thank you

Oya

if you allow search engines access but not guests, search engines can and do come back in a way you can't easily identify then block you for cloaking results

MaXiForum.cz

I am not understand.
What do you mean please? :)

Kermit

Quote from: MaXiForum.cz on November 06, 2010, 03:42:12 PM
Hello,
is there another possibility to allow access to search engines but no for guests?

This is the logic of this mod already

Quote
Is this mod a big risk?
Thank you
Yes,it's a risk,because you can get banned from some search engines
My Mods
Please don't PM/mail me for support,unless i invite you
Formerly known as Duncan85
Quote
"Two things are infinite: the universe and human stupidity; and I'm not sure about the the universe."

A. Einstein

MaXiForum.cz

Thank you!
I can get banned from some search engines or banned from Adsense? :)
And exist another possibility to allow access to search engines but no for guests?
:)

Advertisement: