Spiders Don't Increase Topic Views

[SMFHacks.com Team]

Link to the mod

SPIDERS DON'T INCREASE TOPIC VIEWS v1.1

Donate | Comment On This Mod

Introduction
When spiders are browsing your forum, they will increase the topic views in every topic they visit.  This modification basically detects most well-known spiders , and if so, it won't increase page count.

This will give you a better idea of actually how many REAL people are viewing topics.
It will also save on a few queries.

Installation
Simple & Easy, install. It only alters a source file, therefore no theme edits/language edits will be required whatsoever.

Note: There are TWO versions of the mod for different smf versions
- SMF 2.0.x
- SMF 1.1.x version

Support
Please use the modification thread for support with this modification.
(Please don't ask me to do the edits for you)

Changelog
1.0 - 3rd August 2007
o Initial public release.
1.1 - 16th October 2007
o Improved readme
o Fixed installer bug

[jossanaijr]

Installed ok, but is there any setup?  I cannot find any changes in my adm area.

[karlbenson]

No setup/no settings/no edits required.

[jossanaijr]

Thank you.

[LostProphecy]

this looks interesting and i might try it out...

i have another problem that is related and was wondering if it might be possible to intergrate it into this mod...

my site gets alot of spiders on it and it's almost always showing something like...

60 guests, 10 members

or something like that...

the who's online is a complete nightmare and you have to scroll just to see your members...

i was wondering if there is also anyway of removing the spiders and bots from upping the guest online count and appearing in the who's online... i couldn't give a rats arse how many spiders are on my forum and i'm sure my members don't care either...

anyway just thought i'd ask here cause you seem to know about spiders and all...

~Jessi

[karlbenson]

see the googlebot and spiders mod.

[kill3r]

EXCELLENT mod. i went mad when most of my forum topics crossed 150 views with only 3-4 active members. thanks again

[kill3r]

It would be great, if you could enable us to add an ip address for which the views will not be counted. I need this, as i update the site map regularly. hope you will consider this. thanks

[karlbenson]

OK, I've uploaded v1.1 [Mod]

1.1 - 16th October 2007
o Improved readme
o Fixed installer bug

[dakotaroar]

This will affect the page view statistics too, yes?  So if a spider sees, say, a board index, that won't be added to the page views, right?

[karlbenson]

no. the mod only affects the Topic views.
eg the column which appears on the Message Index

[dakotaroar]

Darn.

[karlbenson]

I have reuploaded this mod and screenshots for it.

I am in the process of doing it for ALL my other mods aswell.  Please bear with me.

[carbonize]

Would this also stop spiders from increasing the download count of attachments? If not would it be possible to add that?

[FragaCampos]

Another great mod by karlbenson 


This affects the count of topic views, so it also affects the column of topic views in the infocenter statistics on the bottom of the forum?

[karlbenson]

This only affects counting it towards the views for each topic.

It doesn't prevent it counting for total pageviews.

[gastan]

Loaded this file okay and it shows up in the list of packages in the Package Manager.  But clicking "apply mod" or "list files" did nothing except refresh the page.  I manually installed it and so far my site is behaving okay ... no new or unexpected problems since install.  But the mod still shows as uninstalled in Package Manager.  Is this normal?  How can I be sure it's working? 

And on a side note, any plans to issue any upgrade for new spiders?  Or is that not a worry?  Or is this functionality gonna be built into v2.0?

[vbgamer45]

Updated for SMF 2.0 RC2!

[Eclipse16V]

Doesn´t work

Error in this

Code Select Expand

<operation>
<search position="replace"><![CDATA[ $smcFunc['db_query']('', '
UPDATE {db_prefix}topics
SET num_views = num_views + 1
WHERE id_topic = {int:current_topic}',
array(
'current_topic' => $topic,
)
);
]]></search>
<add><![CDATA[
// ONLY INCREASE NUMBER OF TOPIC VIEWS IF NOT A SPIDER
$is_spider = detect_spider();
if(!$is_spider) {
$smcFunc['db_query']('', "
UPDATE {db_prefix}topics
SET num_views = num_views + 1
WHERE id_topic = {int:current_topic}',
array(
'current_topic' => $topic,
)
);
]]></add>
</operation>

[vbgamer45]

Error on install? Or function does it work?

[Eclipse16V]

Here the fix

Code Select Expand

<?xml version="1.0"?>
<!DOCTYPE modification SYSTEM "http://www.simplemachines.org/xml/modification">
<modification xmlns="http://www.simplemachines.org/xml/modification" xmlns:smf="http://www.simplemachines.org/">
<name>Spiders Don't Increase Topic Views</name>
<id>karlbenson:SpidersNoTopicViewIncrease</id>
<version>1.1</version>
<homepage>http://dev.smfchat.com</homepage>

<file name="$sourcedir/Display.php">
<operation>
<search position="after"><![CDATA[// The central part of the board - topic display.
function Display()]]></search>
<add><![CDATA[
// DETECT SPIDERS
function detect_spider() {
// LIST OF SPIDERS
$known_spiders = array(
'WISENutbot', 'MSNBot', 'MSN spider', 'W3C Validator',
'Googlebot-Image', 'Googlebot', 'Mediapartners-Google', 'Openbot',
'Yahoo! Slurp', 'FAST-WebCrawler', 'Wget', 'Ask Jeeves',
'Speedy Spider', 'SurveyBot', 'IBM_Planetwide', 'OmniExplorer_Bot/6.68',
'ia_archiver', 'FAST-WebCrawler', 'Inktomi Slurp', 'Feedfetcher-Google',
'FeedBurner/1.0', 'appie', 'ping.blo.gs/2.0', 'http://www.relevantnoise.com',
'omgilibot/0.3', 'GigaBot', 'NewsGatorOnline/2.0', 'Jakarta Commons-HttpClient/3.0.1',
'Jakarta Commons-HttpClient/3.0-rc2',
);

foreach($known_spiders AS $agent) {
if (strpos(strtolower($_SERVER['HTTP_USER_AGENT']), strtolower($agent)) !== false) {
return true;
}
}
unset($known_spiders);
return false;
}
]]></add>
</operation>
<operation>
<search position="replace"><![CDATA[ $smcFunc['db_query']('', '
UPDATE {db_prefix}topics
SET num_views = num_views + 1
WHERE id_topic = {int:current_topic}',
array(
'current_topic' => $topic,
)
);
]]></search>
<add><![CDATA[
// ONLY INCREASE NUMBER OF TOPIC VIEWS IF NOT A SPIDER
$is_spider = detect_spider();
if(!$is_spider) {
$smcFunc['db_query']('', '
UPDATE {db_prefix}topics
SET num_views = num_views + 1
WHERE id_topic = {int:current_topic}',
array(
'current_topic' => $topic,
)
);
}
]]></add>
</operation>
</file>

</modification>

[vbgamer45]

Thanks has been fixed!

[Eclipse16V]

Here with more Spiders

Code Select Expand

<?xml version="1.0"?>
<!DOCTYPE modification SYSTEM "http://www.simplemachines.org/xml/modification">
<modification xmlns="http://www.simplemachines.org/xml/modification" xmlns:smf="http://www.simplemachines.org/">
<name>Spiders Don't Increase Topic Views</name>
<id>karlbenson:SpidersNoTopicViewIncrease</id>
<version>1.1</version>
<homepage>http://dev.smfchat.com</homepage>

<file name="$sourcedir/Display.php">
<operation>
<search position="after"><![CDATA[// The central part of the board - topic display.
function Display()]]></search>
<add><![CDATA[
// DETECT SPIDERS
function detect_spider() {
// LIST OF SPIDERS
$known_spiders = array(
'192.comAgent',
'ABACHOBot',
'abcdatos',
'Accoona',
'Acoon',
'AdsBot-Google',
'appie',
'asterias',
'Ask Jeeves',
'Baiduspider',
'BecomeBot',
'Bloglines',
'BlogRefsBot',
'Charlotte',
'Daumoa',
'DiscoBot',
'DuckDuckBot',
'EnaBot',
'Exabot',
'FAST-WebCrawler',
'Feedburner',
'FeedBurner/1.0',
'Feedfetcher-Google',
'FeedValidator',
'Furlbot',
'FyberSpider',
'Gaisbot',
'GeonaBot',
'GigaBot',
'Girafabot',
'Googlebot-Image',
'Googlebot',
'Googlebot-Mobile',
'GoSeeBot',
'gsa-crawler',
'GSiteCrawler',
'http://www.relevantnoise.com',
'HTTrack',
'ia_archiver',
'ia_archiver-web.archive.org',
'IBM_Planetwide',
'ichiro',
'Inktomi Slurp',
'Jakarta Commons-HttpClient/3.0.1',
'Jakarta Commons-HttpClient/3.0-rc2',
'kalooga',
'LapozzBot',
'link validator',
'Lycos_Spider',
'Mediapartners-Google',
'MJ12bot/v2',
'MLBot',
'MSNBot',
'msnbot-media',
'MSNBOT_Mobile',
'MSN spider',
'MSR-ISRCCrawler',
'msrbot',
'NaverBot',
'NewsGatorOnline/2.0',
'noxtrumbot',
'OmniExplorer_Bot/6.68',
'omgilibot/0.3',
'OnetSzukaj',
'Openbot',
'P3P Validator',
'ping.blo.gs/2.0',
'psbot',
'Scooter',
'ScoutJet',
'Scrubby',
'SearchSight',
'Seeqpod',
'ShablastBot',
'silk/1.0',
'SitiDiBot',
'slurp',
'Snapbot',
'Sogou',
'Sosospider',
'Speedy Spider',
'StackRambler',
'SurveyBot',
'Teoma',
'TinEye',
'Touche',
'twiceler',
'W3C-checklink',
'W3C-mobileOK',
'W3C_CSS_Validator',
'W3C Validator',
'WebAlta',
'Websnapr',
'Wget',
'WISENutbot',
'wisponbot',
'Y!J',
'yacybot',
'yahoo',
'Yahoo-MMCrawler',
'YahooSeeker/M1A1-R2D2',
'Yahoo! Slurp',
'Yeti',
'YodaoBot',
);

foreach($known_spiders AS $agent) {
if (strpos(strtolower($_SERVER['HTTP_USER_AGENT']), strtolower($agent)) !== false) {
return true;
}
}
unset($known_spiders);
return false;
}
]]></add>
</operation>
<operation>
<search position="replace"><![CDATA[ $smcFunc['db_query']('', '
UPDATE {db_prefix}topics
SET num_views = num_views + 1
WHERE id_topic = {int:current_topic}',
array(
'current_topic' => $topic,
)
);
]]></search>
<add><![CDATA[
// ONLY INCREASE NUMBER OF TOPIC VIEWS IF NOT A SPIDER
$is_spider = detect_spider();
if(!$is_spider) {
$smcFunc['db_query']('', '
UPDATE {db_prefix}topics
SET num_views = num_views + 1
WHERE id_topic = {int:current_topic}',
array(
'current_topic' => $topic,
)
);
}
]]></add>
</operation>
</file>

</modification>

[vbgamer45]

Updated for SMF 2.0 RC3!

[distante]

Here with more Spiders

Code Select Expand

<?xml version="1.0"?>
<!DOCTYPE modification SYSTEM "http://www.simplemachines.org/xml/modification">
<modification xmlns="http://www.simplemachines.org/xml/modification" xmlns:smf="http://www.simplemachines.org/">
<name>Spiders Don't Increase Topic Views</name>
<id>karlbenson:SpidersNoTopicViewIncrease</id>
<version>1.1</version>
<homepage>http://dev.smfchat.com</homepage>

<file name="$sourcedir/Display.php">
<operation>
<search position="after"><![CDATA[// The central part of the board - topic display.
function Display()]]></search>
<add><![CDATA[
// DETECT SPIDERS
function detect_spider() {
// LIST OF SPIDERS
$known_spiders = array(
'192.comAgent',
'ABACHOBot',
'abcdatos',
'Accoona',
'Acoon',
'AdsBot-Google',
'appie',
'asterias',
'Ask Jeeves',
'Baiduspider',
'BecomeBot',
'Bloglines',
'BlogRefsBot',
'Charlotte',
'Daumoa',
'DiscoBot',
'DuckDuckBot',
'EnaBot',
'Exabot',
'FAST-WebCrawler',
'Feedburner',
'FeedBurner/1.0',
'Feedfetcher-Google',
'FeedValidator',
'Furlbot',
'FyberSpider',
'Gaisbot',
'GeonaBot',
'GigaBot',
'Girafabot',
'Googlebot-Image',
'Googlebot',
'Googlebot-Mobile',
'GoSeeBot',
'gsa-crawler',
'GSiteCrawler',
'http://www.relevantnoise.com',
'HTTrack',
'ia_archiver',
'ia_archiver-web.archive.org',
'IBM_Planetwide',
'ichiro',
'Inktomi Slurp',
'Jakarta Commons-HttpClient/3.0.1',
'Jakarta Commons-HttpClient/3.0-rc2',
'kalooga',
'LapozzBot',
'link validator',
'Lycos_Spider',
'Mediapartners-Google',
'MJ12bot/v2',
'MLBot',
'MSNBot',
'msnbot-media',
'MSNBOT_Mobile',
'MSN spider',
'MSR-ISRCCrawler',
'msrbot',
'NaverBot',
'NewsGatorOnline/2.0',
'noxtrumbot',
'OmniExplorer_Bot/6.68',
'omgilibot/0.3',
'OnetSzukaj',
'Openbot',
'P3P Validator',
'ping.blo.gs/2.0',
'psbot',
'Scooter',
'ScoutJet',
'Scrubby',
'SearchSight',
'Seeqpod',
'ShablastBot',
'silk/1.0',
'SitiDiBot',
'slurp',
'Snapbot',
'Sogou',
'Sosospider',
'Speedy Spider',
'StackRambler',
'SurveyBot',
'Teoma',
'TinEye',
'Touche',
'twiceler',
'W3C-checklink',
'W3C-mobileOK',
'W3C_CSS_Validator',
'W3C Validator',
'WebAlta',
'Websnapr',
'Wget',
'WISENutbot',
'wisponbot',
'Y!J',
'yacybot',
'yahoo',
'Yahoo-MMCrawler',
'YahooSeeker/M1A1-R2D2',
'Yahoo! Slurp',
'Yeti',
'YodaoBot',
);

foreach($known_spiders AS $agent) {
if (strpos(strtolower($_SERVER['HTTP_USER_AGENT']), strtolower($agent)) !== false) {
return true;
}
}
unset($known_spiders);
return false;
}
]]></add>
</operation>
<operation>
<search position="replace"><![CDATA[ $smcFunc['db_query']('', '
UPDATE {db_prefix}topics
SET num_views = num_views + 1
WHERE id_topic = {int:current_topic}',
array(
'current_topic' => $topic,
)
);
]]></search>
<add><![CDATA[
// ONLY INCREASE NUMBER OF TOPIC VIEWS IF NOT A SPIDER
$is_spider = detect_spider();
if(!$is_spider) {
$smcFunc['db_query']('', '
UPDATE {db_prefix}topics
SET num_views = num_views + 1
WHERE id_topic = {int:current_topic}',
array(
'current_topic' => $topic,
)
);
}
]]></add>
</operation>
</file>

</modification>

has this been added to the install mod?

[vbgamer45]

I don't remember doing so.

[distante]

well I will add them manually

[karanthsrihari]

I installed the mod. I posted one article in twitter and in next second i received some 20 bots. And this was registered in view count of the post. So i guess this mod is not working properly?

[vbgamer45]

It depends on the bots. It checks for major search engines like google, yahoo

[mariusfv]

Is this default in RC5? I have a lot of robots views my forum but they don't increase the number of views and I want to increase?
Vbulletin have for sure a mod who icrease number of views(fake it) because I see on some forum a post with less then 5 minutes with over 5000 number of views, have smf this kind of mod or is against their policy?

[vbgamer45]

Not built into RC5

[Iomega0318]

As I seem to have hundreds of bot visits that still show as guests, until I can get them all sorted is there any way to not count guests views as well?

I think the majority of mine are coming from Twitter, I never saw this many bots until I started posting links on there.. does anyone have a list of bots that come from Twitter?


Also for the bot list, does the bot name need to be in the database or does it just find that bot by name on the forum to block it's view count?

[Iomega0318]

Here is another update with even more bots..

Code Select Expand

<?xml version="1.0"?>
<!DOCTYPE modification SYSTEM "http://www.simplemachines.org/xml/modification">
<modification xmlns="http://www.simplemachines.org/xml/modification" xmlns:smf="http://www.simplemachines.org/">
<name>Spiders Don't Increase Topic Views</name>
<id>karlbenson:SpidersNoTopicViewIncrease</id>
<version>1.1</version>
<homepage>http://dev.smfchat.com</homepage>

<file name="$sourcedir/Display.php">
<operation>
<search position="after"><![CDATA[// The central part of the board - topic display.
function Display()]]></search>
<add><![CDATA[
// DETECT SPIDERS
function detect_spider() {
// LIST OF SPIDERS
$known_spiders = array(
'192.comAgent',
'ABACHOBot',
'abcdatos',
'Accoona',
'Acoon',
'AdsBot-Google',
'appie',
'asterias',
'Ask Jeeves',
'Baiduspider',
'BecomeBot',
'Bloglines',
'BlogRefsBot',
'Charlotte',
'Daumoa',
'DiscoBot',
'DuckDuckBot',
'EnaBot',
'Exabot',
'FAST-WebCrawler',
'Feedburner',
'FeedBurner/1.0',
'Feedfetcher-Google',
'FeedValidator',
'Furlbot',
'FyberSpider',
'Gaisbot',
'GeonaBot',
'Gigabot',
'GigaBot',
'Girafabot',
'googlebot',
'Googlebot-Image',
'Googlebot',
'Googlebot-Mobile',
'GoSeeBot',
'gsa-crawler',
'GSiteCrawler',
'http://www.relevantnoise.com',
'HTTrack',
'ia_archiver',
'ia_archiver-web.archive.org',
'IBM_Planetwide',
'ichiro',
'Inktomi Slurp',
'Jakarta Commons-HttpClient/3.0.1',
'Jakarta Commons-HttpClient/3.0-rc2',
'kalooga',
'LapozzBot',
'link validator',
'Lycos_Spider',
'Mediapartners-Google',
'MJ12bot/v2',
'MLBot',
'msn',
'MSNBot',
'msnbot-media',
'MSNBOT_Mobile',
'MSN spider',
'MSR-ISRCCrawler',
'msrbot',
'NaverBot',
'NewsGatorOnline/2.0',
'noxtrumbot',
'omgilibot',
'OmniExplorer_Bot',
'OmniExplorer_Bot/6.68',
'omgilibot/0.3',
'OnetSzukaj',
'Openbot',
'P3P Validator',
'ping.blo.gs/2.0',
'psbot',
'Scooter',
'ScoutJet',
'Scrubby',
'SearchSight',
'Seeqpod',
'ShablastBot',
'silk/1.0',
'SitiDiBot',
'slurp',
'Snapbot',
'Sogou',
'Sosospider',
'Speedy Spider',
'StackRambler',
'SurveyBot',
'Teoma',
'TinEye',
'Touche',
'twiceler',
'Twitter',
'Twitter Bot',
'W3C-checklink',
'W3C-mobileOK',
'W3C_CSS_Validator',
'W3C_Validator',
'W3C Validator',
'WebAlta',
'Websnapr',
'Wget',
'WISENutbot',
'wisponbot',
'Y!J',
'yacybot',
'yahoo',
'Yahoo-MMCrawler',
'YahooSeeker/M1A1-R2D2',
'Yahoo! Slurp',
'Yeti',
'YodaoBot',
);

foreach($known_spiders AS $agent) {
if (strpos(strtolower($_SERVER['HTTP_USER_AGENT']), strtolower($agent)) !== false) {
return true;
}
}
unset($known_spiders);
return false;
}
]]></add>
</operation>
<operation>
<search position="replace"><![CDATA[ $smcFunc['db_query']('', '
UPDATE {db_prefix}topics
SET num_views = num_views + 1
WHERE id_topic = {int:current_topic}',
array(
'current_topic' => $topic,
)
);
]]></search>
<add><![CDATA[
// ONLY INCREASE NUMBER OF TOPIC VIEWS IF NOT A SPIDER
$is_spider = detect_spider();
if(!$is_spider) {
$smcFunc['db_query']('', '
UPDATE {db_prefix}topics
SET num_views = num_views + 1
WHERE id_topic = {int:current_topic}',
array(
'current_topic' => $topic,
)
);
}
]]></add>
</operation>
</file>

</modification>

[dougiefresh]

Thank you for the wonderful mod!  I thought I'd let everybody know that when I updated my forum from 2.0 RC5 to 2.0 Gold, I had to reinstall all my mods.  This mod installed without issues and without emulation.  Hope this info helps someone!

[FireDitto]

Is it possible to have the mod read off the spiders listed in the spider list of 2.0? It would be easier than having to update the files all the time... And I get so many bots crawling my site, that this mod is barely effective.

[SMFHacks.com Team]

Is it possible to have the mod read off the spiders listed in the spider list of 2.0? It would be easier than having to update the files all the time... And I get so many bots crawling my site, that this mod is barely effective.

It would use another db query  but could be possible.

What kind of bots are getting added? The big ones are bing,google,baidu

[FireDitto]

Odds and ends, mostly. Nothing that strike me as big end bots, but try are there anyway.

A lot of them are those brought in by Twitter, some others I don't know. I have a member base with pretty diverse roots, so some of the spiders make me go o.O haha but I've been adding them all and because they aren't big bots, they aren't going t be on the list, do I thought it would be easier for the mod to draw from te spider listing.




-iPhone auto spell is responsible for weird sentences.

[vbgamer45]

If you have a list I can add more. I use the spider check for a lot of places in my forums for better performance no reason to show information to bots

[FireDitto]

Um, I think this is all of them. I may have missed some, or included a couple that were pre-loaded but I'm pretty sure these are all ones I added to the Spiders option on SMF.
Spider name; User Agent

Code Select Expand

Ahrefs; AhrefsBot
Baidu; Baiduspider
Bing Bot; Bingbot/2.0
Crowsnest; Crowsnest
Embedly; Embedly
Exabot; Exabot
Facebook Bot; facebookxternalhit/1.1
In A Gist; InAGist URL Resolver
JS Kit; JS-Kit URL Resolver
Linked In Bot; LinkedINBot/1.0
Mcafee; MFE_expand
ML-RPC; ML-RPC
MLBot; MLBot
Mozilla; Mozilla/0.19 Beta (Windows)
Orca Ringmaker; Orca Ringmaker V3.0
ScoutJet; ScoutJet
Seznam Bot; SeznamBot/3.0
strawberryj.am; strawberryj.am
TweetmemeBot; TweetmemeBot
Twinkle Bot; Twinkle/1.0
Twitterbot; Twitterbot
UnwindFetchor; UnwindFetchor
Yandex; Yandex

[SMFHacks.com Team]

updated to support SMF 2.1.x
And added more spiders to block for topic views.

[Randem]

Made a minor modification for SMF 2.1 RC2 support.

Code Select Expand


<operation>
<search position="replace"><![CDATA[ $smcFunc['db_query']('', '
UPDATE {db_prefix}topics
SET num_views = num_views + 1
WHERE id_topic = {int:current_topic}
AND id_member_started != {int:current_member}',
array(
'current_topic' => $topic,
'current_member' => $user_info['id'],
)
);
]]></search>
<add><![CDATA[
// ONLY INCREASE NUMBER OF TOPIC VIEWS IF NOT A SPIDER
$is_spider = detect_spider();
if(!$is_spider) {
$smcFunc['db_query']('', '
UPDATE {db_prefix}topics
SET num_views = num_views + 1
WHERE id_topic = {int:current_topic}
AND id_member_started != {int:current_member}',
array(
'current_topic' => $topic,
'current_member' => $user_info['id'],
)
);
}
]]></add>
</operation>


[Strannik64]

not installed in 2.1 RC4 

[vbgamer45]

Haven't updated it yet.

[vbgamer45]

Update added support for SMF 2.1 RC4

[Strannik64]

Update added support for SMF 2.1 RC4

thank you, I will install 

[Strannik64]

error anyway

You cannot view this attachment.

[shadav]

that just means that something else you installed changed what this mod is looking for

https://wiki.simplemachines.org/smf/Error_in_mod_installation

would need to know what the exact error is, what the mod is looking for, what the mod is trying to add/change, and what is actually in your file