Simple Machines Community Forum

Link to the mod (https://custom.simplemachines.org/index.php?mod=882)

---

---

Introduction
When spiders are browsing your forum, they will increase the topic views in every topic they visit.  This modification basically detects most well-known spiders , and if so, it won't increase page count.

This will give you a better idea of actually how many REAL people are viewing topics.
It will also save on a few queries.

Installation
Simple & Easy, install. It only alters a source file, therefore no theme edits/language edits will be required whatsoever.

Note: There are TWO versions of the mod for different smf versions
- SMF 2.0.x
- SMF 1.1.x version

Support
Please use the modification thread for support with this modification.
(Please don't ask me to do the edits for you)

Changelog
1.0 - 3rd August 2007
o Initial public release.
1.1 - 16th October 2007
o Improved readme
o Fixed installer bug

Installed ok, but is there any setup?  I cannot find any changes in my adm area.

No setup/no settings/no edits required.

Thank you.

this looks interesting and i might try it out...

i have another problem that is related and was wondering if it might be possible to intergrate it into this mod...

my site gets alot of spiders on it and it's almost always showing something like...

60 guests, 10 members

or something like that...

the who's online is a complete nightmare and you have to scroll just to see your members...

i was wondering if there is also anyway of removing the spiders and bots from upping the guest online count and appearing in the who's online... i couldn't give a rats arse how many spiders are on my forum and i'm sure my members don't care either...

anyway just thought i'd ask here cause you seem to know about spiders and all...

~Jessi

see the googlebot and spiders mod.

EXCELLENT mod. i went mad when most of my forum topics crossed 150 views with only 3-4 active members. thanks again :)

It would be great, if you could enable us to add an ip address for which the views will not be counted. I need this, as i update the site map regularly. hope you will consider this. thanks

OK, I've uploaded v1.1 [Mod] (http://custom.simplemachines.org/mods/index.php?mod=882)

1.1 - 16th October 2007
o Improved readme
o Fixed installer bug

This will affect the page view statistics too, yes?  So if a spider sees, say, a board index, that won't be added to the page views, right?

no. the mod only affects the Topic views.
eg the column which appears on the Message Index

Darn.

I have reuploaded this mod and screenshots for it.

I am in the process of doing it for ALL my other mods aswell.  Please bear with me.

Would this also stop spiders from increasing the download count of attachments? If not would it be possible to add that?

Another great mod by karlbenson  8)


This affects the count of topic views, so it also affects the column of topic views in the infocenter statistics on the bottom of the forum?

This only affects counting it towards the views for each topic.

It doesn't prevent it counting for total pageviews.

Loaded this file okay and it shows up in the list of packages in the Package Manager.  But clicking "apply mod" or "list files" did nothing except refresh the page.  I manually installed it and so far my site is behaving okay ... no new or unexpected problems since install.  But the mod still shows as uninstalled in Package Manager.  Is this normal?  How can I be sure it's working? 

And on a side note, any plans to issue any upgrade for new spiders?  Or is that not a worry?  Or is this functionality gonna be built into v2.0?

Updated for SMF 2.0 RC2!

Doesn´t work

Error in this

<operation>
<search position="replace"><![CDATA[ $smcFunc['db_query']('', '
UPDATE {db_prefix}topics
SET num_views = num_views + 1
WHERE id_topic = {int:current_topic}',
array(
'current_topic' => $topic,
)
);
]]></search>
<add><![CDATA[
// ONLY INCREASE NUMBER OF TOPIC VIEWS IF NOT A SPIDER
$is_spider = detect_spider();
if(!$is_spider) {
$smcFunc['db_query']('', "
UPDATE {db_prefix}topics
SET num_views = num_views + 1
WHERE id_topic = {int:current_topic}',
array(
'current_topic' => $topic,
)
);
]]></add>
</operation>

Error on install? Or function does it work?

Here the fix
<?xml version="1.0"?>
<!DOCTYPE modification SYSTEM "http://www.simplemachines.org/xml/modification">
<modification xmlns="http://www.simplemachines.org/xml/modification" xmlns:smf="http://www.simplemachines.org/">
<name>Spiders Don't Increase Topic Views</name>
<id>karlbenson:SpidersNoTopicViewIncrease</id>
<version>1.1</version>
<homepage>http://dev.smfchat.com</homepage>

<file name="$sourcedir/Display.php">
<operation>
<search position="after"><![CDATA[// The central part of the board - topic display.
function Display()]]></search>
<add><![CDATA[
// DETECT SPIDERS
function detect_spider() {
// LIST OF SPIDERS
$known_spiders = array(
'WISENutbot', 'MSNBot', 'MSN spider', 'W3C Validator',
'Googlebot-Image', 'Googlebot', 'Mediapartners-Google', 'Openbot',
'Yahoo! Slurp', 'FAST-WebCrawler', 'Wget', 'Ask Jeeves',
'Speedy Spider', 'SurveyBot', 'IBM_Planetwide', 'OmniExplorer_Bot/6.68',
'ia_archiver', 'FAST-WebCrawler', 'Inktomi Slurp', 'Feedfetcher-Google',
'FeedBurner/1.0', 'appie', 'ping.blo.gs/2.0', 'http://www.relevantnoise.com',
'omgilibot/0.3', 'GigaBot', 'NewsGatorOnline/2.0', 'Jakarta Commons-HttpClient/3.0.1',
'Jakarta Commons-HttpClient/3.0-rc2',
);

foreach($known_spiders AS $agent) {
if (strpos(strtolower($_SERVER['HTTP_USER_AGENT']), strtolower($agent)) !== false) {
return true;
}
}
unset($known_spiders);
return false;
}
]]></add>
</operation>
<operation>
<search position="replace"><![CDATA[ $smcFunc['db_query']('', '
UPDATE {db_prefix}topics
SET num_views = num_views + 1
WHERE id_topic = {int:current_topic}',
array(
'current_topic' => $topic,
)
);
]]></search>
<add><![CDATA[
// ONLY INCREASE NUMBER OF TOPIC VIEWS IF NOT A SPIDER
$is_spider = detect_spider();
if(!$is_spider) {
$smcFunc['db_query']('', '
UPDATE {db_prefix}topics
SET num_views = num_views + 1
WHERE id_topic = {int:current_topic}',
array(
'current_topic' => $topic,
)
);
}
]]></add>
</operation>
</file>

</modification>

Thanks has been fixed!

Here with more Spiders
<?xml version="1.0"?>
<!DOCTYPE modification SYSTEM "http://www.simplemachines.org/xml/modification">
<modification xmlns="http://www.simplemachines.org/xml/modification" xmlns:smf="http://www.simplemachines.org/">
<name>Spiders Don't Increase Topic Views</name>
<id>karlbenson:SpidersNoTopicViewIncrease</id>
<version>1.1</version>
<homepage>http://dev.smfchat.com</homepage>

<file name="$sourcedir/Display.php">
<operation>
<search position="after"><![CDATA[// The central part of the board - topic display.
function Display()]]></search>
<add><![CDATA[
// DETECT SPIDERS
function detect_spider() {
// LIST OF SPIDERS
$known_spiders = array(
'192.comAgent',
'ABACHOBot',
'abcdatos',
'Accoona',
'Acoon',
'AdsBot-Google',
'appie',
'asterias',
'Ask Jeeves',
'Baiduspider',
'BecomeBot',
'Bloglines',
'BlogRefsBot',
'Charlotte',
'Daumoa',
'DiscoBot',
'DuckDuckBot',
'EnaBot',
'Exabot',
'FAST-WebCrawler',
'Feedburner',
'FeedBurner/1.0',
'Feedfetcher-Google',
'FeedValidator',
'Furlbot',
'FyberSpider',
'Gaisbot',
'GeonaBot',
'GigaBot',
'Girafabot',
'Googlebot-Image',
'Googlebot',
'Googlebot-Mobile',
'GoSeeBot',
'gsa-crawler',
'GSiteCrawler',
'http://www.relevantnoise.com',
'HTTrack',
'ia_archiver',
'ia_archiver-web.archive.org',
'IBM_Planetwide',
'ichiro',
'Inktomi Slurp',
'Jakarta Commons-HttpClient/3.0.1',
'Jakarta Commons-HttpClient/3.0-rc2',
'kalooga',
'LapozzBot',
'link validator',
'Lycos_Spider',
'Mediapartners-Google',
'MJ12bot/v2',
'MLBot',
'MSNBot',
'msnbot-media',
'MSNBOT_Mobile',
'MSN spider',
'MSR-ISRCCrawler',
'msrbot',
'NaverBot',
'NewsGatorOnline/2.0',
'noxtrumbot',
'OmniExplorer_Bot/6.68',
'omgilibot/0.3',
'OnetSzukaj',
'Openbot',
'P3P Validator',
'ping.blo.gs/2.0',
'psbot',
'Scooter',
'ScoutJet',
'Scrubby',
'SearchSight',
'Seeqpod',
'ShablastBot',
'silk/1.0',
'SitiDiBot',
'slurp',
'Snapbot',
'Sogou',
'Sosospider',
'Speedy Spider',
'StackRambler',
'SurveyBot',
'Teoma',
'TinEye',
'Touche',
'twiceler',
'W3C-checklink',
'W3C-mobileOK',
'W3C_CSS_Validator',
'W3C Validator',
'WebAlta',
'Websnapr',
'Wget',
'WISENutbot',
'wisponbot',
'Y!J',
'yacybot',
'yahoo',
'Yahoo-MMCrawler',
'YahooSeeker/M1A1-R2D2',
'Yahoo! Slurp',
'Yeti',
'YodaoBot',
);

foreach($known_spiders AS $agent) {
if (strpos(strtolower($_SERVER['HTTP_USER_AGENT']), strtolower($agent)) !== false) {
return true;
}
}
unset($known_spiders);
return false;
}
]]></add>
</operation>
<operation>
<search position="replace"><![CDATA[ $smcFunc['db_query']('', '
UPDATE {db_prefix}topics
SET num_views = num_views + 1
WHERE id_topic = {int:current_topic}',
array(
'current_topic' => $topic,
)
);
]]></search>
<add><![CDATA[
// ONLY INCREASE NUMBER OF TOPIC VIEWS IF NOT A SPIDER
$is_spider = detect_spider();
if(!$is_spider) {
$smcFunc['db_query']('', '
UPDATE {db_prefix}topics
SET num_views = num_views + 1
WHERE id_topic = {int:current_topic}',
array(
'current_topic' => $topic,
)
);
}
]]></add>
</operation>
</file>

</modification>

Updated for SMF 2.0 RC3!

Quote from: Eclipse16V on January 24, 2010, 11:13:21 AM
Here with more Spiders

Code Select Expand

<?xml version="1.0"?>
<!DOCTYPE modification SYSTEM "http://www.simplemachines.org/xml/modification">
<modification xmlns="http://www.simplemachines.org/xml/modification" xmlns:smf="http://www.simplemachines.org/">
<name>Spiders Don't Increase Topic Views</name>
<id>karlbenson:SpidersNoTopicViewIncrease</id>
<version>1.1</version>
<homepage>http://dev.smfchat.com</homepage>

<file name="$sourcedir/Display.php">
<operation>
<search position="after"><![CDATA[// The central part of the board - topic display.
function Display()]]></search>
<add><![CDATA[
// DETECT SPIDERS
function detect_spider() {
// LIST OF SPIDERS
$known_spiders = array(
'192.comAgent',
'ABACHOBot',
'abcdatos',
'Accoona',
'Acoon',
'AdsBot-Google',
'appie',
'asterias',
'Ask Jeeves',
'Baiduspider',
'BecomeBot',
'Bloglines',
'BlogRefsBot',
'Charlotte',
'Daumoa',
'DiscoBot',
'DuckDuckBot',
'EnaBot',
'Exabot',
'FAST-WebCrawler',
'Feedburner',
'FeedBurner/1.0',
'Feedfetcher-Google',
'FeedValidator',
'Furlbot',
'FyberSpider',
'Gaisbot',
'GeonaBot',
'GigaBot',
'Girafabot',
'Googlebot-Image',
'Googlebot',
'Googlebot-Mobile',
'GoSeeBot',
'gsa-crawler',
'GSiteCrawler',
'http://www.relevantnoise.com',
'HTTrack',
'ia_archiver',
'ia_archiver-web.archive.org',
'IBM_Planetwide',
'ichiro',
'Inktomi Slurp',
'Jakarta Commons-HttpClient/3.0.1',
'Jakarta Commons-HttpClient/3.0-rc2',
'kalooga',
'LapozzBot',
'link validator',
'Lycos_Spider',
'Mediapartners-Google',
'MJ12bot/v2',
'MLBot',
'MSNBot',
'msnbot-media',
'MSNBOT_Mobile',
'MSN spider',
'MSR-ISRCCrawler',
'msrbot',
'NaverBot',
'NewsGatorOnline/2.0',
'noxtrumbot',
'OmniExplorer_Bot/6.68',
'omgilibot/0.3',
'OnetSzukaj',
'Openbot',
'P3P Validator',
'ping.blo.gs/2.0',
'psbot',
'Scooter',
'ScoutJet',
'Scrubby',
'SearchSight',
'Seeqpod',
'ShablastBot',
'silk/1.0',
'SitiDiBot',
'slurp',
'Snapbot',
'Sogou',
'Sosospider',
'Speedy Spider',
'StackRambler',
'SurveyBot',
'Teoma',
'TinEye',
'Touche',
'twiceler',
'W3C-checklink',
'W3C-mobileOK',
'W3C_CSS_Validator',
'W3C Validator',
'WebAlta',
'Websnapr',
'Wget',
'WISENutbot',
'wisponbot',
'Y!J',
'yacybot',
'yahoo',
'Yahoo-MMCrawler',
'YahooSeeker/M1A1-R2D2',
'Yahoo! Slurp',
'Yeti',
'YodaoBot',
);

foreach($known_spiders AS $agent) {
if (strpos(strtolower($_SERVER['HTTP_USER_AGENT']), strtolower($agent)) !== false) {
return true;
}
}
unset($known_spiders);
return false;
}
]]></add>
</operation>
<operation>
<search position="replace"><![CDATA[ $smcFunc['db_query']('', '
UPDATE {db_prefix}topics
SET num_views = num_views + 1
WHERE id_topic = {int:current_topic}',
array(
'current_topic' => $topic,
)
);
]]></search>
<add><![CDATA[
// ONLY INCREASE NUMBER OF TOPIC VIEWS IF NOT A SPIDER
$is_spider = detect_spider();
if(!$is_spider) {
$smcFunc['db_query']('', '
UPDATE {db_prefix}topics
SET num_views = num_views + 1
WHERE id_topic = {int:current_topic}',
array(
'current_topic' => $topic,
)
);
}
]]></add>
</operation>
</file>

</modification>

has this been added to the install mod? :P

I don't remember doing so.

well I will add them manually :P

I installed the mod. I posted one article in twitter and in next second i received some 20 bots. And this was registered in view count of the post. So i guess this mod is not working properly?

It depends on the bots. It checks for major search engines like google, yahoo

Is this default in RC5? I have a lot of robots views my forum but they don't increase the number of views and I want to increase?
Vbulletin have for sure a mod who icrease number of views(fake it) because I see on some forum a post with less then 5 minutes with over 5000 number of views, have smf this kind of mod or is against their policy?

Not built into RC5

As I seem to have hundreds of bot visits that still show as guests, until I can get them all sorted is there any way to not count guests views as well?

I think the majority of mine are coming from Twitter, I never saw this many bots until I started posting links on there.. does anyone have a list of bots that come from Twitter?


Also for the bot list, does the bot name need to be in the database or does it just find that bot by name on the forum to block it's view count?

Here is another update with even more bots..

<?xml version="1.0"?>
<!DOCTYPE modification SYSTEM "http://www.simplemachines.org/xml/modification">
<modification xmlns="http://www.simplemachines.org/xml/modification" xmlns:smf="http://www.simplemachines.org/">
<name>Spiders Don't Increase Topic Views</name>
<id>karlbenson:SpidersNoTopicViewIncrease</id>
<version>1.1</version>
<homepage>http://dev.smfchat.com</homepage>

<file name="$sourcedir/Display.php">
<operation>
<search position="after"><![CDATA[// The central part of the board - topic display.
function Display()]]></search>
<add><![CDATA[
// DETECT SPIDERS
function detect_spider() {
// LIST OF SPIDERS
$known_spiders = array(
'192.comAgent',
'ABACHOBot',
'abcdatos',
'Accoona',
'Acoon',
'AdsBot-Google',
'appie',
'asterias',
'Ask Jeeves',
'Baiduspider',
'BecomeBot',
'Bloglines',
'BlogRefsBot',
'Charlotte',
'Daumoa',
'DiscoBot',
'DuckDuckBot',
'EnaBot',
'Exabot',
'FAST-WebCrawler',
'Feedburner',
'FeedBurner/1.0',
'Feedfetcher-Google',
'FeedValidator',
'Furlbot',
'FyberSpider',
'Gaisbot',
'GeonaBot',
'Gigabot',
'GigaBot',
'Girafabot',
'googlebot',
'Googlebot-Image',
'Googlebot',
'Googlebot-Mobile',
'GoSeeBot',
'gsa-crawler',
'GSiteCrawler',
'http://www.relevantnoise.com',
'HTTrack',
'ia_archiver',
'ia_archiver-web.archive.org',
'IBM_Planetwide',
'ichiro',
'Inktomi Slurp',
'Jakarta Commons-HttpClient/3.0.1',
'Jakarta Commons-HttpClient/3.0-rc2',
'kalooga',
'LapozzBot',
'link validator',
'Lycos_Spider',
'Mediapartners-Google',
'MJ12bot/v2',
'MLBot',
'msn',
'MSNBot',
'msnbot-media',
'MSNBOT_Mobile',
'MSN spider',
'MSR-ISRCCrawler',
'msrbot',
'NaverBot',
'NewsGatorOnline/2.0',
'noxtrumbot',
'omgilibot',
'OmniExplorer_Bot',
'OmniExplorer_Bot/6.68',
'omgilibot/0.3',
'OnetSzukaj',
'Openbot',
'P3P Validator',
'ping.blo.gs/2.0',
'psbot',
'Scooter',
'ScoutJet',
'Scrubby',
'SearchSight',
'Seeqpod',
'ShablastBot',
'silk/1.0',
'SitiDiBot',
'slurp',
'Snapbot',
'Sogou',
'Sosospider',
'Speedy Spider',
'StackRambler',
'SurveyBot',
'Teoma',
'TinEye',
'Touche',
'twiceler',
'Twitter',
'Twitter Bot',
'W3C-checklink',
'W3C-mobileOK',
'W3C_CSS_Validator',
'W3C_Validator',
'W3C Validator',
'WebAlta',
'Websnapr',
'Wget',
'WISENutbot',
'wisponbot',
'Y!J',
'yacybot',
'yahoo',
'Yahoo-MMCrawler',
'YahooSeeker/M1A1-R2D2',
'Yahoo! Slurp',
'Yeti',
'YodaoBot',
);

foreach($known_spiders AS $agent) {
if (strpos(strtolower($_SERVER['HTTP_USER_AGENT']), strtolower($agent)) !== false) {
return true;
}
}
unset($known_spiders);
return false;
}
]]></add>
</operation>
<operation>
<search position="replace"><![CDATA[ $smcFunc['db_query']('', '
UPDATE {db_prefix}topics
SET num_views = num_views + 1
WHERE id_topic = {int:current_topic}',
array(
'current_topic' => $topic,
)
);
]]></search>
<add><![CDATA[
// ONLY INCREASE NUMBER OF TOPIC VIEWS IF NOT A SPIDER
$is_spider = detect_spider();
if(!$is_spider) {
$smcFunc['db_query']('', '
UPDATE {db_prefix}topics
SET num_views = num_views + 1
WHERE id_topic = {int:current_topic}',
array(
'current_topic' => $topic,
)
);
}
]]></add>
</operation>
</file>

</modification>

Thank you for the wonderful mod!  I thought I'd let everybody know that when I updated my forum from 2.0 RC5 to 2.0 Gold, I had to reinstall all my mods.  This mod installed without issues and without emulation.  Hope this info helps someone!

Is it possible to have the mod read off the spiders listed in the spider list of 2.0? It would be easier than having to update the files all the time... And I get so many bots crawling my site, that this mod is barely effective. :(

Quote from: FireDitto on June 14, 2012, 09:06:37 AM
Is it possible to have the mod read off the spiders listed in the spider list of 2.0? It would be easier than having to update the files all the time... And I get so many bots crawling my site, that this mod is barely effective. :(

It would use another db query  but could be possible.

What kind of bots are getting added? The big ones are bing,google,baidu

Odds and ends, mostly. Nothing that strike me as big end bots, but try are there anyway.

A lot of them are those brought in by Twitter, some others I don't know. I have a member base with pretty diverse roots, so some of the spiders make me go o.O haha but I've been adding them all and because they aren't big bots, they aren't going t be on the list, do I thought it would be easier for the mod to draw from te spider listing.




-iPhone auto spell is responsible for weird sentences. :D

If you have a list I can add more. I use the spider check for a lot of places in my forums for better performance no reason to show information to bots

Um, I think this is all of them. I may have missed some, or included a couple that were pre-loaded but I'm pretty sure these are all ones I added to the Spiders option on SMF.
Spider name; User Agent

Ahrefs; AhrefsBot
Baidu; Baiduspider
Bing Bot; Bingbot/2.0
Crowsnest; Crowsnest
Embedly; Embedly
Exabot; Exabot
Facebook Bot; facebookxternalhit/1.1
In A Gist; InAGist URL Resolver
JS Kit; JS-Kit URL Resolver
Linked In Bot; LinkedINBot/1.0
Mcafee; MFE_expand
ML-RPC; ML-RPC
MLBot; MLBot
Mozilla; Mozilla/0.19 Beta (Windows)
Orca Ringmaker; Orca Ringmaker V3.0
ScoutJet; ScoutJet
Seznam Bot; SeznamBot/3.0
strawberryj.am; strawberryj.am
TweetmemeBot; TweetmemeBot
Twinkle Bot; Twinkle/1.0
Twitterbot; Twitterbot
UnwindFetchor; UnwindFetchor
Yandex; Yandex

updated to support SMF 2.1.x
And added more spiders to block for topic views.

Made a minor modification for SMF 2.1 RC2 support.

<operation>
<search position="replace"><![CDATA[ $smcFunc['db_query']('', '
UPDATE {db_prefix}topics
SET num_views = num_views + 1
WHERE id_topic = {int:current_topic}
AND id_member_started != {int:current_member}',
array(
'current_topic' => $topic,
'current_member' => $user_info['id'],
)
);
]]></search>
<add><![CDATA[
// ONLY INCREASE NUMBER OF TOPIC VIEWS IF NOT A SPIDER
$is_spider = detect_spider();
if(!$is_spider) {
$smcFunc['db_query']('', '
UPDATE {db_prefix}topics
SET num_views = num_views + 1
WHERE id_topic = {int:current_topic}
AND id_member_started != {int:current_member}',
array(
'current_topic' => $topic,
'current_member' => $user_info['id'],
)
);
}
]]></add>
</operation>


not installed in 2.1 RC4  :-\

Haven't updated it yet.

Update added support for SMF 2.1 RC4

Quote from: vbgamer45 on January 30, 2022, 01:09:06 AMUpdate added support for SMF 2.1 RC4

thank you, I will install  :)

error anyway

2022-01-30_114158.png

that just means that something else you installed changed what this mod is looking for

https://wiki.simplemachines.org/smf/Error_in_mod_installation

would need to know what the exact error is, what the mod is looking for, what the mod is trying to add/change, and what is actually in your file