News:

SMF 2.1.4 has been released! Take it for a spin! Read more.

Main Menu

Spiders Don't Increase Topic Views

Started by SMFHacks.com Team, August 06, 2007, 10:28:04 PM

Previous topic - Next topic

Eclipse16V

Here the fix
<?xml version="1.0"?>
<!DOCTYPE modification SYSTEM "http://www.simplemachines.org/xml/modification">
<modification xmlns="http://www.simplemachines.org/xml/modification" xmlns:smf="http://www.simplemachines.org/">
<name>Spiders Don't Increase Topic Views</name>
<id>karlbenson:SpidersNoTopicViewIncrease</id>
<version>1.1</version>
<homepage>http://dev.smfchat.com</homepage>

<file name="$sourcedir/Display.php">
<operation>
<search position="after"><![CDATA[// The central part of the board - topic display.
function Display()]]></search>
<add><![CDATA[
// DETECT SPIDERS
function detect_spider() {
// LIST OF SPIDERS
$known_spiders = array(
'WISENutbot', 'MSNBot', 'MSN spider', 'W3C Validator',
'Googlebot-Image', 'Googlebot', 'Mediapartners-Google', 'Openbot',
'Yahoo! Slurp', 'FAST-WebCrawler', 'Wget', 'Ask Jeeves',
'Speedy Spider', 'SurveyBot', 'IBM_Planetwide', 'OmniExplorer_Bot/6.68',
'ia_archiver', 'FAST-WebCrawler', 'Inktomi Slurp', 'Feedfetcher-Google',
'FeedBurner/1.0', 'appie', 'ping.blo.gs/2.0', 'http://www.relevantnoise.com',
'omgilibot/0.3', 'GigaBot', 'NewsGatorOnline/2.0', 'Jakarta Commons-HttpClient/3.0.1',
'Jakarta Commons-HttpClient/3.0-rc2',
);

foreach($known_spiders AS $agent) {
if (strpos(strtolower($_SERVER['HTTP_USER_AGENT']), strtolower($agent)) !== false) {
return true;
}
}
unset($known_spiders);
return false;
}
]]></add>
</operation>
<operation>
<search position="replace"><![CDATA[ $smcFunc['db_query']('', '
UPDATE {db_prefix}topics
SET num_views = num_views + 1
WHERE id_topic = {int:current_topic}',
array(
'current_topic' => $topic,
)
);
]]></search>
<add><![CDATA[
// ONLY INCREASE NUMBER OF TOPIC VIEWS IF NOT A SPIDER
$is_spider = detect_spider();
if(!$is_spider) {
$smcFunc['db_query']('', '
UPDATE {db_prefix}topics
SET num_views = num_views + 1
WHERE id_topic = {int:current_topic}',
array(
'current_topic' => $topic,
)
);
}
]]></add>
</operation>
</file>

</modification>
I worked with:
SMF 2 in German

Shop:
SID Giessen

vbgamer45

Community Suite for SMF - Take your forum to the next level built for SMF, Gallery,Store,Classifieds,Downloads,more!

SMFHacks.com -  Paid Modifications for SMF

Mods:
EzPortal - Portal System for SMF
SMF Gallery Pro
SMF Store SMF Classifieds Ad Seller Pro

Eclipse16V

Here with more Spiders
<?xml version="1.0"?>
<!DOCTYPE modification SYSTEM "http://www.simplemachines.org/xml/modification">
<modification xmlns="http://www.simplemachines.org/xml/modification" xmlns:smf="http://www.simplemachines.org/">
<name>Spiders Don't Increase Topic Views</name>
<id>karlbenson:SpidersNoTopicViewIncrease</id>
<version>1.1</version>
<homepage>http://dev.smfchat.com</homepage>

<file name="$sourcedir/Display.php">
<operation>
<search position="after"><![CDATA[// The central part of the board - topic display.
function Display()]]></search>
<add><![CDATA[
// DETECT SPIDERS
function detect_spider() {
// LIST OF SPIDERS
$known_spiders = array(
'192.comAgent',
'ABACHOBot',
'abcdatos',
'Accoona',
'Acoon',
'AdsBot-Google',
'appie',
'asterias',
'Ask Jeeves',
'Baiduspider',
'BecomeBot',
'Bloglines',
'BlogRefsBot',
'Charlotte',
'Daumoa',
'DiscoBot',
'DuckDuckBot',
'EnaBot',
'Exabot',
'FAST-WebCrawler',
'Feedburner',
'FeedBurner/1.0',
'Feedfetcher-Google',
'FeedValidator',
'Furlbot',
'FyberSpider',
'Gaisbot',
'GeonaBot',
'GigaBot',
'Girafabot',
'Googlebot-Image',
'Googlebot',
'Googlebot-Mobile',
'GoSeeBot',
'gsa-crawler',
'GSiteCrawler',
'http://www.relevantnoise.com',
'HTTrack',
'ia_archiver',
'ia_archiver-web.archive.org',
'IBM_Planetwide',
'ichiro',
'Inktomi Slurp',
'Jakarta Commons-HttpClient/3.0.1',
'Jakarta Commons-HttpClient/3.0-rc2',
'kalooga',
'LapozzBot',
'link validator',
'Lycos_Spider',
'Mediapartners-Google',
'MJ12bot/v2',
'MLBot',
'MSNBot',
'msnbot-media',
'MSNBOT_Mobile',
'MSN spider',
'MSR-ISRCCrawler',
'msrbot',
'NaverBot',
'NewsGatorOnline/2.0',
'noxtrumbot',
'OmniExplorer_Bot/6.68',
'omgilibot/0.3',
'OnetSzukaj',
'Openbot',
'P3P Validator',
'ping.blo.gs/2.0',
'psbot',
'Scooter',
'ScoutJet',
'Scrubby',
'SearchSight',
'Seeqpod',
'ShablastBot',
'silk/1.0',
'SitiDiBot',
'slurp',
'Snapbot',
'Sogou',
'Sosospider',
'Speedy Spider',
'StackRambler',
'SurveyBot',
'Teoma',
'TinEye',
'Touche',
'twiceler',
'W3C-checklink',
'W3C-mobileOK',
'W3C_CSS_Validator',
'W3C Validator',
'WebAlta',
'Websnapr',
'Wget',
'WISENutbot',
'wisponbot',
'Y!J',
'yacybot',
'yahoo',
'Yahoo-MMCrawler',
'YahooSeeker/M1A1-R2D2',
'Yahoo! Slurp',
'Yeti',
'YodaoBot',
);

foreach($known_spiders AS $agent) {
if (strpos(strtolower($_SERVER['HTTP_USER_AGENT']), strtolower($agent)) !== false) {
return true;
}
}
unset($known_spiders);
return false;
}
]]></add>
</operation>
<operation>
<search position="replace"><![CDATA[ $smcFunc['db_query']('', '
UPDATE {db_prefix}topics
SET num_views = num_views + 1
WHERE id_topic = {int:current_topic}',
array(
'current_topic' => $topic,
)
);
]]></search>
<add><![CDATA[
// ONLY INCREASE NUMBER OF TOPIC VIEWS IF NOT A SPIDER
$is_spider = detect_spider();
if(!$is_spider) {
$smcFunc['db_query']('', '
UPDATE {db_prefix}topics
SET num_views = num_views + 1
WHERE id_topic = {int:current_topic}',
array(
'current_topic' => $topic,
)
);
}
]]></add>
</operation>
</file>

</modification>
I worked with:
SMF 2 in German

Shop:
SID Giessen

vbgamer45

Community Suite for SMF - Take your forum to the next level built for SMF, Gallery,Store,Classifieds,Downloads,more!

SMFHacks.com -  Paid Modifications for SMF

Mods:
EzPortal - Portal System for SMF
SMF Gallery Pro
SMF Store SMF Classifieds Ad Seller Pro

distante

Quote from: Eclipse16V on January 24, 2010, 11:13:21 AM
Here with more Spiders
<?xml version="1.0"?>
<!DOCTYPE modification SYSTEM "http://www.simplemachines.org/xml/modification">
<modification xmlns="http://www.simplemachines.org/xml/modification" xmlns:smf="http://www.simplemachines.org/">
<name>Spiders Don't Increase Topic Views</name>
<id>karlbenson:SpidersNoTopicViewIncrease</id>
<version>1.1</version>
<homepage>http://dev.smfchat.com</homepage>

<file name="$sourcedir/Display.php">
<operation>
<search position="after"><![CDATA[// The central part of the board - topic display.
function Display()]]></search>
<add><![CDATA[
// DETECT SPIDERS
function detect_spider() {
// LIST OF SPIDERS
$known_spiders = array(
'192.comAgent',
'ABACHOBot',
'abcdatos',
'Accoona',
'Acoon',
'AdsBot-Google',
'appie',
'asterias',
'Ask Jeeves',
'Baiduspider',
'BecomeBot',
'Bloglines',
'BlogRefsBot',
'Charlotte',
'Daumoa',
'DiscoBot',
'DuckDuckBot',
'EnaBot',
'Exabot',
'FAST-WebCrawler',
'Feedburner',
'FeedBurner/1.0',
'Feedfetcher-Google',
'FeedValidator',
'Furlbot',
'FyberSpider',
'Gaisbot',
'GeonaBot',
'GigaBot',
'Girafabot',
'Googlebot-Image',
'Googlebot',
'Googlebot-Mobile',
'GoSeeBot',
'gsa-crawler',
'GSiteCrawler',
'http://www.relevantnoise.com',
'HTTrack',
'ia_archiver',
'ia_archiver-web.archive.org',
'IBM_Planetwide',
'ichiro',
'Inktomi Slurp',
'Jakarta Commons-HttpClient/3.0.1',
'Jakarta Commons-HttpClient/3.0-rc2',
'kalooga',
'LapozzBot',
'link validator',
'Lycos_Spider',
'Mediapartners-Google',
'MJ12bot/v2',
'MLBot',
'MSNBot',
'msnbot-media',
'MSNBOT_Mobile',
'MSN spider',
'MSR-ISRCCrawler',
'msrbot',
'NaverBot',
'NewsGatorOnline/2.0',
'noxtrumbot',
'OmniExplorer_Bot/6.68',
'omgilibot/0.3',
'OnetSzukaj',
'Openbot',
'P3P Validator',
'ping.blo.gs/2.0',
'psbot',
'Scooter',
'ScoutJet',
'Scrubby',
'SearchSight',
'Seeqpod',
'ShablastBot',
'silk/1.0',
'SitiDiBot',
'slurp',
'Snapbot',
'Sogou',
'Sosospider',
'Speedy Spider',
'StackRambler',
'SurveyBot',
'Teoma',
'TinEye',
'Touche',
'twiceler',
'W3C-checklink',
'W3C-mobileOK',
'W3C_CSS_Validator',
'W3C Validator',
'WebAlta',
'Websnapr',
'Wget',
'WISENutbot',
'wisponbot',
'Y!J',
'yacybot',
'yahoo',
'Yahoo-MMCrawler',
'YahooSeeker/M1A1-R2D2',
'Yahoo! Slurp',
'Yeti',
'YodaoBot',
);

foreach($known_spiders AS $agent) {
if (strpos(strtolower($_SERVER['HTTP_USER_AGENT']), strtolower($agent)) !== false) {
return true;
}
}
unset($known_spiders);
return false;
}
]]></add>
</operation>
<operation>
<search position="replace"><![CDATA[ $smcFunc['db_query']('', '
UPDATE {db_prefix}topics
SET num_views = num_views + 1
WHERE id_topic = {int:current_topic}',
array(
'current_topic' => $topic,
)
);
]]></search>
<add><![CDATA[
// ONLY INCREASE NUMBER OF TOPIC VIEWS IF NOT A SPIDER
$is_spider = detect_spider();
if(!$is_spider) {
$smcFunc['db_query']('', '
UPDATE {db_prefix}topics
SET num_views = num_views + 1
WHERE id_topic = {int:current_topic}',
array(
'current_topic' => $topic,
)
);
}
]]></add>
</operation>
</file>

</modification>


has this been added to the install mod? :P

vbgamer45

Community Suite for SMF - Take your forum to the next level built for SMF, Gallery,Store,Classifieds,Downloads,more!

SMFHacks.com -  Paid Modifications for SMF

Mods:
EzPortal - Portal System for SMF
SMF Gallery Pro
SMF Store SMF Classifieds Ad Seller Pro

distante


karanthsrihari

I installed the mod. I posted one article in twitter and in next second i received some 20 bots. And this was registered in view count of the post. So i guess this mod is not working properly?
http://onlinefundb.com [nofollow] - Online Fun Database

vbgamer45

It depends on the bots. It checks for major search engines like google, yahoo
Community Suite for SMF - Take your forum to the next level built for SMF, Gallery,Store,Classifieds,Downloads,more!

SMFHacks.com -  Paid Modifications for SMF

Mods:
EzPortal - Portal System for SMF
SMF Gallery Pro
SMF Store SMF Classifieds Ad Seller Pro

mariusfv

Is this default in RC5? I have a lot of robots views my forum but they don't increase the number of views and I want to increase?
Vbulletin have for sure a mod who icrease number of views(fake it) because I see on some forum a post with less then 5 minutes with over 5000 number of views, have smf this kind of mod or is against their policy?

vbgamer45

Community Suite for SMF - Take your forum to the next level built for SMF, Gallery,Store,Classifieds,Downloads,more!

SMFHacks.com -  Paid Modifications for SMF

Mods:
EzPortal - Portal System for SMF
SMF Gallery Pro
SMF Store SMF Classifieds Ad Seller Pro

Iomega0318

As I seem to have hundreds of bot visits that still show as guests, until I can get them all sorted is there any way to not count guests views as well?

I think the majority of mine are coming from Twitter, I never saw this many bots until I started posting links on there.. does anyone have a list of bots that come from Twitter?


Also for the bot list, does the bot name need to be in the database or does it just find that bot by name on the forum to block it's view count?
UOFreeshards.net
UOFreeshards.net 2.1 Aplha Test Site
"I believe in Christianity as I believe in the sun...
not because I see it but because by it I see everything else."
C.S. Lewis

Heed what I say, for it shall be mentioned only once.

Iomega0318

Here is another update with even more bots..

<?xml version="1.0"?>
<!DOCTYPE modification SYSTEM "http://www.simplemachines.org/xml/modification">
<modification xmlns="http://www.simplemachines.org/xml/modification" xmlns:smf="http://www.simplemachines.org/">
<name>Spiders Don't Increase Topic Views</name>
<id>karlbenson:SpidersNoTopicViewIncrease</id>
<version>1.1</version>
<homepage>http://dev.smfchat.com</homepage>

<file name="$sourcedir/Display.php">
<operation>
<search position="after"><![CDATA[// The central part of the board - topic display.
function Display()]]></search>
<add><![CDATA[
// DETECT SPIDERS
function detect_spider() {
// LIST OF SPIDERS
$known_spiders = array(
'192.comAgent',
'ABACHOBot',
'abcdatos',
'Accoona',
'Acoon',
'AdsBot-Google',
'appie',
'asterias',
'Ask Jeeves',
'Baiduspider',
'BecomeBot',
'Bloglines',
'BlogRefsBot',
'Charlotte',
'Daumoa',
'DiscoBot',
'DuckDuckBot',
'EnaBot',
'Exabot',
'FAST-WebCrawler',
'Feedburner',
'FeedBurner/1.0',
'Feedfetcher-Google',
'FeedValidator',
'Furlbot',
'FyberSpider',
'Gaisbot',
'GeonaBot',
'Gigabot',
'GigaBot',
'Girafabot',
'googlebot',
'Googlebot-Image',
'Googlebot',
'Googlebot-Mobile',
'GoSeeBot',
'gsa-crawler',
'GSiteCrawler',
'http://www.relevantnoise.com',
'HTTrack',
'ia_archiver',
'ia_archiver-web.archive.org',
'IBM_Planetwide',
'ichiro',
'Inktomi Slurp',
'Jakarta Commons-HttpClient/3.0.1',
'Jakarta Commons-HttpClient/3.0-rc2',
'kalooga',
'LapozzBot',
'link validator',
'Lycos_Spider',
'Mediapartners-Google',
'MJ12bot/v2',
'MLBot',
'msn',
'MSNBot',
'msnbot-media',
'MSNBOT_Mobile',
'MSN spider',
'MSR-ISRCCrawler',
'msrbot',
'NaverBot',
'NewsGatorOnline/2.0',
'noxtrumbot',
'omgilibot',
'OmniExplorer_Bot',
'OmniExplorer_Bot/6.68',
'omgilibot/0.3',
'OnetSzukaj',
'Openbot',
'P3P Validator',
'ping.blo.gs/2.0',
'psbot',
'Scooter',
'ScoutJet',
'Scrubby',
'SearchSight',
'Seeqpod',
'ShablastBot',
'silk/1.0',
'SitiDiBot',
'slurp',
'Snapbot',
'Sogou',
'Sosospider',
'Speedy Spider',
'StackRambler',
'SurveyBot',
'Teoma',
'TinEye',
'Touche',
'twiceler',
'Twitter',
'Twitter Bot',
'W3C-checklink',
'W3C-mobileOK',
'W3C_CSS_Validator',
'W3C_Validator',
'W3C Validator',
'WebAlta',
'Websnapr',
'Wget',
'WISENutbot',
'wisponbot',
'Y!J',
'yacybot',
'yahoo',
'Yahoo-MMCrawler',
'YahooSeeker/M1A1-R2D2',
'Yahoo! Slurp',
'Yeti',
'YodaoBot',
);

foreach($known_spiders AS $agent) {
if (strpos(strtolower($_SERVER['HTTP_USER_AGENT']), strtolower($agent)) !== false) {
return true;
}
}
unset($known_spiders);
return false;
}
]]></add>
</operation>
<operation>
<search position="replace"><![CDATA[ $smcFunc['db_query']('', '
UPDATE {db_prefix}topics
SET num_views = num_views + 1
WHERE id_topic = {int:current_topic}',
array(
'current_topic' => $topic,
)
);
]]></search>
<add><![CDATA[
// ONLY INCREASE NUMBER OF TOPIC VIEWS IF NOT A SPIDER
$is_spider = detect_spider();
if(!$is_spider) {
$smcFunc['db_query']('', '
UPDATE {db_prefix}topics
SET num_views = num_views + 1
WHERE id_topic = {int:current_topic}',
array(
'current_topic' => $topic,
)
);
}
]]></add>
</operation>
</file>

</modification>
UOFreeshards.net
UOFreeshards.net 2.1 Aplha Test Site
"I believe in Christianity as I believe in the sun...
not because I see it but because by it I see everything else."
C.S. Lewis

Heed what I say, for it shall be mentioned only once.

dougiefresh

Thank you for the wonderful mod!  I thought I'd let everybody know that when I updated my forum from 2.0 RC5 to 2.0 Gold, I had to reinstall all my mods.  This mod installed without issues and without emulation.  Hope this info helps someone!

FireDitto

Is it possible to have the mod read off the spiders listed in the spider list of 2.0? It would be easier than having to update the files all the time... And I get so many bots crawling my site, that this mod is barely effective. :(

Second Pass Weyr<br />An AU Pernse RPG<br /><br />SMF 2.0.6 with SP 2.3.5

SMFHacks.com Team

Quote from: FireDitto on June 14, 2012, 09:06:37 AM
Is it possible to have the mod read off the spiders listed in the spider list of 2.0? It would be easier than having to update the files all the time... And I get so many bots crawling my site, that this mod is barely effective. :(


It would use another db query  but could be possible.

What kind of bots are getting added? The big ones are bing,google,baidu
Disclaimer: SMFHacks.com Team is not affiliated with the SMF Team or the SimpleMachines NPO.
SMFHacks.com -  Paid Modifications for SMF
Latest Mods:
Community Suite
Newsletter Pro SMF Gallery Pro SMF Classifieds SMF Store

FireDitto

Odds and ends, mostly. Nothing that strike me as big end bots, but try are there anyway.

A lot of them are those brought in by Twitter, some others I don't know. I have a member base with pretty diverse roots, so some of the spiders make me go o.O haha but I've been adding them all and because they aren't big bots, they aren't going t be on the list, do I thought it would be easier for the mod to draw from te spider listing.




-iPhone auto spell is responsible for weird sentences. :D
Second Pass Weyr<br />An AU Pernse RPG<br /><br />SMF 2.0.6 with SP 2.3.5

vbgamer45

If you have a list I can add more. I use the spider check for a lot of places in my forums for better performance no reason to show information to bots
Community Suite for SMF - Take your forum to the next level built for SMF, Gallery,Store,Classifieds,Downloads,more!

SMFHacks.com -  Paid Modifications for SMF

Mods:
EzPortal - Portal System for SMF
SMF Gallery Pro
SMF Store SMF Classifieds Ad Seller Pro

FireDitto

Um, I think this is all of them. I may have missed some, or included a couple that were pre-loaded but I'm pretty sure these are all ones I added to the Spiders option on SMF.
Spider name; User Agent

Ahrefs; AhrefsBot
Baidu; Baiduspider
Bing Bot; Bingbot/2.0
Crowsnest; Crowsnest
Embedly; Embedly
Exabot; Exabot
Facebook Bot; facebookxternalhit/1.1
In A Gist; InAGist URL Resolver
JS Kit; JS-Kit URL Resolver
Linked In Bot; LinkedINBot/1.0
Mcafee; MFE_expand
ML-RPC; ML-RPC
MLBot; MLBot
Mozilla; Mozilla/0.19 Beta (Windows)
Orca Ringmaker; Orca Ringmaker V3.0
ScoutJet; ScoutJet
Seznam Bot; SeznamBot/3.0
strawberryj.am; strawberryj.am
TweetmemeBot; TweetmemeBot
Twinkle Bot; Twinkle/1.0
Twitterbot; Twitterbot
UnwindFetchor; UnwindFetchor
Yandex; Yandex
Second Pass Weyr<br />An AU Pernse RPG<br /><br />SMF 2.0.6 with SP 2.3.5

SMFHacks.com Team

updated to support SMF 2.1.x
And added more spiders to block for topic views.
Disclaimer: SMFHacks.com Team is not affiliated with the SMF Team or the SimpleMachines NPO.
SMFHacks.com -  Paid Modifications for SMF
Latest Mods:
Community Suite
Newsletter Pro SMF Gallery Pro SMF Classifieds SMF Store

Advertisement: