Googlebot & Spiders

[Omar Bazavilvazo]

I added that into ModSettings.php and it works great! Thanks!

great is finally working

[LuminaRS]

I guess I accidently lied, at the time I didn't have any spiders on the forum, now that I do I see that it's unfortuinately not working correctly. It currently says 0 Guests, 3 Users online, where there is in fact also two spiders online. So it's half working, just not sowing the seperate group for the spiders.

Count all instances of same spider      (CHECKED)
Display all instances of same spider     (CHECKED)
Display agent instead of name             (UNCHECKED)
Display spiders in its own list         (CHECKED)
Count in "Most Online"                         (UNCHECKED)
Redirect PHPSESSID URLs                  (CHECKED)

[DemonicInfluence]

are u on a non-default theme?

[LuminaRS]

Yes, running Cerberus by Bloc, can anyone help?

[marinelife]

I got it installed but have this in the error log
http://www.masohio.com/index.php 
8: Undefined index: ob_googlebot_stats
File: D:\web\masohio\Sources\Subs.php
Line: 3341

http://www.masohio.com/index.php?action=admin 
8: Undefined index: ob_googlebot_stats
File: D:\web\masohio\Sources\Subs.php
Line: 3341

http://www.masohio.com/index.php?action=viewErrorLog;desc 
8: Undefined index: ob_googlebot_stats
File: D:\web\masohio\Sources\Subs.php
Line: 3341

It seems to work but I did not get them on my test site

[DemonicInfluence]

Yes, running Cerberus by Bloc, can anyone help?

you have to manually mod boardindex.template.php for the cerebrus theme

BTW, are you ever gonna make it so that in messageindex place you can see bots

[LuminaRS]

FIgured such, what do I have to add and where?

[Omar Bazavilvazo]

FIgured such, what do I have to add and where?

check install.xml inside the zip, and you will see what needs to be changed.
Usually, any mod that displays something, only works in default theme, and all the rest needs to be changed manually.

I got it installed but have this in the error log
http://www.masohio.com/index.php 
8: Undefined index: ob_googlebot_stats
File: D:\web\masohio\Sources\Subs.php
Line: 3341

http://www.masohio.com/index.php?action=admin 
8: Undefined index: ob_googlebot_stats
File: D:\web\masohio\Sources\Subs.php
Line: 3341

http://www.masohio.com/index.php?action=viewErrorLog;desc 
8: Undefined index: ob_googlebot_stats
File: D:\web\masohio\Sources\Subs.php
Line: 3341

It seems to work but I did not get them on my test site

You rather uninstall it. was not installed properly, try to install it again, but check in which file you are getting errors. seems that ob_googlebot_stats field wasn't added to settings table on thr DB

[dagod]

I have many mods that modified my source files, so i can not delete googlebot_2.0.0

what are the changes from 2.0.0 to 2.0.2 so that i can update to 2.0.2 by hand?

[LuminaRS]

check install.xml inside the zip, and you will see what needs to be changed.

Works like a charm 

[steve in houston]

i noticed a rush of about 10 IP's from AOL come in at once.
the IP's are similar but it only says MSIE next to it.

is there a way i can add it to the spider list?

[Omar Bazavilvazo]

I have many mods that modified my source files, so i can not delete googlebot_2.0.0

what are the changes from 2.0.0 to 2.0.2 so that i can update to 2.0.2 by hand?

I added a .zip with the report of changes from 2.0.0 and 2.0.2. I used a report tool, it has 2 files, one in html and another un .txt using unix diff format

Works like a charm 

Great!

i noticed a rush of about 10 IP's from AOL come in at once.
the IP's are similar but it only says MSIE next to it.

is there a way i can add it to the spider list?

yeah. In Subs.php in the function ob_googlebot_getagent, you can add more to the list.

ja ne!

[dagod]

wow, Araxis Merge is a very nice tool.... downloading....

[Omar Bazavilvazo]

wow, Araxis Merge is a very nice tool.... downloading....

yeah, it rocks, and i used subversion to store and tag the different versions, so was quite easy o make the diff

[karlbenson]

Dunno whether this is of any use to you. But heres a good list of spiders

http://www.robotstxt.org/wc/active/html/index.html

[slimjack]

Hoping to get some help here...

I installed this but got a table error... I don't think a table was installed properly.

So now I don't see it listed to uninstall it, but I do see some bot stats on the forums.  However, I also see the following now at the top of all of the forum pages as well as some of the admin pages that completely error out to the following:

Notice: Undefined index: ob_googlebot_redirect_phpsessid in /home/www/mywebsite.net/forums/index.php on line 76

Notice: Undefined index: ob_googlebot_redirect_phpsessid in /home/www/mywebsite.net/forums/index.php on line 91

I'm using a custom template of my own that was created from a copy of the default template.  The only thing I really customized were images and colors so it should be alright in that regard.
Any ideas?  I'd love to be able to use it....

[Omar Bazavilvazo]

Hoping to get some help here...

I installed this but got a table error... I don't think a table was installed properly.

In fact, that's the problem, because seems some data was not inserted properly on the database.

You can try to add the data manually from phpMyAdmin, for example. The DB code is:

Code Select Expand


db_query("INSERT IGNORE INTO `{$db_prefix}settings` (`variable`, `value`) VALUES ('ob_googlebot_count_all_instances', '1')", __FILE__, __LINE__);
db_query("INSERT IGNORE INTO `{$db_prefix}settings` (`variable`, `value`) VALUES ('ob_googlebot_display_all_instances', '1')", __FILE__, __LINE__);
db_query("INSERT IGNORE INTO `{$db_prefix}settings` (`variable`, `value`) VALUES ('ob_googlebot_display_agent', '0')", __FILE__, __LINE__);
db_query("INSERT IGNORE INTO `{$db_prefix}settings` (`variable`, `value`) VALUES ('ob_googlebot_display_own_list', '0')", __FILE__, __LINE__);
db_query("INSERT IGNORE INTO `{$db_prefix}settings` (`variable`, `value`) VALUES ('ob_googlebot_count_most_online', '0')", __FILE__, __LINE__);
db_query("INSERT IGNORE INTO `{$db_prefix}settings` (`variable`, `value`) VALUES ('ob_googlebot_redirect_phpsessid', '1')", __FILE__, __LINE__);
db_query("INSERT IGNORE INTO `{$db_prefix}settings` (`variable`, `value`) VALUES ('ob_googlebot_stats', '1')", __FILE__, __LINE__);

db_query("
CREATE TABLE IF NOT EXISTS `{$db_prefix}ob_googlebot_stats` (
  `agent` varchar(40) NOT NULL default '',
  `board` varchar(20) NOT NULL default '',
  `topic` varchar(20) NOT NULL default '',
  `url` varchar(100) NOT NULL default '',
  `lastvisit` int(10) unsigned NOT NULL default '0',
  `frequency` int(10) unsigned NOT NULL default '0',
  `visits` int(10) unsigned NOT NULL default '0',
  `timestamp` timestamp(14) NOT NULL,
  KEY `agent` (`agent`),
  KEY `board` (`board`),
  KEY `topic` (`topic`),
  KEY `url` (`url`)
)", __FILE__, __LINE__);


Replacing {$db_prefix} with smf_ (or whatever prefix you specified)

Let me know if it works

[slimjack]

Okay, strange thing here...  The errors are no longer at the top of the page and the mod now shows as an installed package.

The db has the following in the smf_settings:

ob_googlebot_display_own_list     1
ob_googlebot_count_most_online    1
ob_googlebot_redirect_phpsessid    1
ob_googlebot_display_agent    1

A few shy, eh?
Tried the mysql statement you provided but mysql doesn't like it at all and gives no specific error.

However, I now see at the bottom where it shows who is online...

1 spider 1 spider 1 spider joeblow mikeb (0 buddies)

Is that normal?

[Omar Bazavilvazo]

Heh, my mistake, should be:

Code Select Expand


INSERT IGNORE INTO `smf_settings` (`variable`, `value`) VALUES ('ob_googlebot_count_all_instances', '1');
INSERT IGNORE INTO `smf_settings` (`variable`, `value`) VALUES ('ob_googlebot_display_all_instances', '1');
INSERT IGNORE INTO `smf_settings` (`variable`, `value`) VALUES ('ob_googlebot_display_agent', '0');
INSERT IGNORE INTO `smf_settings` (`variable`, `value`) VALUES ('ob_googlebot_display_own_list', '0');
INSERT IGNORE INTO `smf_settings` (`variable`, `value`) VALUES ('ob_googlebot_count_most_online', '0');
INSERT IGNORE INTO `smf_settings` (`variable`, `value`) VALUES ('ob_googlebot_redirect_phpsessid', '1');
INSERT IGNORE INTO `smf_settings` (`variable`, `value`) VALUES ('ob_googlebot_stats', '1');


CREATE TABLE IF NOT EXISTS `smf_ob_googlebot_stats` (
  `agent` varchar(40) NOT NULL default '',
  `board` varchar(20) NOT NULL default '',
  `topic` varchar(20) NOT NULL default '',
  `url` varchar(100) NOT NULL default '',
  `lastvisit` int(10) unsigned NOT NULL default '0',
  `frequency` int(10) unsigned NOT NULL default '0',
  `visits` int(10) unsigned NOT NULL default '0',
  `timestamp` timestamp(14) NOT NULL,
  KEY `agent` (`agent`),
  KEY `board` (`board`),
  KEY `topic` (`topic`),
  KEY `url` (`url`)
);


Try that again and let me know if it works on phpmyadmin

[slimjack]

That did it.  I see the changes you made, too.  thx!

What about this, though:

~~~~~
1 Spider, 1 Spider, 0 Guests, 2 Users (1 Buddy)
Users active in past 15 minutes:
SlimJack, Dissonance
Spiders active in past 15 minutes:
Googlebot
Spiders active in past 15 minutes:
Googlebot
~~~~~

Should it be displayed this way?....  1 spider, 1 spider and then showing it twice below also?