Googlebot & Spiders

[G6™]

In the first post in this thread you have a link to the mod in the download section.
And yes, it's a package install on this mod

[Skunkie]

Also show in the forum stats what each spider is spidering (like what each user is doing on the board)

This can be done by using the Custom Who.template.php - separated list for users, guests and spiders. It's very handy.


edit:  Looks like Mrs G6 actually had the answer before me, I just didn't notice.  Heheh

[benito]

I make a clean list with all spiders posted here, on who template and google bot spider files. No duplicated entries.

array (
         'agent' => 'Scooter',
         'spidername' => 'Altavista spider',
         'spider' => true,
      ),
      array (
         'agent' => 'FAST-WebCrawler',
         'spider' => true,
      ),
      array (
         'agent' => 'Inktomi Slurp',
         'spider' => true,
      ),
    array (
      'agent' => 'FeedBurner/1.0',
      'spidername' => 'Feedburner',
      'spider' => true,
    ),
    array (
      'agent' => 'Feedfetcher-Google',
      'spidername' => 'Feedfetcher-Google',
      'spider' => true,
    ),
    array (
      'agent' => 'OmniExplorer_Bot/6.68',
      'spidername' => 'OmniExplorer_Bot',
      'spider' => true,
    ),
    array (
      'agent' => 'http://www.relevantnoise.com',
      'spidername' => 'relevantNOISE',
      'spider' => true,
    ),
    array (
      'agent' => 'NewsGatorOnline/2.0',
      'spidername' => 'NewsGatorOnline/2.0',
      'spider' => true,
    ),
    array (
      'agent' => 'ping.blo.gs/2.0',
      'spidername' => 'ping.blo.gs/2.0',
      'spider' => true,
    ),
    array (
      'agent' => 'Jakarta Commons-HttpClient/3.0.1',
      'spidername' => 'Amazon',
      'spider' => true,
    ),
    array (
      'agent' => 'Jakarta Commons-HttpClient/3.0-rc2',
      'spidername' => 'Amazon',
      'spider' => true,
    ),
      array (
        'agent' => 'WISENutbot',
         'spidername' => 'Looksmart spider',
      'spider' => true,
      ),
      array (
         'agent' => 'MSNBot',
         'spidername' => 'MSN spider',
         'spider' => true,
      ),
      array (
         'agent' => 'W3C_Validator',
         'spidername' => 'W3C Validaator',
      'spider' => true,
      ),
      array (
         'agent' => 'Googlebot-Image',
         'spidername' => 'Google-Image Spider',
         'spider' => true,
      ),
      array (
         'agent' => 'Googlebot',
         'spidername' => 'Google spider',
         'spider' => true,
      ),
      array (
         'agent' => 'Mediapartners-Google',
         'spidername' => 'Google AdSense spider',
         'spider' => true,
      ),
    array (
         'agent' => 'Openbot',
         'spidername' => 'Openfind spider',
         'spider' => true,
    ),
      array (
        'agent' => 'Yahoo! Slurp',
         'spidername' => 'Yahoo spider',
         'spider' => true,
      ),
    array (
         'agent' => 'Wget',
         'spider' => true,
    ),
      array (
         'agent' => 'Ask Jeeves',
         'spider' => true,
      ),
      array (
         'agent' => 'Speedy Spider',
         'spider' => true,
      ),
      array (
         'agent' => 'SurveyBot',
         'spider' => true,
      ),
      array (
         'agent' => 'IBM_Planetwide',
         'spider' => true,
      ),
    array (
         'agent' => 'GigaBot',
         'spider' => true,
      ),
      array (
         'agent' => 'ia_archiver',
         'spider' => true,
      ),
    array (
         'agent' => 'appie',
         'spidername' => 'Walhello spider',
         'spider' => true,
      ),

[desistyle]

does this work wid smf 1.1 rc3

[Skipdawg]

does this work wid smf 1.1 rc3

Yes it dose. But we are eagerly awaiting a new and improved version too. 

[Omar Bazavilvazo]

in this post works with rc3
http://www.simplemachines.org/community/index.php?topic=38003.msg701729#msg701729

[Omar Bazavilvazo]

hi
Well, i said i was going to finish this weekend, and I finished, only problem is that i couldn't finish the package. but the coding is finished.

i plan to finish the packaging in the next 2 days, because i have to rebauild install.xml, check some stuff about queries to don't delete old DB data, and most important, check that it workswith both 1.0.8 and rc3. is a little time-consuming, so please bear with me

just for the curious, here is one screenshot of 1 of the 2 new features


sorry for the delay, but it was a unexpected funny weekend, so i wasn't in my house at all

ja ne!

[Skipdawg]

Sounds like it will be well worth the wait. I like that "google visited last..." that is kewl 
Eagerly and patiently awaiting your masterpiece. 

[Omar Bazavilvazo]

New version

v2.0.0, must work with 1.1 RC3 and 1.0.8

is a little beta, so please help me with any bug you see, and, because i'm not native english speaker, if my english strings are incorrect, please provide me with the correct ones. I hope you enjoy it

[Omar Bazavilvazo]

Released 2.0.1

Topics count on google stats was ignored, so you can uninstall and install again or:

In Subs.php
find:

Code Select Expand


if (isset($context['current_board'])) // We are on a board
{
$key = 'board';
$value = "$context[current_board].$context[start]";
}
elseif (isset($context['current_topic'])) // We are on a topic
{
$key = 'topic';
$value = "$context[current_topic].$context[start]";
}


replace with:

Code Select Expand


if (isset($context['current_topic'])) // We are on a topic
{
$key = 'topic';
$value = "$context[current_topic].$context[start]";
}
elseif (isset($context['current_board'])) // We are on a board
{
$key = 'board';
$value = "$context[current_board].$context[start]";
}


ja ne!

[Skipdawg]

Right on about to install and check it out. 

[Skipdawg]

Installed fine but something is not right. All bots are showing by name down by "users online" but on the who page they show up as guest. 

[Omar Bazavilvazo]

er....

the mod doesn't changes anything in the who page.

in the first post is the URL of the who modified file:

Is STRONGLY recommended to install this mod also (for a detailed view of what is being crawled):
http://www.simplemachines.org/community/index.php?topic=19243.0

ja ne!

[Skipdawg]

Right on thanks Omar. I must of had that before and forgot to put it back after going to RC3. OOPS! 

[Omar Bazavilvazo]

Right on thanks Omar. I must of had that before and forgot to put it back after going to RC3. OOPS! 

Good is working

[Skipdawg]

Well 2 of my Global Moderators now have stated the same thing. lol 

"This Mods Rocks!" 

[~dragonfly~]

Hello

I am on RC3 and I currently have installed Googlebot v105 for RC2 installed.  I changed RC2 to RC3 in the package-info.xml and it installed with no problems.  I went today to uninstall it so that I could install the latest version, and I am getting this:

Installing this package will perform the following actions:
   Type    Action    Description
1.    Execute Modification    ./Sources/BoardIndex.php    Test successful
2.    Execute Modification    ./Sources/ManagePermissions.php    Test successful
3.    Execute Modification    ./Sources/ModSettings.php    Test successful
4.    Execute Modification    ./Sources/Subs.php    Test successful
5.    Execute Modification    ./Themes/theme5/languages/Modifications.english.php    Test failed
6.    Execute Modification    ./Themes/theme5/BoardIndex.template.php    Test successful
7.    Execute Modification    ./SSI.php    Test failed

I did not continue with the uninstall at that point, as I didn't know if it would cause problems if I did.  What can I do at this point to successfully upgrade to the latest version of Googlebot?

Thanks. 

[Omar Bazavilvazo]

If you didn't modified SSI.php with another mod, you can overwrite with a clean copy.

In modifications.english.php, open that file and you will see the block that belongs to googlebot, erase that text, is quite easy to identify.

[maydayke]

Hi, im getting these errors since i installed Googlebot 2.0.1:

Code Select Expand

http://www.mysite.com/index.php?action=post;topic=260.0;num_replies=0
8: Undefined index: start
File: /home/forecadd/public_html/Sources/Subs.php
Line: 3349

Code Select Expand

http://www.mysite.com/index.php?action=post2;board=18;preview;xml
8: Undefined index: start
File: /home/forecadd/public_html/Sources/Subs.php
Line: 3354

Code Select Expand

http://www.mysite.com/index.php?action=post;board=18.0
8: Undefined index: start
File: /home/forecadd/public_html/Sources/Subs.php
Line: 3354

I didn't had those errors before i installed googlebot.

Im using smf 1.1rc3, TP 0.95 and arcade mod.
Anyone has an idea what causes these errors and how to fix them?

Much appriciated!

[Omar Bazavilvazo]

New version 2.0.2, fixed 'start' error, and checked compatibility with TP 0.9.5

Let me know if you have any problems