Cron.php

[gkawa]

I apologize if it's a silly question. I tried to find more information about Cron.php and nothing came up.

As I understand it, Cron.php does all the time-based tasks and it's called from a java script as the users browse the forum. And it's safe to call it at any time. In fact, it is recommended to call it from a server chron job, to keep it running even if no one is accessing the forum.

My question is if anyone knows why is Googlebot constantly asking for Cron.php.

It's not a big deal, not a safety concern. I'm just puzzled. It can't get that URL from the site pages. The source code doesn't have it. But the worst part is that it's calling a URL that never existed.

The forum was originally installed on the root directory. So, the URL was /cron.php. More than 10 years ago it was moved to /forum. Googlebot is asking for /chron.php every single day, every minute or so. I thought it was checking URLs from way back. But the ts is current! It doesn't match the timestamp of the request, they're not serialized in any way. It's anything from an hour ago to a week. It couldn't be me, even if the cron.php URL were on the page they would be all pointing to /forum. Even worse, Googlebot is not asking for /forum/cron.php.

Where is it getting those URLs? And why?..

[shawnb61]

I believe that googlebot is emulating user activity in the way they crawl - even invoking js.  So yes, they trigger cron.   

I don't think many other crawlers do that.

That should be a good thing, as it helps ensure scheduled tasks & mailings will all happen timely, even with zero activity.

If only it weren't buggy.  Yes, if your forum is in a subfolder, it seems to not invoke cron properly - it tries to run it at root.  But it's not there... 

That, I think, is a bug in googlebot.

[gkawa]

Yes, it could be a good thing... if it calls the right URL for a change!
Not that it makes a real difference. If your forum is very active, the same activity is triggering cron. If it's not, there's no need to.

But Googlebot doing that? Is it just me or it's completely bonkers?
And, as far as I can see, no other bot is doing that. Maybe Google knows something I don't.

I was wondering if someone else noticed that behavior. Probably not if Googlebot is using the right cron URL. I found it because it's filling my error logs.

[Kindred]

Googlebot is not hitting the cron.php itself....  it's triggering it as any guest would

It's just acting differently from other bots

[gkawa]

Googlebot is not hitting the cron.php itself....  it's triggering it as any guest would

It's just acting differently from other bots

It's not triggering, in this case. The issue here is that I found this because Google is calling a cron.php that doesn't exist with a current timestamp. To trigger it, the index.php page should be available and it's not.

I checked the logs

This is a normal triggering, a user browsing the forum and calling cron.php from there.
[nofollow]
This is Googlebot triggering, about the same except it doesn't call all the objects.
[nofollow]
This is Googlebot calling the wrong cron.php out of the blue. I thought it could be a delayed call, it's about 15 seconds off, compared with the previous call from a user. But I couldn't find a call to index.php around that time, the right one or the wrong one.
[nofollow]
This is Googlebot calling the wrong cron.php with a TS from almost a year ago. The forum was migrated from root more than 10 years ago.
[nofollow]
And here's what I think could be the root of the problem. Googlebot automatically created the wrong URL from the right one. Both calls with the same TS.
[nofollow]

My theory, Googlebot is hitting dynamic URLs that it (with its marvelous AI...) assumes may have a timestamp parameter to get the most current response. It's wrong, but... Google...
It has the cron.php stored from years ago and keeps it because it knows about the new one and thinks it may be a more canonical form. Does it make sense?
It doesn't explain why it's keeping the URLs it creates, even after a 404. That's craziness, according to Rita Mae Brown...

[Kindred]

Again... nothing should be hitting cron.php diesctly.

That's not a url that Google should ever hit.

Cron.php is triggered by activity in the forum.  So, Googlebot is browsing through the forum and that activity it triggering cron.php to run...

And not every page view triggers cron (I believe)

[shawnb61]

And yet it does.  Incorrectly.

[gkawa]

Again... nothing should be hitting cron.php diesctly.
That's not a url that Google should ever hit.

That's what I'm saying. Googlebot is doing exactly that, hitting cron.php directly.

By the way, and I don't think it's a surprise, Googlebot is completely ignoring robots.txt.

Disallow: /chron.php*
Disallow: /forum/chron.php*

From Google's article about robots.txt

robots.txt rules may not be supported by all search engines.
The instructions in robots.txt files cannot enforce crawler behavior to your site; it's up to the crawler to obey them. While Googlebot and other respectable web crawlers obey the instructions in a robots.txt file, other crawlers might not. Therefore, if you want to keep information secure from web crawlers, it's better to use other blocking methods, such as password-protecting private files on your server.

[shawnb61]

You have a typo there.  cron, not chron.

But it will still invoke it at root.

[gkawa]

You have a typo there.  cron, not chron.

But it will still invoke it at root.

You're right!
I wrote what I was used to writing when I think about "chron".
Maybe it'll work now... even when Google keeps ignoring all the other disallows... 

Thanks. Good catch!