Simple Machines Community Forum

SMF Support => SMF 2.1.x Support => Topic started by: Shkic on September 09, 2015, 04:08:21 AM

Title: Google and cron.php 404 errors
Post by: Shkic on September 09, 2015, 04:08:21 AM
Google search engine want to index cron.php file andin Google Webmaster tools inform me that there is a lot 404 errors:

   
Quote992
cron.php?ts=1441513590
404
15.9.7

993
cron.php?ts=1441513455
404
15.9.7

994
cron.php?ts=1441538655
404
15.9.7

995
cron.php?ts=1441566435
404
15.9.7

996
cron.php?ts=1441536885
404
15.9.7

997
cron.php?ts=1441534305
404
15.9.7

998
cron.php?ts=1441513575
404
15.9.7

999
cron.php?ts=1441541745
404
15.9.7

1000
cron.php?ts=1441573380
404
15.9.7

any suggestions?
Title: Re: Google and cron.php 404 errors
Post by: Kindred on September 09, 2015, 06:37:35 AM
are you actually running SMF 2.1 beta2?
Title: Re: Google and cron.php 404 errors
Post by: Suki on September 09, 2015, 09:13:22 AM
I've seen this on my site as well, haven't really determinated exactly why googlebot is complaining, perhaps they are been too picky on the 1x1 pixel image cron.php can produce.

Either way, that file been indexed has no real benefit, it is beneficial if bots crawl that page, that is, "to visit" cron.php  but actually indexing that page, not so much.

You can safely added it to your robots.txt, use the googlebot user agent to target google alone:

User-agent: googlebot
Disallow: /cron.php

Can't remember if robots.txt prevents crawling + indexing altogether or just prevents indexing.
Title: Re: Google and cron.php 404 errors
Post by: Shkic on September 09, 2015, 09:50:37 AM
Quote from: Kindred on September 09, 2015, 06:37:35 AM
are you actually running SMF 2.1 beta2?

Yes.

Quote from: Suki on September 09, 2015, 09:13:22 AM
You can safely added it to your robots.txt, use the googlebot user agent to target google alone:

User-agent: googlebot
Disallow: /cron.php

Can't remember if robots.txt prevents crawling + indexing altogether or just prevents indexing.

Thanks. I will try.
Title: Re: Google and cron.php 404 errors
Post by: Shkic on November 17, 2015, 03:44:14 PM
Quote from: Shkic on September 09, 2015, 09:50:37 AM
Quote from: Suki on September 09, 2015, 09:13:22 AM
You can safely added it to your robots.txt, use the googlebot user agent to target google alone:

User-agent: googlebot
Disallow: /cron.php

Can't remember if robots.txt prevents crawling + indexing altogether or just prevents indexing.

Thanks. I will try.

Still got the same Google errors:

   
Quote780
cron.php?ts=1445697300
404
15.10.24

781
cron.php?ts=1445710335
404
15.10.24

790
cron.php?ts=1445709735
404
15.10.24

812
cron.php?ts=1445053875
404
15.10.24

181
404
15.10.24

371
cron.php?ts=1445666595
404
15.10.23