Check out the SMF Function DB!
Started by butchs, January 15, 2011, 11:00:37 AM
User-agent: *Disallow: /cgi-bin/Disallow: /smf/Sources/Disallow: /smf/Themes/Disallow: index.php?action=admin*Disallow: index.php?action=calendar*Disallow: index.php?action=login*Disallow: index.php?action=printpage*Disallow: index.php?action=profile*Disallow: index.php?action=register*Disallow: index.php?action=search*Disallow: index.php?action=stats*Disallow: index.php?PHPSESSID=*Disallow: index.php?*rss*Disallow: index.php?*wap*Disallow: index.php?*wap2*Disallow: index.php?wwwRedirect*Crawl-delay: 5
Quote from: An0nymousHelper on March 04, 2011, 09:08:32 PMThats just two of them and there are quite a few more. If anyone knows what this is it would be greatly appreciated if you could let me no! Thanks!
Quote from: butchs on March 05, 2011, 08:19:45 AM2899 is from a mobile phone and as far as I know Google is not using a mobile phone.
Quote from: butchs on March 05, 2011, 08:19:45 AMWhy would google need to find your Sources directory? They should not be snooping there. I think you should block access in your robots.txt file. Making changes to your robots.txt file is a important security measure. It keeps the good bots away from sensitive areas. I recommend a robots.txt file like the following:...
Quote from: butchs on March 05, 2011, 08:19:45 AMThe latest BB mod has a nice reverse DNS check that catches several fake bot attempts.
Quote from: Glasso on March 05, 2011, 10:43:35 AMButchs, this is causing some genuine googlebot requests to be blocked as I see in my webmaster logs. I can send you some trail that I had with Michael where he thought roundtripdns is not fully reliable, if you believe it is worthwhile to go deeper.
QuoteThat could have been avoided. Not to be rude but please do not make a negative statement unless you have read and understand all the help "?" icons. This question will be answered to it its correct place: BB forum.
Quote from: Jesna on March 09, 2011, 12:26:45 PMIm thinking of using the country test but im not sure about those country codes i have to fill in. Can I use the country codes from here http://countrycode.org/example: South africa is ZA / ZAF and then i put ZA / ZAF in the field