Customizing SMF > Modifications and Packages
Forum Firewall
butchs:
Not sure since I do not know your settings. You could be blocking good guys... However, I created that feature because Bots were pretending to be google were hitting my site hard. This mod stopped them. If you search back some you will see that I recommend that you get a google webmasters account so you can adjust their settings and use the robots feature.
Texan78:
I have a google webmasters account set up. What is strange is when I visit those links they are valid links. I have attached a screenshot of my settings.
butchs:
Robots text validation is incorrectly set up. Read this.
If you have a robots text file it was not read by the mod during installation. If you do not have one you should make one. A correctly installed robots file will have the robots actions field populated. You need to set the crawl delay at google and in your robots text file to prevent blocking the wrong bot.
Texan78:
Thanks, I am trying to make some sense of this. I did and do have a robots.txt file in place. I added the additions suggested in the link you posted. What I am not sure of is how to set the crawl delay but not affect Google and what's a good crawl rate? Also where exactly in webmaster tools is this set? I couldn't find it. I have attached my robots.txt file. Does it look right?
-Thanks
butchs:
You uploaded two files... Please delete the robots,txt you are not using... Check out this link...
8)
PS: Change your salt. Everyone knows it now...
Navigation
[0] Message Index
[#] Next page
[*] Previous page
Go to full version