Topic: Search Engines

[petewadey]

I have Search Engines as a member group and given them certain permissions, somewhere between Guests and Members.
Most bots turn up in this group as if by magic, obviously something SMF do behind the scene. How often is this updated?
I ask because when I check guests ip's, I sometimes find genuine bots, and add them manually to the search engine group.
Mainly because I have given them more permissions than they would otherwise have as a guest.
Am I wasting my time?, Should I just be patient and wait for the data to update?
Thanks
Pete

[Illori]

not all bots will crawl your site and define themselves as a bot. if they dont they will not let the forum put them in the group. IMO you will hurt your ranking more if you give spiders more access then a guest gets.

[Kindred]

it actualy violates Google's terms to have the bot see more than a standard guest.

[petewadey]

Sorry, I was wrong in my original message. I allow guests to see that pages are there for members only, but don't wish bots to see those. That is the only difference.
Can anyone answer my original message please?

[Illori]

since spiders dont always identifiy themselves you will not always get what you want from that. as if they dont identify they will see the pages and see when they do they cant and down rate your site.

[petewadey]

In that case, I'd better rethink my permissions. Thanks for your help

[Kindred]

Additionally, if I can see something, I should be able to get to it.
If I click on something and the system tells me "you have to register to complete this action" - I leave the site and never come back.