How is Google viewing member profiles?

[ralent]

I was a little shocked when I saw that Google is viewing my member profiles. The only Guest permissions are Search for posts and topics and View Attachments.

I recently upgraded to 2.1.1 with no mods installed.

https://signalquiz.com [nofollow]

[Oldiesmann]

SMF is showing what Google is attempting to do - it doesn't check whether they actually have permission to do that (because that's far more work). In reality, Google is seeing a screen saying they need to login or register in order to do that.

[ralent]

SMF is showing what Google is attempting to do - it doesn't check whether they actually have permission to do that (because that's far more work). In reality, Google is seeing a screen saying they need to login or register in order to do that.

Ok well that's good. I can see them following the link to 'johntheman's profile' because he was online today but I wonder how they found a link to 'StefanSchweitzer's profile', he was last active a month ago?

[Aleksi "Lex" Kilpinen]

Search bots collect lists of urls to check, and keep an index to periodically return to in order to check if they've changed. It's just how they work, all it takes is for them to see a link somewhere once - or recognize the platform well enough to guess url structures.

[Arantor]

SMF is showing what Google is attempting to do - it doesn't check whether they actually have permission to do that (because that's far more work). In reality, Google is seeing a screen saying they need to login or register in order to do that.

As per the screenshot, Google did get an error trying to view registration (because you can see the little red (!) icon there), but I don't see it getting a little (!) icon for the others...

I just tried it myself and correctly got stopped so maybe the logging code doesn't work so well there?

[ralent]

Search bots collect lists of urls to check, and keep an index to periodically return to in order to check if they've changed. It's just how they work, all it takes is for them to see a link somewhere once - or recognize the platform well enough to guess url structures.

Yes that makes perfect sense. Thank you.

[ralent]

SMF is showing what Google is attempting to do - it doesn't check whether they actually have permission to do that (because that's far more work). In reality, Google is seeing a screen saying they need to login or register in order to do that.

As per the screenshot, Google did get an error trying to view registration (because you can see the little red (!) icon there), but I don't see it getting a little (!) icon for the others...

I just tried it myself and correctly got stopped so maybe the logging code doesn't work so well there?

Thanks. I didn't know what the (!) signified. And yes I tried to reach the profiles as a Guest numerous times and none of my attempts have been recorded.

[Steve]

As it appears that you're satisfied with the explanations, I'll mark this solved. If you have any other questions about this specific topic, by all means, mark it unsolved and ask away. 

[live627]

Also, hover over the icon to see the error message.

[ralent]

As it appears that you're satisfied with the explanations, I'll mark this solved. If you have any other questions about this specific topic, by all means, mark it unsolved and ask away. 

Yes. Thanks a lot.

[ralent]

Also, hover over the icon to see the error message.

Good tip. Thanks!

[movierchives]

I've just solved this problem myself today because I noticed the bots has far more access since the upgrade to 2.1, I just created a new membergroup called it Spider and put the Spiders under that only allowing them public access like a guest

[Kira_]

You can use the file "robots.txt " to control the behavior of search bots. Ask google for details.
For example, this is my robots.txt

Code Select Expand

User-agent: *
Allow: /*type=rss
Allow: /*rss
Allow: /*board
Allow: /*topic
Allow: /*.xml
Allow: /*.css$
Allow: /*.js$
Allow: /*.png$
Allow: /*.jpg$
Allow: /*.gif$
Allow: /?action=sitemap
Allow: /?action=sitemap;xml
Allow: /Themes/default/css/
Allow: /Themes/default/images/
Allow: /Themes/default/scripts/
Disallow: /administrator/
Disallow: /cache/
Disallow: /components/
Disallow: /editor/
Disallow: /help/
Disallow: /images/
Disallow: /includes/
Disallow: /language/
Disallow: /mambots/
Disallow: /media/
Disallow: /modules/
Disallow: /templates/
Disallow: /installation/
Disallow: /go.php
Disallow: /*action
Disallow: /*board*wap2
Disallow: /*topic*wap2
Disallow: /*wap2
Disallow: /*topic*.msg
Disallow: /*.msg
Disallow: /*.new
Disallow: /*board*sort
Disallow: /*sort
Disallow: /*topicseen
Disallow: /*prev_next
Disallow: /*all
Disallow: /*PHPSESSID
Disallow: /attachments/
Disallow: /avatars/
Disallow: /Packages/
Disallow: /Smileys/
Disallow: /Sources/
Disallow: /Themes/
Disallow: /android-app:
Disallow: /ios-app:
Sitemap: https://xxx/?action=sitemap;xml
Sitemap: https://xxx/sitemap.xml
Crawl-delay: 5
Clean-param: PHPSESSID /index.php
Host: xxx

Note: SMF 2.1.x can't "?action=sitemap"

[eloycartw]

Is there any way to just eliminate guests entirely? Then anyone not registered hits the same login wall. Also, I realize this might should be an entirely separate thread so please feel free to move it, if so...
Thank you

[Steve]

Turn registration off, and in each board, don't allow guest access.

[Arantor]

I'd go for Admin > Configuration > Features and Options > Allow guests to access the forum

Makes any access require a login.

[Steve]

Or that.

[eloycartw]

We did both, but still find bots trying to index the board. Presumably their efforts are ineffective, but in case wee have a client that asks about it sometime...

[shadav]

think of it like this
they knocked at the door so it triggered a log
but no they weren't able to actually enter

and keep in mind if you deny guests access to the forums you will most likely not receive many new members as they will not be able to see any content on your site and have no real reason to sign up