spider question

Started by pepf, December 03, 2021, 01:40:22 AM

Previous topic - Next topic

pepf

The spider log shows Alexa accessing, among other, many user profiles.
Question, do spiders see anything more than Guests do in the profile, which is nothing, and is it necessary to have additional protection?
In the robots.txt the profile is already shown as being off-limits for spiders.

TIA, Peter

Kindred

1- no. Spiders are guests 

2- just be aware,  many Spiders ignore robots.txt - like baidu
Слaва
Украинi

Please do not PM, IM or Email me with support questions.  You will get better and faster responses in the support boards.  Thank you.

"Loki is not evil, although he is certainly not a force for good. Loki is... complicated."

pepf

Thanks for reassuring me...

Advertisement: