Simple Machines Community Forum

SMF Support => SMF 2.0.x Support => Topic started by: casp3r on May 06, 2008, 05:00:56 PM

Title: Stop spiders viewing profiles?
Post by: casp3r on May 06, 2008, 05:00:56 PM
Version: SMF 2.0 Beta 3.1 Public

I've noticed in the 'Search Engine' logs that some are viewing members profile - How do I stop this?

Many thanks in advance

Michael
Title: Re: Stop spiders viewing profiles?
Post by: Kindred on May 06, 2008, 05:03:07 PM
turn off the guest's ability to view profiles in permissions
Title: Re: Stop spiders viewing profiles?
Post by: casp3r on May 06, 2008, 05:08:39 PM
Already have done that weeks ago but tonight I noticed that a profile was still being viewed.
Title: Re: Stop spiders viewing profiles?
Post by: karlbenson on May 06, 2008, 05:18:07 PM
Making them private still won't stop spiders attempting to crawl them.

You would need to use a robots.txt to stop them crawling them.
In the base of your domain create a file called robots.txt and paste into it.

User-agent: *
Disallow: /index.php?action=profile


You can also block alot of other sections or individual spiders.  You can take a look at my robots.txt (http://www.youposted.com/robots.txt) which block different things for different spiders.
Title: Re: Stop spiders viewing profiles?
Post by: casp3r on May 06, 2008, 05:20:54 PM
Many thanks. Added to txt file. Will post back in a few days and let you know how it goes.
Title: Re: Stop spiders viewing profiles?
Post by: Kindred on May 06, 2008, 07:38:54 PM
please do note...   if Guest viewing of profiles is OFF, then the spider is NOT viewing the profile...  merely attempting to do so.