News:

Wondering if this will always be free?  See why free is better.

Main Menu

X-Robots-Tag HTTP header vs Robots.Txt

Started by galanb, April 01, 2018, 05:55:51 PM

Previous topic - Next topic

galanb

I have a small SMF Forum that is closed to guests and only accessible to registered users.  This forum is a private forum and what I want to do is to stop it being indexed (and if possible not listed) by search engines.  My assumption is that because a login is required to gain access to the forum, none of the content is accessible to the search engines, other than the forum's home (login) page.

I have looked at articles about setting up a robots.txt file but I am not sure what should be in the file or where it should be placed in the server or forum file hierarchy.  I would appreciate any help with this.

A second approach I read about on this page https://yoast.com/prevent-site-being-indexed/ [nofollow]  (which includes a short video by Matt Cutts, Google's former head of webspam team).  This page talks about using an X-Robots-Tag in the .htaccess file as being a better approach than using a robots.txt file.  If I were to take this approach, which .htaccess file in the server or forum hierarchy should I put the text in?  Thanks.

vbgamer45

Robots.txt is good a start. You can make your forum de indexed from google. With google's webmaster tools.
I haven't heard of the X-Robots tag before.
Community Suite for SMF - Take your forum to the next level built for SMF, Gallery,Store,Classifieds,Downloads,more!

SMFHacks.com -  Paid Modifications for SMF

Mods:
EzPortal - Portal System for SMF
SMF Gallery Pro
SMF Store SMF Classifieds Ad Seller Pro

Advertisement: