Simple Machines Community Forum

SMF Support => SMF 2.0.x Support => Aiheen aloitti: pepf - maaliskuu 05, 2020, 01:40:32 AP

Otsikko: robots.txt and addon domain question
Kirjoitti: pepf - maaliskuu 05, 2020, 01:40:32 AP
Domain setup:    smfforum.addondomain.com points to the folder maindomain.com/forum, with SMF placed in the 'forum' folder.
Access should be via the addondomain.com.

Should the robots.txt be
Disallow: /index.php?action=profile*
or
Disallow: /forum/index.php?action=profile*
or both, just in case a spider finds maindomain.com/folder/

Thanks in advance
Otsikko: Re: robots.txt and addon domain question
Kirjoitti: drewactual - maaliskuu 05, 2020, 12:06:06 IP
your first one should be sufficient. 

here's to hoping they actually honor it, right?

(https://www.cfb51.com/Smileys/fantasticsmileys/singing.gif)
Otsikko: Re: robots.txt and addon domain question
Kirjoitti: Kindred - maaliskuu 05, 2020, 12:52:03 IP
or just turn off profile access for guests
Otsikko: Re: robots.txt and addon domain question
Kirjoitti: pepf - maaliskuu 06, 2020, 12:54:26 AP
Thank you both. Turned guest access off.
Didn't even know that setting exists. Just noted that two SE spiders were visiting the profiles.