Otsikko: robots.txt and addon domain question
Kirjoitti: pepf - maaliskuu 05, 2020, 01:40:32 AP
Kirjoitti: pepf - maaliskuu 05, 2020, 01:40:32 AP
Domain setup: smfforum.addondomain.com points to the folder maindomain.com/forum, with SMF placed in the 'forum' folder.
Access should be via the addondomain.com.
Should the robots.txt be
Disallow: /index.php?action=profile*
or
Disallow: /forum/index.php?action=profile*
or both, just in case a spider finds maindomain.com/folder/
Thanks in advance
Access should be via the addondomain.com.
Should the robots.txt be
Disallow: /index.php?action=profile*
or
Disallow: /forum/index.php?action=profile*
or both, just in case a spider finds maindomain.com/folder/
Thanks in advance
Otsikko: Re: robots.txt and addon domain question
Kirjoitti: drewactual - maaliskuu 05, 2020, 12:06:06 IP
Kirjoitti: drewactual - maaliskuu 05, 2020, 12:06:06 IP
your first one should be sufficient.
here's to hoping they actually honor it, right?
here's to hoping they actually honor it, right?
Otsikko: Re: robots.txt and addon domain question
Kirjoitti: Kindred - maaliskuu 05, 2020, 12:52:03 IP
Kirjoitti: Kindred - maaliskuu 05, 2020, 12:52:03 IP
or just turn off profile access for guests
Otsikko: Re: robots.txt and addon domain question
Kirjoitti: pepf - maaliskuu 06, 2020, 12:54:26 AP
Kirjoitti: pepf - maaliskuu 06, 2020, 12:54:26 AP
Thank you both. Turned guest access off.
Didn't even know that setting exists. Just noted that two SE spiders were visiting the profiles.
Didn't even know that setting exists. Just noted that two SE spiders were visiting the profiles.