Customizing SMF > Building Your Community and other Forum Advice

I think that my robot.txt has a problem

<< < (2/2)

Illori:
you would have to ask google, it depends there is no set amount of time that it will use to crawl any one site.

deprueba:

--- Quote from: Illori on November 19, 2011, 06:33:56 AM ---you would have to ask google, it depends there is no set amount of time that it will use to crawl any one site.

--- End quote ---

... is a question of time.

deprueba:

--- Quote from: Kcmartz on November 18, 2011, 05:57:45 PM ---Your welcome :) If it doesn't work, feel free to post again :)

--- End quote ---

That's working! Thank you very much.

nutn2lewz:

--- Quote from: deprueba on November 18, 2011, 12:34:29 PM ---... Yesterday I read in Google that their robots don't read the robots.txt if are uploaded in a subdomain. Then I deleted my forum robot.txt in the subdomain and paste the contents within the file robot.txt in the root. ....

--- End quote ---


Google does read robots.txt files in subdomains ...

https://developers.google.com/webmasters/control-crawl-index/docs/robots_txt
[nofollow]

Just thought I would post so others know that Google DOES look for robots.txt files in subdomains. Google will look for robots.txt at subdomain.domain.com/robots.txt which is probably located in your domain/subdomain directory.

Good luck, Barry

Navigation

[0] Message Index

[*] Previous page