News:

SMF 2.0.19 has been released! Please update. Read more.

Main Menu

Robots.txt help

Started by hercululu, March 02, 2011, 02:04:03 PM

Previous topic - Next topic

hercululu

Ok, briefly, I have a fairly new board, the gist of which is parenting older kids. Privacy is an issue as no one wants their kids randomly googling up info of a personal nature that mom put on the internet. :)

I installed a very basic robots.txt

User-agent: *
Disallow: /

but decided that I wanted adsense, so I added

User-agent: Mediapartners-Google
Allow: /

I gave the bot forum access, but it still can't get past the robots.txt. Advice on how to allow only the mediapartners bot in? Incidentally, I don't mind if the registration page is googleable, but it just seemed easier to block it all for now. At some point the landing page will have articles and rss feeds instead of linking directly to the forum, but that won't be for at least a few weeks. At that point I will allow that part of the site to be spidered (and will probably be askign for help again LOL)

I know this has to be simple, but I am jsut not turning up much relevant to my needs in searches.

Advertisement: