Ok, briefly, I have a fairly new board, the gist of which is parenting older kids. Privacy is an issue as no one wants their kids randomly googling up info of a personal nature that mom put on the internet. :)
I installed a very basic robots.txt
User-agent: *
Disallow: /
but decided that I wanted adsense, so I added
User-agent: Mediapartners-Google
Allow: /
I gave the bot forum access, but it still can't get past the robots.txt. Advice on how to allow only the mediapartners bot in? Incidentally, I don't mind if the registration page is googleable, but it just seemed easier to block it all for now. At some point the landing page will have articles and rss feeds instead of linking directly to the forum, but that won't be for at least a few weeks. At that point I will allow that part of the site to be spidered (and will probably be askign for help again LOL)
I know this has to be simple, but I am jsut not turning up much relevant to my needs in searches.