Basicly i have around 700 active web crawls i believe its google. Not sure tho cud be yahoo.
But problem ive seen is that they are now posting? I dont believe they actualy make the post but would be interested to know why crawler is actualy even doing this as it didnt before for sure lol.
Guest (74.6.21.161) 08:58:24 am Posting in MOON LANDING......
Guest (74.6.8.32) 08:58:06 am Posting in Recent weird occurrences....
Guest (74.6.18.117) 08:57:35 am Posting in Judgement Day.
Would denying ip 74.6.*
Kill alot more than just these crawlers?
Not sure if it be wise or not.
The crawlers just follow links, so as the reply and new topic items are links, they are followed.
According to ARIN, 74.6.0.0/16 is allocated entirely to Yahoo/Inktomi. You should be safe blocking that range at the price that you won't be picked up on their index.
What I did is add the noindex metatag to the post.template.php. That's seemed to have stopped them from crawling my post reply links (since I have guest posting enabled in one forum).
Just open up your post.template.php file and find the <head> tag. If there is a meta tag already there, add this around that. If not, just add it anywhere in the head (before the </head> tag).
<META NAME="ROBOTS" CONTENT="NOINDEX, NOFOLLOW">
EDIT: Btw, the IP for my google crawler (on my website) is 66.249.67.184. So I don't think banning the IP will help you.