Web search crawlers seem to be entering posting mode lol

Ryan · maaliskuu 18, 2008, 05:11:31 AP

Basicly i have around 700 active web crawls i believe its google. Not sure tho cud be yahoo.
But problem ive seen is that they are now posting? I dont believe they actualy make the post but would be interested to know why crawler is actualy even doing this as it didnt before for sure lol.

Guest (74.6.21.161) 08:58:24 am Posting in MOON LANDING......
Guest (74.6.8.32) 08:58:06 am Posting in Recent weird occurrences....
Guest (74.6.18.117) 08:57:35 am Posting in Judgement Day.

Ryan · maaliskuu 18, 2008, 05:14:03 AP

Would denying ip 74.6.*
Kill alot more than just these crawlers?

Not sure if it be wise or not.

青山素子 · maaliskuu 18, 2008, 11:36:34 AP

The crawlers just follow links, so as the reply and new topic items are links, they are followed.

According to ARIN, 74.6.0.0/16 is allocated entirely to Yahoo/Inktomi. You should be safe blocking that range at the price that you won't be picked up on their index.

Eliana Tamerin · maaliskuu 18, 2008, 12:29:18 IP

What I did is add the noindex metatag to the post.template.php. That's seemed to have stopped them from crawling my post reply links (since I have guest posting enabled in one forum).

Just open up your post.template.php file and find the <head> tag. If there is a meta tag already there, add this around that. If not, just add it anywhere in the head (before the </head> tag).

<META NAME="ROBOTS" CONTENT="NOINDEX, NOFOLLOW">

EDIT: Btw, the IP for my google crawler (on my website) is 66.249.67.184. So I don't think banning the IP will help you.

Simple Machines Community Forum

Uutiset: