Possible 'new' way to control A.I. bots?

[AlanDewey]

crawler.amazonbot.amazon killing my forum.  My forum has over 200,000 posts so it would take quite a while for the bots to go through it all.  They are using dozens of I.P. addresses, so it is probably impossible to manually ban by I.P.

So I turned on hostname lookups and made a permanent ban on amazonbot.amazon.  22,000 ban log entries in the last 26 hours and they are not slowing down.  I understand that hostname lookups puts a lot of overhead on the server, but it would be less work than serving 200,000 posts, plus attachments.

So I have two thoughts/questions:

1) Instead of SMF replying with the ban information, why not just ignore all requests (do not reply with even one packet.) Maybe they would go away after some time.  My server indicates that it replies with 3386 bytes at each bot request; that was 74 megabyte in the last 26 hours. Or, instead of serving ban reason, just reply with HTTP 500 ?  (or 418  :-)

2) Instead of 22,000 hostname lookups each day, build a store of information containing the I.P.s used by the bot today?  The ban would, after the 2nd or 3rd IP used for that bot, look at the store of IPs and ban that IP.  This "ip ban" database(?) would be cleared once(?) per day(?) and start over.  I think that then there would only need to be hostname lookups less than 100 times per day (depending how many IPs they are using.)



"AI" is simply a huge industry plagiarizing everybody's work.

[Aleksi "Lex" Kilpinen]

1) The ban is designed to be informative, because it is designed with actual users in mind. Consider banning on the server level, through htaccess for example, if you want to stop the bots from ever reaching SMF in the first place.

2) See number 1, this sounds more suited to be a mod. Some mod might even already exist that does something similar. You could also just find out the IP range used, and ban them all at once, through htaccess for example.