Abuse prevention: List the IP address that did the most accesses in last hours

[hartiberlin]

Hi,
what I would need is a Abuse prevention or
better said a stats mod, which shows,
which user has accessed the forum the most times
in the last 1 day or 2 days or last hour.

This way I can find out, if a user is trying to use a script to
call up the forum all the time and make too traffic to bring the
forum down.

This should also register the traffic from non members,
so the IPs from the guests must also be logged...

Is this possible ?

Many thanks.

Regards, Stefan.

[hartiberlin]

Does Google analytics show this
or which online service, which offers a script to include into
the forum pages could  show traffic per IP address per day ?

Many thanks.

[karlbenson]

Analytics would probably be a good idea.

As for logging all that information via smf, that would likely slow your forum down considerably.

[hartiberlin]

Okay,
a friend of mine programmed now a PHP script for
me, who logs all the IP addresses and counts the hits up
to my server into another database, so I can now see,
which IP address is making how many accesses.

So I found now, that the Google spider crawler just made
much too many accesses and traffic.

I just changed the Google bot via the Google Webmaster Tools
to crawl my site slower...

By the way, what would be the optimal settings
in robots.txt to allow Google only crawling for maybe 20 Mbytes
traffic a day ?

Many thanks.

Regards, Stefan.

[karlbenson]

just set a crawl-delay via robots.txt for each spider.

Basically to tell it to wait 20 minutes before spidering another page.

[hartiberlin]

Unfortunately the Google bot
does ignore the Crawl-delay
command inside robots.txt

This is also stated in Google Webmaster-Tools
help.

Any other possibility ?

Is there any script out that can just limit the
traffic guests can do ?
Maybe after 100 MB/day just give a 503 error page ?
Many thanks

Regards, Stefan.

[karlbenson]

Googlebot isnt normally the problem bot.
It is usually pretty constant throughout the day eg 1 per 5 minutes, every 10 minutes.
So its affect over your server is minimal.

But Yahoo Slurp will do 1 or 2 all day then suddenly make 30 requests a minute.
This can have an effect over your server and amount to a ddos attack.

[hartiberlin]

The Google Bot really makes the most traffic on my
forum
www.overunity.com

As I have a new script that tracks every hit behind
the bot-trap.de script, that comes before it,
I can see it right down to each page call.

Via bot-trap.de all the other spiders are already canceled out,
only Google and Yahoo Inktomi come still through
as this shows:

ip-address                      counter                    lasthit
66.249.72.241                 6278        2007-12-26 02:58:02
74.6.8.105                       937            2007-12-26 02:57:26


The first is Google, the second Yahoo Inktomi.
This is after a few hours.
Normal power user have accesses in the 500 to 600 hits range.