Website copiers/downloaders!

[nahawand]

Hi all,

I recently discovered that some of my members have been using website copiers (like this one: hxxp:www.httrack.com/ [nonactive]) to download the entire content of the smf forum! I have noticed huge surges in bandwidth usage and cpu usage. And after some detective work, I found out that two members (banned forever!) were using the afore-mentioned site copier.

I looked through the mod site for a mod that would solve this problem but found nothing. I also looked through the admin options in my forum, but found nothing relevant.

Is there anything that I, as an administrator, can do to block people from downloading the entire contents of the my forum using such programs?

Thanks,

[青山 素子]

Unless you have access to the server itself, you really can't do much to limit use or anything like that. While you can try blocking certain user agents, those are usually quite easy to change to look like a normal browser.

[Dragooon]

What if we have?
How can we block these types of things? Any ideas or links?

[青山 素子]

I've had to deal with these things a bit, and you really can't stop them unless you catch them in action and block by IP. You can, however, slow them down so they don't cause too much harm when their 20 threads try to connect.

I normally use mod_cband (site seems down at the moment) on Apache 2.x.

The settings I'm currently using for the sites that seem to get the most abuse is:

CBandRemoteSpeed 5000 10 10

This sets each client to 5mbit max speed (basically unlimited for my purposes), but only 10 connections max (good for real browsers that make multiple connections) and 10 requests per second. This keeps the server from getting overwhelmed by connections most of the time.

[motumbo]

There really isn't anything you can do other than write your own script to record each and every page view into a database, then check on each pageview how many that user has and if they have more than you allow, you can blank screen them or redirect them somewhere else.

I've been thinking about doing something like this.  Say, if you view more than 10 pages per minute, cut them off.

[青山 素子]

I've been thinking about doing something like this.  Say, if you view more than 10 pages per minute, cut them off.

* Motoko-chan points at mod_cband

You can easily limit the number of requests per second, which would do most of what you want. It won't ban, just slow down, so you don't ban actual users who are just click crazy (loading 20 tabs all at once, say).

[H]

I've been thinking about doing something like this.  Say, if you view more than 10 pages per minute, cut them off.

* Motoko-chan points at mod_cband

You can easily limit the number of requests per second, which would do most of what you want. It won't ban, just slow down, so you don't ban actual users who are just click crazy (loading 20 tabs all at once, say).

Indeed if you resort to banning you will probably end up banning real users. On WHT I was temporarily banned during a time they were having a DDOS as I frequently opened lots of connections while going through the unread messages list.