banning entire countries

[galahad]

I guess I'm sorry I even asked the question.  So that some of you may better understand why I did....

I own and operate a web forum dedicated to firearms.  While there are many countries that allow private ownership of firearms, one country that does not, and happens to represent somewhere around 75% of the current spam we are getting, is China.   While it is theoretically possible for a Chinese citizen to want to participate in a firearms forum I "suspect" that it would violate Chinese law and is highly unlikely. 

So, I would like to eliminate signups from China.  In conjunction with that I would most probably post a note in the News area that would advise anyone who is having difficulty registering to contact the administrator who will override the system to allow registration.

As an example, I have routinely been denying membership to those with IP's from Bangladesh.  I received an e-mail from a citizen of Bangladesh and he is now a member.   So yes, there are instances where an exception must be allowed but when it is one in a thousand I believe that it justifies actions against entire countries. 

[Illori]

not all admin are knowledgeable to know what could happen by banning a whole country. not to mention that with dynamic ip addresses it could be possible to accidentally ban someone in another county.  if this were to be done a mod is best as then the admin has a choice if they want the feature, this would keep those admin that do not know what it could do away from killing their possible membership.

[青山 素子]

With the fragmentation of IPv4 (and the exhaustion by APNIC, the regional authority), and the portability of IPv6 addresses, trying to block countries will be very, very difficult.

It's an awful, awful technical solution.

[galahad]

It may be technical but what I'm using now, http://whatismyipaddress.com/ip-lookup  is what I use to make the determination now.  If it's inaccurate there isn't much I can do about it.  Automating it would be a godsend and since there is a location given, if the location says "China" then I want to not accept the account.

Technical?  Perhaps it is to make it 100% accurate but since I would allow a manual fall-back process I do not require 100% accuracy. 

[Wizzlefits]

It seems to me that a more efficient/easier to maintain blocking, would make use of an external api, that if a lookup returned China, block em and send them to a page explaining why.

If there's a mod that does that already, bloody great! if not... well who's gonna make one first? LOL!

[butchs]

Well if you or your host has GeoIP installed or you use Cloudflare, Forum Firewall will block based on country code.  As I said earlier I block with htaccess first and let FF polish things up.

Both FF and GeoIP are ipv6 compatible.

[青山 素子]

It may be technical but what I'm using now, http://whatismyipaddress.com/ip-lookup  is what I use to make the determination now.  If it's inaccurate there isn't much I can do about it.  Automating it would be a godsend and since there is a location given, if the location says "China" then I want to not accept the account.

What I meant was that it's a bad technical solution. As in, it's a technical solution that is bad. Most IP registries aren't 100% accurate, and services that bundle those databases are likely even more inaccurate (using old data). That "China" or "Cambodia" might very well have been reallocated to New Zealand or Australia a few weeks ago.

A better solution is to use questions that are not general-purpose and rather specific so that someone who would be a good contributor to your forum would be able to answer.

Between focused questions and httpBL for SMF, I get less than one spam post per week (on a guest-post board) and nearly 0 bad registrations.

[Vince S]

Is this discussion sufficient to show that some sort of enhanced spam management feature should be built in? And get it on the "to do" list?

To reiterate, spam is a problem FOR EVERYBODY. All that varies is a) how good they are at implementing solutions, and b) what the geographic (or other) nature of their forum is.

This discussion has focussed on the reality that certain countries do seem to produce a large quantity of particular types of spam, this should not be ignored! The debate here has got a bit derailed by whether or not country banning is a good idea, whereas the actual question is whether it is practically possible - where some doubt obviously exists. Whether YOU want it doesn't need to go through a filter of how smart a cookie you are. A lot of people could use this and that's a fact!

For my part we are a social motorcycle group. Happy to share technical info with the world but have no need whatsoever for their participation. How many others use SMF for "local" discussions, have to be a reasonable percentage I would guess and, by the nature of these groups, would be likely to have the least vocal members, especially in a forum like this. I have certainly seen many abandon the whole concept of trying to run a forum for no other reason than nobody knew how to deal with the spam as it is very embarrassing to be actively sending all these socially unacceptable offers to your friends. Period! This reality cannot be ignored, surely we can grasp that?

The simple, bottom line position is that it would be really handy to have country exclusions. Or, put the other way, Country INCLUSIONS and everybody else can just suck it up or ask nicely to be let in. Remember we are talking about well meaning volunteers here, not IT gurus! And we are just talking registrations too. How much more helpful would it be to have what a webmaster sees be polite requests from interested persons for them to register, rather than registration requests from probable spammers that one has to check out their geography as best as one is able, guess from the name they register, and decide whether to permit them or not, only to have to go back and delete them and their posts if you get it wrong?

To expand the logic further it is a far better test that a potential registree sends a 2 line request to be let in than any captcha, question or any add-on filter can manage. Not saying it is an answer for everybody, but it would be a damn fine feature for many! So can we elevate the discussion to make a decision and take some action please?

[Vince S]

Let me just add the perspective that I have implemented a kinda country ban for some years, as I noted below the details are posted here http://www.simplemachines.org/community/index.php?topic=433172.msg3042424#msg3042424.

Edit: for anyone that doesn't know this is using the "Ban Triggers" feature under Members and works for 1.* and 2.0, even carries over on upgrade.

Sure as you can see in the above referenced post I have shut out a lot of countries where the spam attempts are one off's. But look at how many requests I DIDN'T have to deal with as a consequence of taking this action, I have just downloaded in hit order the spam registration attempts per below. Now my descriptions are not necessarily accurate, but this is a clue what the order of magnitude of the problem has been over the last few years:

66.*.*.*    66. Canada    3299    
194.*.*.*    194. Virgin Islands    1506    
94.*.*.*    94. France, Italy et    767    
95.*.*.*    95. Russia    432    
92.*.*.*    92. Ukraine    242    
89.*.*.*    89. Germany    225    
74.*.*.*    74. Texas    77    
91.*.*.*    91. Ukraine    71    
77.*.*.*    77. Russia    66    
78.*.*.*    78. Great Britain    53    
68.*.*.*    68. United States    51    
173.*.*.*    173. United States    48    
85.*.*.*    85. Sweden    46    
217.*.*.*    217. Germany    39    
64.*.*.*    64. United States    38    
222.*.*.*    222. Japan - China    19    
188.*.*.*    188. Latvia    17    
201.*.*.*    201. Brazil    17    
212.*.*.*    212 Israel    14    
195.*.*.*    195. Latvia    8    
88.*.*.*    88. Finland    6    
98.*.*.*    98. USA    6    
178.*.*.*    178. Ukraine    6    
109.*.*.*    109. Russia    5    
76.*.*.*    76. Finland (?)    5    
24.*.*.*    24. Charter Communic    5    
82.*.*.*    82. Russia    5    
129.*.*.*    129.* USA    4    
71.*.*.*    71. Finland & US    4    
84.*.*.*    84. Russia    4    
111.0-219.*.*    111.0-219 Pakistan    4    
204.*.*.*    204. US    4    
96.*.*.*    96. USA    4    
174.*.*.*    174. United States    3    
180.*.*.*    180. China Japan    3    
46.*.*.*    46. Middle East    3    
69.*.*.*    69. United States    3    
60.0-223.*.*    60.0-223. China    3    
93.*.*.*    93. Netherlands    3    
110.*.*.*    110. Pakistan, China    3    

[Kindred]

well, I'll let a developer have the final word.... but IMO, this is not something that belongs in SMF's default installation.

[青山 素子]

Is this discussion sufficient to show that some sort of enhanced spam management feature should be built in? And get it on the "to do" list?

What kind of "enhancements" would you like?

To reiterate, spam is a problem FOR EVERYBODY.

I don't think anyone is debating this.

For my part we are a social motorcycle group. Happy to share technical info with the world but have no need whatsoever for their participation. How many others use SMF for "local" discussions, have to be a reasonable percentage I would guess and, by the nature of these groups, would be likely to have the least vocal members, especially in a forum like this.

For local groups, anti-spam questions are even more effective! Especially if the forum is used as an adjunct to in-person meetings. Something as simple as "where do we go to eat after the meetings?" or "Who is currently secretary at the meetings" should stop all fake registrations dead.

Do you know the name of the character who wears a seifuku in the show Inuyasha? You may not, but I'm sure basically anyone who wants to participate in my anime community would know.

The simple, bottom line position is that it would be really handy to have country exclusions. Or, put the other way, Country INCLUSIONS and everybody else can just suck it up or ask nicely to be let in. Remember we are talking about well meaning volunteers here, not IT gurus!

I think the fact that you're not a networking person shows when you think something is simple like banning a country. It's not easy, the address allocations change fairly quickly, and it's bad technically for that reason. I know a lot of "security" products that blackholed requests from "bad" unallocated ranges. Guess what happened when IANA finally allocated them? Easy answer: lots of problems as legitimate people were banned from a lot of places until the equipment could be updated (if it was under support) or replaced fully (if it was not). I'm sure there are still a few places having problems.


Here, I'll take an example from your list above to show the problems:

95.*.*.*    95. Russia    432    

95.0.0.0 - 95.15.255.255 = Turkey
95.16.0.0 - 95.23.255.255 = Spain
95.24.0.0 - 95.32.255.255 = Russia
95.33.0.0 - 95.33.255.255 = Germany
95.34.0.0 - 95.34.7.255 = Norway
and so on....

92.*.*.*    92. Ukraine    242    

92.0.0.0 - 92.31.255.255 = United Kingdom
92.32.0.0 - 92.35.255.255 = Sweden
92.36.0.0 - 92.36.127.255 = Russia
92.36.128.0 - 92.36.255.255 = Bosnia and Herzegovina
92.37.0.0 - 92.37.7.255 = Slovenia
and so on...


I think that's enough. I don't want to get rate-limited by APNIC and RIPE for all the lookups.

Hopefully that's enough of a show that blocking /8 blocks is dangerous and will ban places you might want to access your site. Heck, I was seeing blocks as small as /28, although I condensed where they were in the same country.

[Wizzlefits]

I'll have to second that. Mainly because a mod would be much better, and be able to adapt and change code as needed soooooooo much faster.
Take Butchs FF as an example.

[Vince S]

Could I further explain, I am an engineer and a real trap for "this type" is to get mired in the details while everyone else's eyes glaze over, upshot of doing that here is we get the "do nothing" option. Which is fine if we decide that but is a shame if that is just what happens without proper decision making. I have certainly glossed over "some details", which 青山 素子 is quite practically and helpfully pointing out, and particularly including implementation issues which are very important. Believe you me I have spent enough time "dissecting" IP's to see what is really going on so I well understand a lot of aspects of the issues, which I meant to cover by "my descriptions are not necessarily accurate".

What I actually do after getting a spam registration is note the range that the IP check says it covers, sometimes it is a lot but usually fairly tight. Then I will replace the second field with a 0 and see what range I get, then repeat with 1 above the highest in the last range, ad infinitum until I get a picture of what is going on generally with the IP range, sometimes jumping to 255 and working backwards. THEN if all the countries that come up are areas that are irrelevant to us I take the full first number IP out, but any uncertainty and I just pick whatever range seems applicable. My naming comes from the country that initiated the spam, not the countries covered by the IP range nominated. Please remember that this is NOT an insult to the good folk of the particular country, if ever they want to join up for any reason the rejection note tells them how, so they have to manually send in a request, it is not an insult that one of their countryfolk behaved badly, nor would it be a surprise to most.

I agree that asking stronger questions SOUNDS like a potentially good solution, but in reality what we anyway want to do is about 70% newsletter info dissemination and 30% engagement on topics. What we most want is physical participation in activities but in practice people go through a staged engagement process from first hearing about "oh there's something on the web about a local group" to when they would know the answer to many q's. My fundamental philosophy is that it needs to be hard for the spammers and easy for the real people, not the other way round. Webmasters of course are real people, this needs to be remembered!

What this thread fundamentally needs to answer is something like this, which I say as an outsider not embroiled in the doing of this type of work:

a) "Do we have a problem that existing tools / feature plans won't handle well, or at least adequately?"

b) "Is there a viable approach to the problem we can identify?"

c) "So what is the "feature brief" that we would like to pass to the development team?"

What this FUNDAMENTALLY requires is a decent definition of "the problem", as much to check if it is real as to consider if we can solve it. There is no drama saying the problem is real, but recognise that trying to solve it before IPv6 hits the masses would be a waste of effort. That might mean the real solution is to produce a bit of a guide to people dealing with spam - and explain what things like "oh, I use htaccess" mean out of IT-fairy land to real people (this is the non-IT perspective of the world, and it can certainly be managed fairly and easily).

Before I go on am I on a path here that is Ok.........?

[Vince S]

I guess if I was to answer the simple q of "What kind of "enhancements" would you like?", and jumping to the end of the process I tried to map out in my just gone post, it would be:

Preferably an option on the registration page to select "Limit registrations by IP?", which opened up a separate tab with a table that can have any disclaimer it likes about currency of the info. Above the table would be a "Refusals Action" drop down which had options like "Refuse completely with standard rejection email" , "Tell registrant that their IP fails our default filters so they must pass test questions to register", "Send the following message to registrants" with a text input box below, or whatever, using the detailed knowledge of methods that work here.

Then below that have a table that is hopefully derived from some regularly updated source and has columns for "Permit" or "Refuse" or "Test Q's" registrations from the following IP Ranges: There should be "Select All" buttons at the top and obviously the drop down actions and table buttons are complementary, possibly we only need the columns. This is detail and needs to be tested for the most effective easy to use solution.

Get this right and I'll just bet the uptake / conversion rate of SMF would go through the roof as so many people are sick of dealing with spam. Once the news gets out that there is a real answer here for this stuff you just watch them come a-running! But is something like this practically achievable and is there the appetite to do it?

[青山 素子]

The problem here is that spam is an economic and social problem, and you're trying to use a technological measure to fix it. That kind of thing never works well.

There is no such thing as a way to end all spam, or even a single way to stop most spam. Look at the progression of spamming:

• Most spammers use simple automated tools. Webmasters implement something simple like a CAPTCHA/image verification to stop these
• The spammers update their tools to use simple OCR to solve CAPTCHA. Webmasters upgrade to more difficult images or services like reCAPTCHA.
• The spammers update their tools to solve the more complex images, or use special human-backed services to solve them. Webmasters switch to using simple question/answer pairings (2+2=?).
• The spammers move to use real humans that work for a very minimal cost.

That's where we are at now. Notice that each time the response is technical, which is fairly quickly worked around by the spammers who have a large economic interest in keeping their posts rolling.

The latest technical idea is to ban IPs from certain countries to prevent direct access to the forums. The simple solution the spammers will use is to move to zombie computers based in more "desirable" countries, which I already see happening.

The only solution is to find ways to make posting on your forum economically painful for spammers. If your forum is too expensive to spam, it won't happen.



Preventing unwanted registrations/posts is like planning for security. The more difficult you make it to pass, the more complex and painful it is for legitimate users to get past it. When planning, you have to determine a few things:

• What's your intended audience?
• How much spam are you willing to put up with? (None is not an option, unless you don't want to run a forum.)
• How difficult are you willing to make it for registration?

Depending on the answers, there are many different ways to make spamming too expensive to pursue.

For a small, limited audience you can use manual registration approval. This will stop all spam posts, and if the potential members let you know their registration e-mail, you can easily approve their accounts while cleaning bad registrations.
Audience: medium. Level of spam: none. Difficulty: high.

For a forum used as an extension of a club, manual account creation, or using questions where the answers are only known to members will work very well.
Audience: small. Level of spam: low. Difficulty: moderate.

For general-membership but niche forums, questions related to the subject matter that are obscure enough to not be found in 5 seconds in a search will also do wonders.
Audience: large. Level of spam: low to moderate. Difficulty: moderate

What will not work is banning large IP blocks. The spammers will just move fully to using botnets on US and other European IPs. More expensive, yes, but still cheaper than the earnings from spam.



Edit:

Latest bans via Fail2Ban on my server:
BellSouth.net Inc., Atlanta, Georgia, USA
Beijing Sanxin Shidai Co. Ltd, Haidian District, Beijing, PRC
Orange Business Services, Blagnac, France
China Mobile Communications Corporation, Xicheng District, Beijing, PRC
CI-Net Limited, Langford Locks, Kidlington, Oxford, United Kingdom
SAKURA Internet Inc., Minami Honmachi, Chuo-ku, Osaka, Japan
Psychz Networks, Walnut, California, USA
GoDaddy.com, Inc., Scottsdale, Arizona, USA

Total items: 8
Items from "undesirable" places: 2

So, only 1/4 are from "bad" places, while a full 3/4 are from places I wouldn't want to ban by country...

[Vince S]

Oh, OK. Thanks 青山 素子. A very reasoned and sensible explanation. Sounds like a classic stuck between a rock and a hard place situation. Which maybe leaves compiling a "best practice" guide for SMF spam mgt as the only practical answer going into the future?

And I should consider myself lucky that our rate of "real" registrations is small so manual approval is no drama and I have in this blunt tool an answer that works for us. I am thankful of the few thousand times this strategy saved me from having to waste a few seconds, or more typically a minute, vetting and rejecting each of the spam registrations that the country bans have taken care of. As you point out this approach has a finite life anyway because IPv6 is bound to bring real ubiquitousness to IP addresses, if some form of "2 jumps ahead" solution isn't forced on spammers by country IP banning gaining traction first.

I would also point out that, even as a transient methodology, country IP filtering would save a lot of pain even if it only had a finite life. So the real q could be how hard it could be to do? But, since 2.0 is stubbornly refusing to go gold, that ain't gunna even get picked up for some time to come, if anybody wanted to!

So if the answer here is "No" for all practical purposes, does that not lead to closing the topic and starting a new one that is about more robust built in SPAM management processes going forward? Which is the real question / problem anyway! I must admit that, right at this minute, I have not done a proper search for whatever may already be "on the slab", and I doubt this topic is going to slip "under the radar" for most of us any time soon!

[青山 素子]

I would also point out that, even as a transient methodology, country IP filtering would save a lot of pain even if it only had a finite life. So the real q could be how hard it could be to do?

It isn't hard to ban large IP blocks. Doing ban-by-country with any kind of granularity? That's going to be difficult. The best you could do easily is geographic region (some part of Asia, New Zealand, Australia / some part of Europe / some part of Africa). It's like using a wrecking ball to pound in a nail. It'll work, but you're going to have a lot of collateral damage.

So if the answer here is "No" for all practical purposes, does that not lead to closing the topic and starting a new one that is about more robust built in SPAM management processes going forward?

Well, the final decision is always that of the developers, but banning by country is a flawed approach to begin with, so it'll likely be given a no as a core feature.

I must admit that, right at this minute, I have not done a proper search for whatever may already be "on the slab", and I doubt this topic is going to slip "under the radar" for most of us any time soon!

As spamming is very much an economic-driven activity, the goal is to make it so that getting on your site is not cost-effective for the people controlling the spamming.

The best approach I've seen so far is a blended method. My current recommended mix:

• An image verification of some sort to filter out simple postbots. The built-in verification system with "medium" difficulty works well for this purpose
• An IP reputation service to filter known threats. The policies of Project Honeypot are sound and IPs are only added if they trip the honeypot, so it's good data. The httpBL modification works well for using this resource. The CloudFlare filtering proxy service is another good option.
• Custom question/answer on registration with topic-specific questions. If you use more complex questions, you'll make registration and/or posting take too long if the prior defenses didn't work. If you slow down registration and posting, you'll make it too expensive to post on your site.

Of course, if you're making a private site, manual registration works great and prevents all spam, except that of actual members.

[butchs]

Don't forget Bad Behavior mod, it has its own anti-spam core and httpBL.

Blocking countries requires signing up to a service or host that maintains an up to date GeoIP database.  Due to copyrights and cost this is not something SMF wants to sign up with.  But many hosts subscribe to GeoIP.

Spam protection is like virus protection on your computer.  It changes frequently and would be a hassle for the SMF developers to maintain.  All they would do is spam protection and that is a waste of their time.  This is why they leave it to the mods.

The protection you mention is available in some sort of mod.  Look around...

But again you should start with a base htaccess protection scheme to reduce the load on the mod.  I recommend GeoIP.   Have you host install it and then do something like this in your htaccess file:

Code Select Expand

GeoIPEnable On
GeoIPDBFile /usr/local/share/GeoIP/GeoIP.dat

SetEnvIf GEOIP_COUNTRY_CODE CN BlockCountry
SetEnvIf GEOIP_COUNTRY_CODE RU BlockCountry
# ... place more countries here

Deny from env=BlockCountry

[Vince S]

Blocking countries requires signing up to a service or host that maintains an up to date GeoIP database.  Due to copyrights and cost this is not something SMF wants to sign up with.  But many hosts subscribe to GeoIP.

Gawd dang Butchs, that's a cool answer for the OP's problem, and anyone else that thinks the same way. And here is a real ABC of a free way to do it: http://www.siteground.com/tutorials/geoip/forums.htm. That is actually the host we use. Cheap, loads of free features you don't get elsewhere and brilliant at doing the job, only issue I have there is synchronising Texas based servers to local Aussie time, which is a small issue indeed. Not that I am meaning to do free adverts, but nothing like a mention of a gold star in a sea of mud as there sure are a lot of dud hosts out there!

I am yet to implement the above referenced answer, but just figured it was worth logging here before I study it. The process looks a little complex and surely could be wrapped up into a tidy mod so people don't need to think about it? That was thinking aloud, I will be more certain when I have tried it.

[Vince S]

I take that back, it is that easy! And what a cool rap for SMF! I will post the relevant info here for posterity:

GeoIP Implementation in SMF

SMF is a popular open-source php forum script. Detailed instructions about its installation and management can be found in the SiteGround's SMF tutorial:

http://www.siteground.com/tutorials/smf/

There is not an official GeoIP modification for SMF. However, you can use the corresponding API for your needs.

In order to filter the forum registration based on the country of the visitors you should complete the following modifications:

    1. Download the geoip.inc (http://www.maxmind.com/download/geoip/api/php/geoip.inc) and the GeoLite Country Binary Format (http://geolite.maxmind.com/download/geoip/database/GeoLiteCountry/GeoIP.dat.gz) files. Extract the last and put them in your SMF root folder.
    2. Open the Sources/Register.php file and add the following code below the if (!defined('SMF')) die('Hacking attempt...'); line:

$website_root_path='/home/user/public_html/smf/';
require($website_root_path . 'geoip.inc');

$ip=$_SERVER['REMOTE_ADDR'];
$gi=geoip_open($website_root_path . 'GeoIP.dat',GEOIP_STANDARD);

$ccode=geoip_country_code_by_addr($gi, $ip);
if ($ccode=="XX" OR $ccode=="YY")
{
die("The forum registration is currently not available for your country.");
}
geoip_close($gi);

In the above code swap XX and YY with the country code of the countries for which the forum registration should be disabled. Also, as the $website_root_path value enter the absolute path to your SMF root directory.

By following the instructions from the GeoIP Tutorial: Implementation you can modify the above code according to your specific needs.

Edit: Adding the linked instructions in the above line as the actual link gets lost on paste, plus added the file links above:

GeoIP Implementation (http://www.siteground.com/tutorials/geoip/implementation.htm)

In this part of the SiteGround's GeoIP tutorial we will give an example how to use the GeoIP API in your web site.

Lets imagine that you have a custom developed php web site and you want to get the visitors' location using the GeoIP API.

First, you should get the geoip.inc (http://www.maxmind.com/download/geoip/api/php/geoip.inc) and the GeoLite Country Binary Format (http://geolite.maxmind.com/download/geoip/database/GeoLiteCountry/GeoIP.dat.gz) files. Extract the archive of the last file on your local computer and upload them in your web site's root folder.

Then open the file where you want to include the GeoIP code. Usually such a file will be called every time the index.php file is loaded. Or you can insert the code directly in the index.php file.

Check the following sample code:

$website_root_path='/home/user/public_html/';

require($website_root_path . 'geoip.inc');

$ip=$_SERVER['REMOTE_ADDR'];
$gi=geoip_open($website_root_path . 'GeoIP.dat',GEOIP_STANDARD);

echo geoip_country_code_by_addr($gi, $ip) . " ";
echo geoip_country_name_by_addr($gi, $ip);

geoip_close($gi);

In the first line we set a new variable. Its value is the absolute path to the web site's root folder.

Then we include the geoip.inc file. Next, the ip variable gets the web site visitor's IP.

After that the GeoIP database is opened.

The visitor's IP address is compared against the IP ranges included in the database through the corresponding functions. As a result the visitor's country code and the country name are visualized. At the end the access to the database file is closed.

You can put the above code excerpt in your web site's code and get the country location of your web site visitors. Then you can use the results for your own purposes.