Tidal wave of spambots attacks SMF 1.1.x - How to protect your forum

[Col]

Here's what I did:     

• I created an approval board for Newbies.
• Newbies are the post-count group with less than '1' post.
• The approval board description explains the reason for the board, and encourages them to write a few words about why they joined. I've coloured the description text red to make sure it is noticed.
• This board is at the top of the forum, and is accessable only by Newbies, Moderators and Admins.
• I added a modification so that that members can view only topics they created, and enabled this for the Approval board: http://custom.simplemachines.org/mods/index.php?mod=687
• Moderators and admins can view any thread on the Approval board.
• Post count is  not increased by posting to the Approval board.
• Moderators can simply move a post/thread from the Approval board to a standard forum board, and this will increase the post count of the genuine members to '1'. This means they are no longer a 'Newbie', and they now have posting rights to all the main boards. Since they are no longer a 'Newbie', they no longer have access to the Approval board.

The reason for the Any Thread mod is to protect Newbies from viewing any spam. Some of the spam we received was pretty extreme in content, but this mod means that spammers and genuine members can only view their own threads. Of course, members with a post count of '1' or more, do not even know that the approval board exists (it is not accessable to guests either).

I also added a sticky within Approval board to explain why it is there. I might be helpful to include a link directly to the 'New Topic' form. Stickies, with the Any Topic mod, are viewable by all. The sticky is locked to stop spam from being posted there.

All this worked well. Since all the forum's main boards increase the post count of members, moving a thread to a normal forum board increases the post count of the new member. This means that it is usually unnecessary for their post count to be manually increased by an admin.

Although this resolved the problem, I became tired of all the bogus registrations. Increasing the CAPTCHA codes to maximum did make a big difference, but did not stop the spam entirely. At the same time, at the maximum setting, I found the CAPTCHA codes difficult to read, and so I assumed so would most of my potential members. I've since upgraded the registration system with reCAPTCHA, and couldn't be more pleased. It stops the bots in their tracks, and is easier for humans to read at the same time. I'm keeping the approval board for now though, just in case. Some of the attempted spam was highly offensive, so a manual check is still desireable.

Edit: I also denied the ability of Newbies to send PMs and view the profiles of other members, just to be on the safe side.

[Deprecated]

That's a clever means of dealing with the spambots, and thank you for explaining it.

As I've already said, I consider the reCAPTCHA mod to be unbeatable, at least with today's spambots and today's reCAPTCHA. I'll be surprised if a single bot gets past it on a single forum. I think for the present that is impossible, unless they are "humbots" or humans paid to do bot work. There are some of those where people in countries with low standards of living and low wages are paid to do nothing but type in the CAPTCHA letters (the bot does the rest automatically). There is little you can do in that case.

My new Anti-Spam Verification Questions mod should be equally secure, but for different reasons. In fact in the case of humbots it might even have an advantage if the humbots don't speak the forum's language. It's one thing to learn 26 English letters and quite another to learn to communicate in English, for example (if the forum's language is English). I think the Anti-Spam Verification Questions might even work fine with CAPTCHA set on medium, and that's SMF's native CAPTCHA system. It might even work on low, relying on the questions rather than the CAPTCHA. That might be good in some forums where for example their members are senior citizens with poor eyesight. I know I've had troubles with getting some CAPTCHAs right myself.

Note that the Anti-Spam Verification Questions can be in any language, so it is not restricted only to English forums.

[forumite]

A big thanks for the mod Deprecated. I haven't been hit (yet), but I plan to install your mod ASAP.

[MrPhil]

I think for the present that is impossible, unless they are "humbots" or humans paid to do bot work. There are some of those where people in countries with low standards of living and low wages are paid to do nothing but type in the CAPTCHA letters (the bot does the rest automatically). There is little you can do in that case.

My new Anti-Spam Verification Questions mod should be equally secure, but for different reasons. In fact in the case of humbots it might even have an advantage if the humbots don't speak the forum's language. It's one thing to learn 26 English letters and quite another to learn to communicate in English, for example (if the forum's language is English).

Well, humbots don't necessarily even have to be low paid people in third-world countries. I've heard of CAPTCHA images being immediately transferred to porn sites, where horny guys trying to get in will break the images quickly. If they assume it's a real CAPTCHA, they may even give the correct interpretation of it. The only defense against this would be to have a strict (and very short) time limit for the user to enter the answer.

That might be good in some forums where for example their members are senior citizens with poor eyesight. I know I've had troubles with getting some CAPTCHAs right myself.

Of course, it's an arms race of more sophisticated CAPTCHAs against more sophisticated crackers (machine vision/A.I.) until you get to the point where too many of your intended audience can't decipher the images. And yes, I too sometimes have trouble with CAPTCHA images (is 50 old?).

Note that the Anti-Spam Verification Questions can be in any language, so it is not restricted only to English forums.

You have to be careful with natural language questions and answers. First, you might unwittingly introduce cultural biases (e.g., "What are colors of the Flag?" The U.S. flag? Another country's flag?).  Math questions spelled out COBOL style might not slow down the bots for long -- there are only so many ways to phrase an operation (minus, subtract, take away from, reduce by, plus, add to, augment). Using tricky phrasing and obscure words will eventually start to trip up your target audience, and you have to be able to generate so many permutations of sentences that bots won't be able to store each case (sentence template). Plus, user responses may have many subtle differences (spelling, capitalization, etc.) that you have to allow for. I remember being given a PC version of "Jeopardy!" a couple of decades ago -- I only played it a few times, because it was so frustrating to give the correct answer, but it didn't exactly match the canned answer!

It may well get to the point that all new users have to be on "probation" until they've shown that they are behaving themselves, and this may include a "why I want to join" essay.

For pure bots (non-human), we may be able to trip them up for a while, such as by randomly introducing hidden questions (constantly changing) that shouldn't be seen (and answered) by a human, but who knows how long that will succeed!

[catfished]

I don't want to sound smug but I really don't understand why so many of you are having so many problems. I am the starter of the original thread about this: http://www.simplemachines.org/community/index.php?topic=273648.msg1792741#msg1792741 and I simply upped the image verification to high and added an age limit. All spam registrations immediately ceased on all my SMF forums (all 1.1.7) after doing these two simple things.

Now 11 days have passed without a single successful spam registration so it should work for those running 1.1.7.

[Burke ♞ Knight]

I don't want to sound smug but I really don't understand why so many of you are having so many problems. I am the starter of the original thread about this: http://www.simplemachines.org/community/index.php?topic=273648.msg1792741#msg1792741 and I simply upped the image verification to high and added an age limit. All spam registrations immediately ceased on all my SMF forums (all 1.1.7) after doing these two simple things.

Now 11 days have passed without a single successful spam registration so it should work for those running 1.1.7.

Do not forget the following:

1. Some members are not as experienced with settings.
2. Some members are actually being attacked by other spambots that may not be attacking other people, including you.
3. Some people may be running other mods and such that replace the normal SMF verification system.
4. Some people may not be able to set the verification to high, as that can cause problems for their soon to be valid members who may not be able to see to well.

We all have reasons why some things work, or do not work for us.

[Deprecated]

Of course, it's an arms race of more sophisticated CAPTCHAs against more sophisticated crackers (machine vision/A.I.) until you get to the point where too many of your intended audience can't decipher the images. And yes, I too sometimes have trouble with CAPTCHA images (is 50 old?).

Old is relative. It is what happens if you don't have the decency to die young.

You have to be careful with natural language questions and answers. First, you might unwittingly introduce cultural biases (e.g., "What are colors of the Flag?" The U.S. flag? Another country's flag?).  Math questions spelled out COBOL style might not slow down the bots for long -- there are only so many ways to phrase an operation (minus, subtract, take away from, reduce by, plus, add to, augment). Using tricky phrasing and obscure words will eventually start to trip up your target audience, and you have to be able to generate so many permutations of sentences that bots won't be able to store each case (sentence template). Plus, user responses may have many subtle differences (spelling, capitalization, etc.) that you have to allow for. I remember being given a PC version of "Jeopardy!" a couple of decades ago -- I only played it a few times, because it was so frustrating to give the correct answer, but it didn't exactly match the canned answer!

Well that is the beauty of my new mod. Every forum owner gets to decide for themselves what the questions (and answers) are. But your points are all good ones, and the forum owners should take your advice.

I'm providing the mod. There are NO questions in the mod. Each forum operator must generate their own questions, and those questions must make sense in their own culture.

[catfished]

Do not forget the following:

1. Some members are not as experienced with settings.
2. Some members are actually being attacked by other spambots that may not be attacking other people, including you.
3. Some people may be running other mods and such that replace the normal SMF verification system.
4. Some people may not be able to set the verification to high, as that can cause problems for their soon to be valid members who may not be able to see to well.

We all have reasons why some things work, or do not work for us.

All very good points, thanks for waking me up. It's never as simple as I made it sound, I should know better.

[societyofrobots]

I simply upped the image verification to high and added an age limit. All spam registrations immediately ceased on all my SMF forums (all 1.1.7) after doing these two simple things.

Now 11 days have passed without a single successful spam registration so it should work for those running 1.1.7.

I don't think it will take long for the spammers to figure that out and mod the bots . . . I only see it as a temp fix. I'm sure the spammer is even reading this thread and making the changes as we speak.


Deprecated, what would stop the spammer from programming your mod into the bot? It wouldn't take the spammer more than 10 minutes figuring out all the questions/answers and adding it to the bot, no? It'll slow him down, but definitely not stop him in the long term.


There needs to be a much greater variety in anti-spam mods out there. A spammer can defeat one or two mods, but the effort to create a bot that can defeat ever-improving ten or more mods might actually be cost prohibitive . . .

[Xavi-Nena]

1st - i dont think it was the version of my smf because i was using 1.1.7

2nd - i had almost as col suggested where all members had to post welcome introduction message before being able to access other boards.

Let me say my spam was not im message topics on the board but on my template files. Im not expert but i thought they had to have access to admin panel for that? I did not approve anyone that did not send welcome email to me first why they wanted to join and then no one who registered was able to view the boards until they make introduction post .

Again my spam was not as actual post topics but w/in the files of my pages.

Im not sure if that helps any but I figured I would mention it.

[metallica48423]

I simply upped the image verification to high and added an age limit. All spam registrations immediately ceased on all my SMF forums (all 1.1.7) after doing these two simple things.

Now 11 days have passed without a single successful spam registration so it should work for those running 1.1.7.

I don't think it will take long for the spammers to figure that out and mod the bots . . . I only see it as a temp fix. I'm sure the spammer is even reading this thread and making the changes as we speak.


Deprecated, what would stop the spammer from programming your mod into the bot? It wouldn't take the spammer more than 10 minutes figuring out all the questions/answers and adding it to the bot, no? It'll slow him down, but definitely not stop him in the long term.


There needs to be a much greater variety in anti-spam mods out there. A spammer can defeat one or two mods, but the effort to create a bot that can defeat ever-improving ten or more mods might actually be cost prohibitive . . .

This is true -- it could be made to have the questions and answers stored... but can it possibly have EVERY possible verification question an admin might use preprogrammed into it?  realistically, its an endless pool of possibilities, and thats where the strength in this method lies.

[societyofrobots]

realistically, its an endless pool of possibilities, and thats where the strength in this method lies.

*If* I was a spammer, I'd write a script that would ask me (the human) to answer any new questions it finds, while automatically answering ones it already knows the answers to.

So if I were to spam 100 forums, and all 100 had 5 questions, and each question took 5 seconds of thought . . . it'll take me 41 minutes to answer all of them. I don't expect many forum owners to have that many questions, or even change them that often . . . The time it takes to change your 5 questions would be much greater than the time it'd take to answer the 5 new ones, given the script automates most of it.

And since its easy questions (yes or no, or some number), I'm sure about 10% can be easily guessed by brute force.

I think the best defense is not to have a monoculture of defenses.

[swyl]

18 pages wow... Just gonna add my two penneth 

I recently changed registration to include email activation but a latest sign up complained through another member that she couldnt get the email acti. So I tried it myself and it seems she is right... I indeed got no acti email. But the bots are still getting in.

I'm not being funny but how does that work ? Real people arent getting the acti email but the bots are wow.

[Deprecated]

Deprecated, what would stop the spammer from programming your mod into the bot? It wouldn't take the spammer more than 10 minutes figuring out all the questions/answers and adding it to the bot, no? It'll slow him down, but definitely not stop him in the long term.

Although Justin answered this, let me add my own version.

How many thousands of SMF forums are there? Let's just say 10,000 although I believe that is far too low a number. Let's say that most of them pick different questions, so perhaps there are 40,000 different question/answer sets. How could it be possible for the botmasters to build up a list of the correct answers for all those questions? And they change every day too! If you found that spammers were blowing past your questions, just change them!

You say what if there is a human in the loop? Well that's the problem, but it's not our problem, it's the bot master's problem. The only reason the spammers turned into a tidal wave is because the process was fully automated. I'm pretty sure they would be running a multi-threaded client, meaning that while they are spamming your forum they are also joining up with or spamming 10, 20 or 50 other forums at the same time. That's what creates the tidal wave.

But if a bot master has to sit there and answer questions all day he can't possibly bother 50 forums at the same time. Instead of running through our forums he'll be crawling through them, and dragging a ball and chain on one ankle. It is not practical for the bot master to spam our forums unless he can do it with no more effort than turning on the robot. If he has to mind the bot all day while it's doing its dirty work he is going to go nuts and finally he'll put in a setting that tells the bots to ignore SMF forums because they can't be spammed or they aren't worth the trouble.

That is why I think it will work. There's too many different possible questions, they'll never succeed in making a list of them all, the list will change on a daily basis, and the spambots are not practical if they require human operators.

[societyofrobots]

But if a bot master has to sit there and answer questions all day he can't possibly bother 50 forums at the same time. Instead of running through our forums he'll be crawling through them, and dragging a ball and chain on one ankle. It is not practical for the bot master to spam our forums unless he can do it with no more effort than turning on the robot. If he has to mind the bot all day while it's doing its dirty work he is going to go nuts and finally he'll put in a setting that tells the bots to ignore SMF forums because they can't be spammed or they aren't worth the trouble.

I completely agree, its about making it less worth their effort by giving them more work . . . however I'm unconvinced they won't adapt or that we've finally solved the worlds spam problem

I just see this as a one up in the arms race for now . . . but I think they'll one up us in a year or so too . . .

[swyl]

Does anyone know about this

Real people arent getting the acti email but the bots are wow.

[SlammedDime]

It's very easy to 'pipe' emails through a script, be it php, cgi, perl, whatever, and have that script parse out the url, visit it, thus activating the account.  Perhaps adding visual verification to the verification of email would be a good idea.

[societyofrobots]

Perhaps adding visual verification to the verification of email would be a good idea.

A great idea, and it'd help too . . . but a lot of email services block images by default for security reasons (like gmail). Bad enough that hotmail and yahoo considers all SMF email spam . . .

Visitors to my site are considered above average intelligence (robot building forum) . . . but for others the extra image step might be too much . . .

[mashby]

Upping the image verification has nothing to do with email services...it's an on-screen thing. If your above average intelligence visitors cannot read the high verification image, maybe you should consider this mod:
http://custom.simplemachines.org/mods/index.php?mod=1516

Ask them something like, what is 2+2.

[societyofrobots]

mashby you misinterpreted me entirely . . . ok i'll reword . . .

Google and other email services remove images from emails as a security precaution. For a non-techie to get a confirmation email with an image contained in it, and requires that non-techie to see that image but the email service disables the image by default, this could be a problem.

My site visitors are more intelligent on average and most won't have that problem - they can just enable the image and understand why too. However if your forum is about gardening, or train collecting, or socializing, by default those visitors would be non-techies.

SlammedDime had a good suggestion, but I can see user problems with it for non-techie forums.