How to tackle SPAM with SMF?

Started by flapjack, April 18, 2010, 06:40:38 PM

Previous topic - Next topic

flapjack

Note: if you like this tutorial, I appreciate if you use my referral link to install SugarSync,
the best tool for synchornizing data between different computers:
https://www.sugarsync.com/referral?rf=cxw28qzmsn3vj

1) What is "spam"?

Spam is the abuse of electronic messaging systems (including most broadcast media, digital delivery systems) to send unsolicited bulk messages indiscriminately. While the most widely recognized form of spam is e-mail spam, the term is applied to similar abuses in other media: instant messaging spam, Usenet newsgroup spam, Web search engine spam, spam in blogs, wiki spam, online classified ads spam, mobile phone messaging spam, Internet forum spam, junk fax transmissions, social networking spam, television advertising and file sharing network spam.

Spamming remains economically viable because advertisers have no operating costs beyond the management of their mailing lists, and it is difficult to hold senders accountable for their mass mailings. Because the barrier to entry is so low, spammers are numerous, and the volume of unsolicited mail has become very high. The costs, such as lost productivity and fraud, are borne by the public and by Internet service providers, which have been forced to add extra capacity to cope with the deluge. Spamming is universally reviled, and has been the subject of legislation in many jurisdictions.

People who create electronic spam are called spammers.


2) Forum spam

Forum spam is the creating of messages that are advertisements, abusive, or otherwise unwanted on Internet forums. It is generally done by automated spambots. Recent trends shows increasing number of human spammers.

Most forum spam consists of links to external sites, with the dual goals of increasing search engine visibility in highly competitive areas such as weight loss, pharmaceuticals, gambling, pornography, real estate or loans, and generating more traffic for these commercial websites. Some of these links contain code to track the spambot's identity if a sale goes through, when the spammer behind the spambot works on commission.

Spam posts may contain anything from a single link, to dozens of links. Text content is minimal, usually innocuous and unrelated to the forum's topic, or in a very old thread that is revived by the spammer solely for the purpose of spamming links. Some text is included to prevent the post being caught by automated spam filters that prevent posts which consist solely of external links from being submitted. Full banner advertisements have also been reported.

Alternately, the spam links are posted in the user's signature, in which case the spambot will never post. The link sits quietly in the signature field, where it is more likely to be harvested by search engine spiders than discovered by forum administrators and moderators. Some forum engines, including SMF, offers their users possibility to add a link under their avatar. This may also be used, and usually information hidden in this field is very unlikely to be spotted by administration.

Trends are changing very fast. Recently, one of SMF users reported a new way to hide the website in by spammers in user's profile, as in example [1]:
Quote<td><b>Website: </b></td>
   <td><a href="http://www.ro-bot.de" target="_blank"></a></td>


3) Spam prevention
   
Spam prevention and deletions measurably increase the workload of forum administrators and moderators. The amount of time and resources spent keeping a forum spam free contributes significantly to labor cost, and the skill required in the running of a public forum. Marginally profitable or smaller forums may be permanently closed by administrators.   

Most forum software, including SMF, have ability for creating different member groups [2]. Different membergroups may use different sets of permissions, such as ability to post links, include links in their signatures, access to certain parts of the forum, etc. Assigning new members to a membergroup with limited features might have a positive effect of not giving spam users a chance to advertise their services.

Best idea to eliminate spam, is to cut out spam registrations. There is a number of ways to achieve this.
- employ CAPTCHA [3] (visual confirmation) routines on the registration pages to prevent spambots carrying out automated registrations. Simple CAPTCHA systems which display alphanumeric characters have proven vulnerable to optical character recognition software but those that scramble the characters appear to be far more effective.
- request email confirmation for newly created accounts. this way spammer needs to confirm his email address by clicking the confirmation being sent by the forum engine in order to fully activate their account.
- some forums employ manual activation by administration. this wat, forum administrator may check the legitimacy of a new user before accepting or refusing registration.

All SMF approved mods for Spam Prevention may be found here: http://custom.simplemachines.org/mods/index.php?action=search;type=19
Later in this article we will demonstrate which ones may be helpful against different kind of spammers.


3.1) Registration control

Apart from default prevention methods built in SMF, additional measures may be used. Usually they are being provided by 3rd party modifications (mods).
Here is a list of mods that will help you to target spammers during the registration process:
- Anti-Spam Verification Questions for SMF 1.1.7 - http://custom.simplemachines.org/mods/index.php?mod=1516
- Avatar Verification (1.1.10, 2.0 RC1.2, 2.0 RC2, 1.1.11) - http://custom.simplemachines.org/mods/index.php?mod=2182
- reCAPTCHA for SMF  (all versions) - http://custom.simplemachines.org/mods/index.php?mod=1044

Note: this list doesn't include mods that are targetting spammers by an "authoritative voice" method. This method uses 3rd party resources, such as list of known spammer's IP addresses, their email details, and other, which may help to detect the spammer. Those method, and mods that employs it, will be explained in the last chapter of this manual.


3.2) Posting limits

In order to limit the amount of spam, administrator may force users to wait for a short interval between making posts to the forum, thus preventing spambots from flooding the forum with repeated spam messages. In SMF, this may be tackled by setting a time interval between sending two messages from the same IP address [4]. For both 1.x and 2.x this may be found here:
* Administration Center » Posts and Topics » Post Settings » Time required between posts from the same IP
By changing the number here, we change the amount of seconds between accepting by SMF two posts from the same IP address (which usually means the same computer).

Some spammers try to send Personal Messages (PMs) instead of posting rogue links. SMF allows to limit a number of PM sent hourly, you will find appriopriate setting in membergroups settings.

There is no built in method to limit the amount of posts an user may send hourly in SMF. Additionaly, there is no mod for such available at the mods page at this moment. So apart from setting a time interval between sending two posts, there is no other way of limiting an user to flood the forum with their posts.

3.2.1) Limiting posting links

As most of spammer's post are just links to other websites, another approach is not to allow spammers post links. This may be achieved different ways, such as disabling BBC for certain groups, or using mods to limit spammer's capability.
- Anti Spam: prevents posting links by group (1.1.4) - http://custom.simplemachines.org/mods/index.php?action=search;type=19;bool=and;asc;start=0
- Anti-Spam Links (2.0 RC2, 2.0 RC3) - http://custom.simplemachines.org/mods/index.php?mod=2404
- Spam Black List (1.1.4) - http://custom.simplemachines.org/mods/index.php?mod=467
- Stop Spam Links (1.1.9) - http://custom.simplemachines.org/mods/index.php?mod=1855

and another one which target only guests (when you enable guest posting on your forum):
- No Spam by Guests! (1.0.5, 1.0.6, 1.0.7, 1.1 RC1, 1.1 RC2, 1.1 RC3, 1.1.1, 1.1.2, 1.1.3, 1.1.4) - http://custom.simplemachines.org/mods/index.php?mod=369


4) "Authoritative voice" method

Using an external filtering service, to get a verdict if the data is spam or not. There is many 3rd party sites which would allow to check user's data against various databases.

- Akismet Spam Protection (1.1, 1.1.1, 1.1.2, 1.1.3, 1.1.11, 2.0 RC2) - http://custom.simplemachines.org/mods/index.php?mod=544
- httpBL (1.1.10, 1.1.11, 2.0 RC2) - http://custom.simplemachines.org/mods/index.php?mod=2155
- Project Honey Pot MOD (1.1.9, 1.1.10, 2.0 RC1-1, 2.0 RC2, 1.1.11) - http://custom.simplemachines.org/mods/index.php?mod=1849
- Stop Forum Spam (1.1.7, 1.1.8, 1.1.9, 1.1.10, 1.1.11) - http://custom.simplemachines.org/mods/index.php?mod=1519
- Stop Spammer (1.1.11, 2.0 RC2) - http://custom.simplemachines.org/mods/index.php?mod=1547

5) Other solutions

- Bad Behavior mod (2.0 RC3) - http://custom.simplemachines.org/mods/index.php?mod=2502

Bad Behavior complements other link spam solutions by acting as a gatekeeper, preventing spammers from ever delivering their content, and in many cases, from ever reading your site in the first place.  Bad Behavior analyzes the delivery method as well as the software the spammer is using.  In this way, Bad Behavior can stop spam attacks even when nobody has ever seen the particular spam before.
Bad Behavior is designed to work alongside existing spam prevention services to increase their effectiveness and efficiency. Whenever possible, you should run it in combination with a more traditional spam prevention service.

----------------------------------------------------------------------------------------------------------

Resources:
Wikipedia.org
SimpleMachines.org

[1] http://www.simplemachines.org/community/index.php?topic=375738.0
[2] http://docs.simplemachines.org/index.php?topic=167
[3] http://en.wikipedia.org/wiki/CAPTCHA
[4] http://en.wikipedia.org/wiki/Ip_address


----------------------------------------------------------------------------------------------------------

Note: keep in mind this is a Work In Progress. If you found any information that should be included in this manual, please give feedback and I will include it. There is more content to come, so be patient!

----------------------------------------------------------------------------------------------------------
- updated on 01.07.2010

Note: if you like this tutorial, I appreciate if you use my referral link to install SugarSync,
the best tool for synchornizing data between different computers:
https://www.sugarsync.com/referral?rf=cxw28qzmsn3vj

Kill Em All



My Site: KEAGaming.com

Manual Installation of Mods
Prevent Spam and Forum Attacks
Please do not PM or email me for support unless offered, help should be publicly displayed to others.

Adish - (F.L.A.M.E.R)

If there was a 'Like' button, I would have been hitting it continuously. ;)

Kill Em All

flapjack, would you like this moved the the Tips/Tricks area?


My Site: KEAGaming.com

Manual Installation of Mods
Prevent Spam and Forum Attacks
Please do not PM or email me for support unless offered, help should be publicly displayed to others.

flapjack

why not, as long as people will be still able to post in it, I may not cover all aspects of spamming myself :)

Kill Em All



My Site: KEAGaming.com

Manual Installation of Mods
Prevent Spam and Forum Attacks
Please do not PM or email me for support unless offered, help should be publicly displayed to others.

butchs

Nice article but please note that Bad Behavior mod (2.0 RC2 & RC3) does not use a third party site.  It uses php code developed by the Bad Behavior core author after many years of research to inspect the visitors internet information and rejects them before they can get a chance to post on your forum.  The package has no external links, has had extensive testing and is extremely fast.
8)

I guess if you had to call it something "gatekeeper" will fit it best.
:)
I have been truly inspired by the SUGGESTIONS as I sit on my throne and contemplate the wisdom imposed upon me.

flapjack

you are correct. care to write (snatch from author's page) couple lines of how it actually works, and why it's one of the best solutions available? ;)

butchs

If I had to pick a couple of lines:
Bad Behavior complements other link spam solutions by acting as a gatekeeper, preventing spammers from ever delivering their content, and in many cases, from ever reading your site in the first place.  Bad Behavior analyzes the delivery method as well as the software the spammer is using.  In this way, Bad Behavior can stop spam attacks even when nobody has ever seen the particular spam before.

Bad Behavior is designed to work alongside existing spam prevention services to increase their effectiveness and efficiency. Whenever possible, you should run it in combination with a more traditional spam prevention service.

I am too tired to write my own stuff this morning.   O:)
I have been truly inspired by the SUGGESTIONS as I sit on my throne and contemplate the wisdom imposed upon me.

societyofrobots

Hi, I was 'one of SMF users reported a new way to hide the website in by spammers in user's profile'.

I have reCAPTCHA, Anti-Bot Registration Puzzles, and Stop Spammer all installed, yet I still get from 5 to 10 spam registrations per day. Probably humans hired in 3rd world countries doing it. 50 cents/hour is good pay in many of these countries.

Post Spam:
These days, most spammers don't post in my forum. If they do, posts get flagged pretty quick by users. That said, some spammers copy/paste text from other parts of the forum to give the impression its a legit post. Often times its out of context or seems too much out of place, but most users never notice and therefore don't flag. It doesn't appear to be automated, but whenever it is automated, we are in deep s&$t.

Profile Website Spam:
For the last year there was a lot of spam website links posted in the website line of spam bot user profiles. This is fairly easy to delete.
Search your member database for users with zero posts and a website of *.*
99% of those are spam bot accounts, and will take seconds to delete.

Signature Spam:
But most recently, spammers have found a way around my above solution. They instead post spam links into the signature section. SMF has no search option for signatures, forcing you to open up profiles manually. I get ~30 registrations per day (I have a large forum), so its a pain in the arse to do this. I could of course install a 'must post X times before signature allowed' mod, but that'll either encourage spammers to post X times, or make it impossible to identify it as a spam account (no spam links). If only SMF had a 'search signature' mod . . .


That said, for all my spam defenses touted as perfect a year or two ago, spammers have found a way around them. reCAPTCHA no longer stops them, for example. I can even see a spammer filling the Stop Spammer database with so much false information it becomes useless. Point being, we have to continually change and diversify our defenses every year, so I doubt there will ever be a forgive and forget solution for a webmaster.

ps - I'm thinking of installing the MOD Honeypot, but their site is down for heavy maintenance right now . . . but I doubt it'll help against 3rd world human spam registrations.

flapjack

as for signatures, there's already a mod for 2.0. you can also disable signatures for a given group using a simple hack injected in template files, and setting profile as off-limits for people that are not logged in and robots

Arantor

You can disable signatures for any group with permissions. The only template hack you need is if you're planning on hiding them from guests.

societyofrobots

I don't want to disable signatures. What will happen is the spam accounts will still be created, but since they won't do anything spam like, I wouldn't be able to identify and thereby delete them. 30 spam accounts a day, for 1 year, is almost 11k fake members.

I'd rather a search signatures option.

butchs

#13
I was getting several a day.  Plus a bunch of bandwidth wasting hits on my site.

I have not seen a fake spam account for several moths and now my bandwidth is under control.

Besides blocking all Countries that are not in my target area I have the following spam prevention installed:


  • old version of crawlprotect --  custom edit by me
  • Bad Behavior  -- strict mode
  • httpBL "Honey pot" --  45/10/30
  • Stop Spammer
  • Avatar Verification --  over 100 images

Nothing, nadda, zip!
;D
I have been truly inspired by the SUGGESTIONS as I sit on my throne and contemplate the wisdom imposed upon me.

Arantor

How many regular members do you get signing up though, out of interest?

* Arantor is curious to know the site to see if it considers me a spammer...

butchs

#15
My site is not a heavy traffic site with 75-180 guests/ day.  I see 2-3 new members a month.   Yes, people have logged in since my anti-spam measures.
:P

I do not block the UK.  Go ahead and see if you are a spammer:  [edit:  test was completed the link was removed]


I have been truly inspired by the SUGGESTIONS as I sit on my throne and contemplate the wisdom imposed upon me.

Arantor

Well, it let me onto the registration page, 3 of the avatar based images were broken on first load.

And it let me on. (test_signup, if you want to delete the account)

I have to admit I was a little surprised at the captcha not having any noise in the image.

All in all, didn't feel much more cumbersome than normal sign up and if it keeps the spammers out, go for it.

butchs

#17
Ok I will delete "test_signup".

The broken image issue is from the "avatar_verification" mod.  In the past I would fix the mod but in all honesty whenever I do that a revision comes out right after I do it.  Besides I do not know much about images in SMF and would have to spend a bunch of time doing it.  So I set it to unlimited login attempts just in case the image to pick was messed up.  I guess one of these days the mod author will find the problem.
8)

The spammers were reduced with each change/ mod but it was not until I finished Bad Behavior that my spammer count dropped to zero.  The spammer attack was why I made the mod in the first place.  They made me mad!  O:)
I have been truly inspired by the SUGGESTIONS as I sit on my throne and contemplate the wisdom imposed upon me.

societyofrobots

Anyone know when the httpBL "Honey pot" will be out of maintenance? Its been like this for at least a week:
http://www.projecthoneypot.org/

I also just added image verification.

But I'm very convinced its humans doing the registration and not bots . . . so not sure how much either will help . . . probably just slow them down a few seconds.

flapjack

add 2-3 niche specific questions during the registration process

Advertisement: