Spam Prevention using SurgeMail Features

The quick answer

In brief, use the config checker (web admin tool) to turn on spam handling features, and set g_friends_spam_score "10" and
G_FRIENDS_DEFAULT_MODE "smite"


The slow answer

NOTE: Turning on the recommended settings WILL NOT block email from servers without SPF. This seems to be the number one confusion. Many customers are reluctant to turn on the settings recommeded as they fear their server will then bounce all email from badly configured mail systems, this is NOT the case, please try the recommended settings as a starting point!!

How it works

Spam prevention has gone through many changes over the last few years, initially people tried to filter spam based on the 'content' although this worked well initially it soon started to fail as spammers adjusted their spam. The focus of successful spam prevention is based on a multi pronged attack, where the 'source' of the message is verified in various ways, and the content of the message is checked, and then finally the 'friends' system catches and automatically deals with any messages that still get through while it also automatically white lists known associates.

For more information see the detailed technical description of how SurgeMail stops spam when correctly configured.

How to enable SPAM handling

  1. Upgrade to the latest stable release
  2. Press the 'config checker' button in the web admin interface and turn on the settings it suggests
  3. Either set G_FRIENDS_DEFAULT_MODE "smite" or "silent" or "list"
  4. Tell all users to individually turn on/set/adjust their friends options in the user self admin settings.
  5. For general purpose email systems, set the default to block at g_friends_spam_score '10'
  6. Advise your users how to turn on/off user configurable options (this is done in the user self admin pages or via options in surgeweb) or by looking up the user in the web admin accounts interface.

How users can report/train spam

  1. IMap users can drop spam messages into the 'Spam' folder
  2. SurgeWeb users can click on 'Is Spam'
  3. POP users must forward spam to 'isspam@yourdomain.com' (this is the least accurate method as the server must then untangle the sent message, options 1,2 above are better)

List of recommended settings with notes (These are set in the config checker, you don't need to read this section!)

g_orbs_list name="YOURID.zen.spamhaus.org" action="stamp" stamp="zen.spamhaus.org , ip= "

This setting tells surgemail to check the IP address with an RBL service (in this case spamhaus) This setting improves the spam scoring features. Please check http://www.netwinsite.com/surgemail/help/rbl.htm for more infomation on RBL's.

To get 'YOURID' you will need to register on the spamhaus website https://www.spamhaustech.com/. 

g_verify_mx_skip "10.0.0.2"

This setting should list your other MX hosts (low priority mx host). However our general recommendation is to REMOVE low priority mx hosts entirely as they serve no useful purpose and will tend to allow spam through your system.

g_spam_allow "10.2.192.98-117"

This setting lets you list the ip addresses of known trusted hosts.

g_spam_userconfig "TRUE"

This setting lets the users change their own spam settings.

g_spam_internal "true" ("Enable ASpam" setting in user interface)

This turns on the aspam scoring system.

g_spam_catcher "fred@your.domain"

This setting is used to train the aspam filter with spam that comes to special email addresses on your system, place these email addresses on your web pages so that spammers will accidentally train your system for you :-)

g_url_enable "true"

This adds some url scoring using a netwin provided database that is updated every few hours, you should also use SURBL as well.

g_vanish_bad_bounces "TRUE"

This gets rid of bounces that didn't originate from your server.

g_verify_smtp "TRUE" (Probably not needed when using spf)

This setting checks if the connecting smtp server is open on port 25. The spam scoring is adjusted if the test fails.

g_spf_mode "strict" (absolutely essential!!!)
g_spam_block "true" (absolutely essential!!!)
g_spam_allow_known "true" (this allows more spam through but cuts down on rejections)
g_spam_grey_dflt "false"
g_spam_grey_dflt_bad "true"
g_spam_grey_bounce "10" (explained below)

These settings turn on SPF see https://netwinsite.com/spf.htm. In addition the g_spam_block setting makes it actually block all the spam that fails spf tests. However to reduce impact the grey settings mean that failures are grey listed, and only fully blocked if grey listing fails, or if too many messages arrive within a short time period (1 message)

g_surbl name="multi.surbl.org" stamp="sc.surbl.org,ws.surbl.org,phishing,ob.surbl.org,ab.surbl.org,jp"

This setting is critical to spam detection, the surbl database is used to detect urls that spammers are trying to promote.

g_spam_grey_bounce "10"

This setting lets surgemail bounce a message that looks 'spammy' if it failed some spf tests but got past the grey listing mechanism. This cuts down on spam but does often bounce real emails (it uses an allow bounce so the sender can fix it)

It's probably better for individual users to use their friends settings instead.

This used to default to 3 and still is on any version prior to "SurgeMail Version 3.8". The new default value is 10 which lets more spam through, but reduces accidental bounces. We now recommend a value of '10' unless you are happy with some real legit email bouncing.


g_sf_list "true"
g_sf_rules "true"

These two setting enable a different module for calculating the score based on features, this should result in more consistent results.

Settings you SHOULD REMOVE.

We often find problems occur when non standard settings or obsolete settings have been turned on, here are the main culprits you should remove. These will break the normal spam prevention occuring and or cause massive complaints from users due to bounces.

(remove) g_spf_default_noblock "TRUE"
(remove) g_spam_grey "TRUE"
(remove) g_spam_grey_dflt "TRUE" (optional)
(remove) g_spam_allow_disable "TRUE"

(remove) g_badfrom_check "TRUE"
(remove) g_badfrom_stamp "TRUE"

 

Frequently Asked Questions FAQ

Can I avoid backscatter from friends?

Yes use this setting

G_FRIENDS_CHECK_SPF "true"

What are the recommended best practise techniques to avoid spam on my server?

See the list of settings above, primarily you want SURBL, RBL's and SPF (in strict mode with the g_spam_block turned on). Also avoid using front end filter systems as these will prevent the best spam features in surgemail working. And suggest users turn on 'friends' with a friends bounce level of about 4.

Doesn't SPF rely on the senders creating spf records ?

No, in strict mode surgemail makes up an spf record for all incoming domains so it works for everyone. When the made up spf record fails (which is rare) surgemail then provides other checks and mechanisms so real email can still get through.

Is there something else I should be doing to prevent spam, why do I get so much when other people get none?

Although these mechanisms can stop almost all spam, there is another way to get rid of spam, and if you use it, then you can adjust the filters to be very 'forgiving' so that real messages are never caught by them. So here's the trick, the BEST way to avoid spam, is to change your email address! and keep your new email address private:

What are the likely side effects and implications of using these measures?

You will bounce some real mail messages and because some people don't read the bounce messages they will actually fail to respond correctly to get past the automated spam prevention. The above settings only require respones from about 1-2% of people so most mail gets through without any trouble, but a small percentage will be bounced and if the user sending doesn't respond then the message will fail to be delivered. This results in a loss of about 0.1% of messages, much lower than letting humans do the filtering, but still not perfect.

How do I measure how effective these techniques are? (my manager needs a report to justify costs)

In the advanced status section in surgemail there is a 'spam' section, this has figures on the various filter hit rates, it's a little hard to interpret but it gives a fairly good idea of how much spam has been blocked.

How are false positives handled? Each email is important to me, and I must avoid false positives at all costs, how can I monitor email identified as spam until I am confident that the system has no / minimal false positives?

With SPF and friends false positives result in some form of bounce, the user sending the message must then respond to the bounce to get their original message delivered. (With SPF failures they must resend, with Friends they need not). You will only loose messages when the person sending to you does not read the bounces. From the user web interface you can search through all the bounces manually and release messages pending confirmation via friends, and fix SPF failures.

How can spam that was not caught be submitted (by users)? and how do users/admin get feedback that their submissions are actually doing something?

You or any user can send messages to isspam@your.domain or notspam@your.domain, this will improve the scoring in future. From the managers web admin pages for spam you can also paste in a message and get it analyzed, or trained. This process should not be over emphasized, it is good for fine tuning the filters slightly but it is not at all critical that you submit every failed message or every false positive. The messages can be sent as attachments or redirects, it doesn't matter much which is done as the system is forgiving. If a messages is sent to the wrong training address, just resend it to the other address to nullify the training.

How should I as a user configure my spam controls on my email. There seem to be several ways of configuring filters + friends + spam/spf etc to work together. Why should / whould I not use a particular combination. Are there particular things that I probably should not configure?

This is very important, if you get 'lots' of spam and want to get none.

If you get a small amount of spam but want to get rid of 'most' of it, without much risk of ever bouncing a real message:

Are there any significant performance effects? (on 100 / 5000 / 100000 user system) Both in increased load that these measures put on system resources (disk / cpu / open channels / resposiveness etc) and reduced load by not having to deal with spam. How can I measure these effects?

Not really, the spam system in SurgeMail is very efficient and the SPF features and vanish bad bounce settings do reduce real load on heavily spammed servers, so the spam prevention tends to result in a slight performance improvement, and reduced network bandwidth usage.

We do STRONGLY recommend the use of the AVAST virus scanner product as it is enormously more efficient than some of the free unix command line scanning utilities that you can use with SurgeMail (mainly because it does not get activated for each scan as it's part of the server)

Also using external spam checking systems (which you can do if you really want to) is also strongly discouraged, these generally won't increase your filtering accuracy but will badly affect performance.

I want to counter some rules in ASPAM - for example NakedCR.

ASPAM's filter rules are stored in aspam_mfilter.txt, you cannot edit this file as it is updated regulary so any changes you make will be overwritten. You need to edit the file local.rul where you can add your own rules.

     if (isin("X-NakedCr","body")) then
          call spamdetect(0.1,"NakedCR")
     end if

In general, look through aspam_mfilter.txt find the rule and then write the same rule in local.rul but with a negative score to cancel the scoring in aspam_mfiler.rul. The string/reason in local.rul must be _exactly_ the same as the string in aspam_mfilter.rul for the rule to overide the first one.

New settings that we hope to make recommended in future.

These settings are typically new settings only available in the latest beta builds, and may be unstable, but once stable we expect to become recommended settings so you might want to experiment with these.

g_domainkeys_check "true"

Checks incoming email for for signatures and if found verify, this will help avoid bounces from domains that use domainkeys instead of spf.

g_domainkeys_sign "true" (see note below)

Use the web admin to create your finger print and then save in your dns first.

g_spam_share "true"

Use and contribute to shared whitelist database via netwinsite.com to avoid spf bounces for well known sites that are not spammers but fail spf tests.

Integrating external SPAM filters

You can in addition to the normal surgemail spam features run an external spam filter which is a command line program that examines the message then returns non zero numbers if it thinks the message is spam. This can then contribute to the SurgeMail score for that message.

We recommend this external filter, it's a reasonable price and seems to work reasonably well: http://www.armresearch.com/message-sniffer/ we are keen to hear feedback from anyone using filters like this.

These settings require SurgeMail 3.8-20 or later, email support@netwinsite.com if you need this build to try this new feature.

Surgemail.ini setting:

Install Message Sniffer, for Windows, choosing "Other" for the mail server type.

  
  G_SPAM_CMD "c:\snf-installation-directory\SNFClient.exe $FILE$"

Create a text file named sf_mfilter_local.txt, place it in the surgemail root folder, e.g. c:\surgemail typically or /usr/local/surgemail.  Inside the file, place this code:


	if (isin("X-SpamCmd","Is Spam")) then   
		call feature_manual(0.99,"ArmResearchSpam")   
	end if   
	if (isin("X-SpamCmd","Not Spam")) then   
		call feature_manual(0.01,"ArmResearchOK")
	end if

Issue the commands: tellmail reload,  tellmail sf_train


Forgeries of the form From=To

There has recently been an increase in spam where the From and To headers are set to be the same as the user. To block this type of spam ensure you have done the following steps

Optional settings to stop more spam...

Some of these are a bit 'strict' so use with caution depending on your tastes...

 
A new training module can be used which uses a binary tree to learn what features are signficant, many people find this gives better results:
    g_sf_binary "true"
After setting that run a manual train just to make sure the new rules are in place.  tellmail sf_train

You may wish to try this setting, it will black list any ip address that is the source of a isspam training event for an hour or so, this is most useful with your catcher addresses as it means any spammer who sends to your spam catcher will find themselves blocked from sending any email to your server for an hour or so. You may need a whitelist for a few large sites to avoid issues with deleted users causing a large mail server to get blocked. Hence the g_black_white setting given as an example...

 

g_black_isspam "true"

g_black_white "1.2.3.*,*ebay.com"

 

 

You can tell surgemail to try and guess the language of each message, you can then set for any account in your spam settings the langauge you expect, if you get messages that are not in your language (e.g. english) then the message will be assumed to be spam until proven otherwise (So it goes to your friends pending folder), this will reduce spam significantly for those of us who really only speak one language :-). Be warned it does not always guess correctly, and is more likely to be wrong with non english messages.

g_spam_lang "true"