- Potentially problematic feedback between greylisting and spam filtering
- How to use obscurity to improve security (a tiny bit)
- Making your servers a bit more "green" with smart system adminstration, part 3: the right tool for the right job
- Minimizing Apache httpd downtime with logrotate
- Sysadmin war stories: (disk)size does matter
Fighting spam with greylisting
May 13, 2009,
Spam is one of the number one oline irritations for people. There are some effective ways to get rid of a lot of spam, at least until spammers catch up again. One of the most used methods right now is greylisting, which yields really good results, but at a price.
As a hosting company we care about spam fighting. The amounts of spam travelling on the Internet are gigantic. A very large amount of spam is filtered out in mail servers and mail clients, but despite that the spamming business is still profitable enough, otherwise spammers would have stopped a long time ago.
Spam is irritating for hosting companies like us, because it unnecesarily takes up bandwidth and CPU power, and it is irritating for our customers, because they pay for the bandwidth. It is irritating for individual users as well: cleaning spam from a mail account costs a lot of time and there is always the risk that mail is accidentily deleted that should not have been deleted when a user is in a "delete spam" frenzy.
One of the methods we use to fight spam is greylisting. Greylisting does not actively block mail, like blacklisting, but uses SMTP error codes to send a temporary delivery error. A mail server that talks SMTP properly will know it should retry later. Most spam software does not correctly talk SMTP, but try to blast out as much mail as possible in a short amount of time before the host they are sending from is put into blacklists.
Greylisting implementations (such as postgrey for Postfix) keep a database in which is recorded which machine or subnet tried to send mail. For a certain period of time (which is configurable) the mail is rejected with a temporary SMTP error. After that period the mail is passed to the mail system. Many spam scripts will never come back.
Greylisting is, for now, very effective. When we deployed it we saw immediate results, with about a 90% reduction in spam. Even domains that were in an explicit whitelist (and for which no mail was greylisted) saw a significant drop in spam (we put them back into greylisting eventually).
Greylisting has a few drawbacks. The most noticable one is that a lot of users are used to the fact that e-mail is real time, even though the specifications for SMTP clearly indicate it is not and emails can take quite a long time to be delivered. Greylisting adds a delay to mail delivery which, depending on how often a SMTP server retries, can be anything between half an hour to a full day. This might be very upsetting for users, who will complain that their mail is being blocked or is getting lost. Fortunately many greylisting implementations have an auto-whitelist feature, which will make this seem like a temporary problem to most users.
A much bigger problem is that there legitimate servers that don't seem to come back after getting a temporary SMTP error, or come back immediately, get another temporary error and just give up. Other sources for problems with greylisting are scripts on webservers that directly send mail and don't correctly process SMTP errors.
All in all greylisting is working really well for us and the amount of spam received has dropped significantly, but as soon as spammers start implementing SMTP correctly greylisting will not be as effective anymore. Recent increases in the amount of spam and gut feeling indicates that this is slowly happening. The fact that some legitimate mail servers can't deal with greylisting is also a problem.
A better technical solution would be to fix the problem at the source of where spam comes from: correctly configure mail servers, perform egress filtering, patch security holes in web applications, patch client machines, block ISPs that are bulk spammers, and so on. The best solution of all would be to stop buying from spammers.