The art of autoresponders (part 1)
April 20, 2009,
Every once in a while some of our customers go offline for an extended period of time, for things like vacation. Because most of them have a job that has something to do with the Internet a lot of communication is done via email. When our customers are not online for a few days they want to let people know they are not available, without them wondering why they don't get an answer, so we were asked to let our mailserver send autoresponses.
Sending an autoresponse sounds quite trivial: process the incoming mail, send a reply and you're done. But, as always, the devil is in the details. The hard part of sending an autoresponse is to only send it when it is actually needed. Examples of mails that should not be autoreplied are:
- mailing list posts
- automatically generated mails (cron job mails, bounces, newsletters)
- mail from other people that you have already autoresponded to (they usually know after the first mail that you are not there)
One thing you have to keep in mind when making an autoresponder is that it is better to not send an autoresponse to an address that should have gotten one, than to falsely send an autoresponse to an address that should not have gotten one. It is better to err on the safe side and not send anything (compare this to a spamfilter, where you'd rather let a spam pass, than reject legitimate mail).
Sending an autoresponse comes down to determining as fast as possible if you need to send an autoresponse. The mails we get on our systems that don't need to be replied to can be divided in various categories. The categories are listed below, roughly sorted by appearance.
A large amount of mails are spams. There is no need to send an autoresponse to the sender of a spam message, since headers are often faked. If the mail would actually reach the spammer you don't want them to verify that your account actually works. Of course, An autoresponder is not the right place to determine if a mail is spam. This should be done by programs such as SpamAssassin. SpamAssassin sets a few headers that indicate if a mail is spam or not (depending on the configuration of SpamAssassin):
- X-Spam-Flag is set to YES if a mail is spam or likely a spam ("spammy")
- X-Spam-Score has the spam score as determined by SpamAssassin
In our experience a mail that is marked as "spammy" is nearly always a spam, so when X-Spam-Flag is set to YES you can easily ignore the mail and not send an autoresponse.
It also helps a lot to have tight anti-spam measures, such as greylisting, SMTP checks and so on.
Bounces are sent by mail servers as a response to incorrectly addressed mails or indicate other errors. These are machine generated and it makes no sense to autorespond to these mails.
It is very irritating when a mail on a mailinglist triggers someone's autoresponder and the autoresponse is sent to the whole list, triggering angry posts, which are then autoresponded, an so on. This is unnecesary since most proper lists set headers with list information with which you can easily determine if a mail was sent on a list, such as List-Id, List-Unsubscribe, List-Help, List-Info and more. Other mailing lists might have other headers.
The Auto-Submitted header is fairly recent. It is described in RFC 3834 (Recommendations for Automatic Responses to Electronic Mail). The standard defines a few default values, namely auto-generated (for mails that are generated by a system), auto-replied (specifically for autoreplies) and no (for user generated mails). There are more and more systems that are adding support for this header, both when they send mails, as well as process mails.
When mails have the Auto-Submitted header and it is set to something else than no, you can ignore the mail and stop processing, because the sender has clearly indicated that the mail is machine generated.
The Precedence header has been in use for a long time to indicate whether a mail can be ignored by autoresponders. There are three values in widespread use that indicate the mail can be ignored by an autoresponder: list, bulk and junk.
noreply envelope sender
Many systems set the envelope sender to a sender that should be recognizable as a sender that should not be replied to. A few common senders that we have encountered are: noreply, no_reply, no-reply, do-not-reply and bounce. Apart from that there are tons of other addresses in use where it is quite clear for a normal human that no reply should be sent to that address, but it is not clear for a machine.
Examples of (Unix) system accounts which are often used to send mail are www, www-data, wwwrun, nobody, apache, root, nagios, httpd, postmaster and svn (to name a few). These accounts are not used by real humans, so it is no use to send an autoreply to.
A lot of mail programs, such as systems for sending newsletters, websites, discussion forums or webshops, implement their own custom X-Mailer header, for no apparent reason, except perhaps vanity. You can easily filter these headers. Very often these headers are just used for one particular domain, so you easily end up with a list with several hundred values for the X-Mailer header that should be checked. If you host mail for a lot of people that have very different mail behaviour it might easily become much more than that.
Some X-Mailer headers are used by programs that are used by humans, as well as scripts. One example is 'The Bat!' which is used a lot by spam scripts, but which is also a legitimate mail client.
Another header that is used by some programs is User-Agent. Some scripts that send automated mails set this header, although there are far more that use X-Mailer.
There are tons of systems that send mail that have their own non-standard headers.
Once you start writing an autoresponder you'd better forget to get everything 100% correct, because it simply will not happen. There are a lot of mails that should not be responded to, but for which it is really hard to determine if they were sent by a human, or automatically generated. One prime example are vacation messages that are sent by certain versions of Microsoft Exchange, where no extra information is added to headers, except to the Subject, which differs per Exchange version and language.
Of course, there are things you can do yourself to make it all a little bit easier for all of us. The first is of course to be nice and only send what you would like to receive. So when you automatically send mails (like newsletters or autoreplies), add an Auto-Submitted header (see RFC 3834).
It turns out that RFC 3834 is fairly unknown and it people are very hesitant to implement it. We've mailed with companies, asking if they could implement it, because it would help us with filtering. Quite a few companies said they would not implement it, because they feared it would mean getting caught in spam filters. We tried to explain that spammers would not do that, because they want their mails to appear as normal mail sent by a user. If they would adopt Auto-Submitted and admins would filter on it, it would very quickly be abandoned again (and you also should never take Auto-Submitted headers into account when determining if a mail is spam or not). So RFC 3834 could use some extra promotion and your help is needed.
In the next article we will describe an implementation of the autoresponder we use in combination with Postfix.