Steve Friedl's Weblog

« Speaking at SCALE 4X | Main | Another milestone: 25 years of UNIX »

January 14, 2006

"What happened? Did you kill all the spammers?"

The amount of spam received has been on the rise nearly everywhere, much of it due to malware spambots harbored by those who can't properly secure their computers. SpamAssassin and the various blocklists can only go so far in stemming the tide of this onslaught, so I looked into greylisting: it's been a huge hit on my own mailserver and at a customer.

Greylisting relies on the fact that spambots are not real mail transfer agents: they "fire and forget" to the target mailserver, and if the message is not accepted for whatever reason, they just drop it. Real mailservers queue the messages and retry later, spambots don't.

With greylisting, the first inbound connection from a remote is turned away with a 4xx temporary failure, and the sender/recipient/IP is recorded into a local database: the message is not accepted for delivery. If the remote tries again more than 5 minutes later, the previous attempt is remembered, and the connection permitted.

Temporary failure conditions have always been part of the SMTP protocol (example: 452 = Insufficient system storage), and real mailservers simply know how to deal with. They'll be back, but malware won't.

Issues to think about

I have always understood what greylisting did in the technical sense, but avoided it for some time because I didn't really have a real-world sense for what issues might arise. Of course it should mean less spam, but does it require babysitting? Are there bad effects? I just didn't know, but now I have a better handle on it.

This is in the context of the wonderful Postfix mailserver using the postgrey module. My mailserver runs on Linux.

• There is a one-time delay the first time a particular sender/recip/IP of a remote mailserver tries to deliver to you: postgrey's default window is 5 minutes, but you're still at the mercy of the other end to retry - if its queue-retry interval is an hour, you'll wait an hour. You can whitelist particular recipients (support@, sales@) if you simply can't stand this delay in the general case.
• You can whitelist friendly mailservers so that they never recieve a delay. Though a delay is only really applied to the sender/recipient combination, two large enterprises which talk to each other a lot have a very large number of permutations: whitelisting the server this off.
• Some mailservers have a very long retry interval (say, 12 hours), and this is substantial enough that postgrey provides a mechanism to whitelist some servers known to be problematic. Likewise, some don't retry at all, and can be whitelisted on the same basis.
• Greylisting tests are done on a class-C block level, not at the individual IP level: many of the larger mail systems - Hotmail, Yahoo!, AOL - have large farms of sending servers, and each connection could come from a different IP address. This would play havoc with greylisting.
• In this context, whitelisting doesn't mean "is trusted not to send spam", but "it's a real mailserver": there is no benefit in turning away a real mailserver if it's just going to try again shortly thereafter, so we may as well just let it through. The use of greylisting doesn't obviate the use of other antispam mechanisms such as blocklists or content-based filtering.
• Postgrey uses Postfix's clever policy delegation mechanism. Rather than build in custom support for greylisting, one can create a "policy server" which is handed a sender/recipient/IP triplet at the start of an SMTP conversation, and this policy server tells Postfix what to do about it. This is really nice because you can handle policy any way you like.
• Postgrey comes with a small program which can scan the maillog and produce a report of blocked mailservers and all the various parameters: this can be reviewed periodically to look for systems to whitelist, but I've not found anything yet worth taking action on (nor have I looked much).

I perform backup mail relay for several domains, and a few of them have been the subject of relentless brute-force attacks via email. This was many thousands of messages a day which had to run through the blacklists, SpamAssassin, and be bounced due to bad recipients. It made for a very busy mailserver.

Once I implemented greylisting, this shut down to nothing. In the last six days, more than 70,000 connections have been turned away on my little mailserver, and I get just a few spams a day now.

One of my customer's users asked "What happened? Did you kill all the spammers?". If wishing made it so :-)

The spammers will adapt, of course, but I don't think I've ever done anything which made as big a difference in my receives spam as greylisting (the previous winner was using the Spamhaus blocklist).

This is highly recommended for anybody running a real mailserver.

Posted by steve at January 14, 2006 09:17 AM

Trackback Pings

TrackBack URL for this entry:
http://www.unixwiz.net/mt/trackback/60

Comments

Posted by: Daniel at January 16, 2006 02:00 AM

Posted by: Paul at January 22, 2006 06:12 PM