Steve Friedl's Weblog

« November 2005 | Main | February 2006 »

January 19, 2006

SPF Tip: only one TXT record

A customer reported that large numbers of automated in-response-to-users emails were not being received by users at the big email providers (Yahoo!, Hotmail, AOL, etc.), and this was killing their customer service. "I forgot my password" recovery, "you have new messages", "confirm your email address" were getting lost by the thousands into the ether.

Digging in, I found the problem to be with SPF (Sender Policy Framework) records which the customer provided in DNS. He had a subdomain allocated to the mailsender machines ("mailout.example.com"), and SPF lets you specify what amounts to a list of IP addresses which are allowed to send email with that domain.

Mailservers which check SPF look up a TXT record in DNS from the sender's domain and see if the IP address is listed. If so, it's passed: otherwise it's a forgery. Sites which publish no SPF data don't get any SPF verification or treatment.

The problem is that the customer published two TXT records, thinking they were cumulative. They had a long list of individual IP addresses of mailservers, so they were spread across two records. But this doesn't seem to be allowed, so sites asking for SPF data got only one of them at random. So half the time their email server's IP was in the TXT record which was discarded, and email was getting dumped in massive volumes.

Fortunately, the fix was easy: consolidate two records into one, and convert the list of individaul IP addresses into a smaller number of /24 ranges. Now it all fits in one record, is easier to read, and email stops getting bounced.

Moral of this story: bad SPF is worse than no SPF

Posted by steve at 10:10 AM | Comments (0) | TrackBack

January 16, 2006

Another milestone: 25 years of UNIX

Last September, I reported my 20th anniversary as a consultant, but this month marks another milestone: In January 1981 I used my first UNIX system.

I was taking Dr. Michael Rothstein's Microcomputer Programming class in the Math Department at Kent State University, and we were using an Onyx Z8000 system running the Seventh Edition. Editing was done with ed - no fullscreen editors - and it was quite a change from the Burroughs mainframe which we'd been using for previous classes. UNIX was good.

We also sometimes got to go across the hall to use the VAX-11/780 running Berkeley UNIX - now this was an impressive system. Big as three refrigerators, raised floor with in-room A/C, and real live VT100™ terminals. I still have good memories of virtually living in the VAX room.

I also learned C, and I remember sitting in the student ACM lounge trying to wrap my head around pointers and the fork() system call. Memories of those difficulties certainly informed how I taught UNIX/C later in my career.

Dr. Rothstein was my mentor throughout college, and I'm still grateful for his guidance. I learned much from him, and I'd like to think he picked up a little from me too. In retrospect, I do feel kinda bad about the "big foo" prank, but I'm pretty sure he's gotten over it by now ;-)

But mainly I'm grateful for the incredible good fortune for falling into such a great operating system so early in my career. Nobody had any idea that it would take off like it did, and though I have my feet firmly planted in both UNIX/Linux and Windows worlds, in my heart I'm still a UNIX guy.

Posted by steve at 10:27 AM | Comments (0) | TrackBack

January 14, 2006

"What happened? Did you kill all the spammers?"

The amount of spam received has been on the rise nearly everywhere, much of it due to malware spambots harbored by those who can't properly secure their computers. SpamAssassin and the various blocklists can only go so far in stemming the tide of this onslaught, so I looked into greylisting: it's been a huge hit on my own mailserver and at a customer.

Greylisting relies on the fact that spambots are not real mail transfer agents: they "fire and forget" to the target mailserver, and if the message is not accepted for whatever reason, they just drop it. Real mailservers queue the messages and retry later, spambots don't.

With greylisting, the first inbound connection from a remote is turned away with a 4xx temporary failure, and the sender/recipient/IP is recorded into a local database: the message is not accepted for delivery. If the remote tries again more than 5 minutes later, the previous attempt is remembered, and the connection permitted.

Temporary failure conditions have always been part of the SMTP protocol (example: 452 = Insufficient system storage), and real mailservers simply know how to deal with. They'll be back, but malware won't.

Issues to think about

I have always understood what greylisting did in the technical sense, but avoided it for some time because I didn't really have a real-world sense for what issues might arise. Of course it should mean less spam, but does it require babysitting? Are there bad effects? I just didn't know, but now I have a better handle on it.

This is in the context of the wonderful Postfix mailserver using the postgrey module. My mailserver runs on Linux.

• There is a one-time delay the first time a particular sender/recip/IP of a remote mailserver tries to deliver to you: postgrey's default window is 5 minutes, but you're still at the mercy of the other end to retry - if its queue-retry interval is an hour, you'll wait an hour. You can whitelist particular recipients (support@, sales@) if you simply can't stand this delay in the general case.
• You can whitelist friendly mailservers so that they never recieve a delay. Though a delay is only really applied to the sender/recipient combination, two large enterprises which talk to each other a lot have a very large number of permutations: whitelisting the server this off.
• Some mailservers have a very long retry interval (say, 12 hours), and this is substantial enough that postgrey provides a mechanism to whitelist some servers known to be problematic. Likewise, some don't retry at all, and can be whitelisted on the same basis.
• Greylisting tests are done on a class-C block level, not at the individual IP level: many of the larger mail systems - Hotmail, Yahoo!, AOL - have large farms of sending servers, and each connection could come from a different IP address. This would play havoc with greylisting.
• In this context, whitelisting doesn't mean "is trusted not to send spam", but "it's a real mailserver": there is no benefit in turning away a real mailserver if it's just going to try again shortly thereafter, so we may as well just let it through. The use of greylisting doesn't obviate the use of other antispam mechanisms such as blocklists or content-based filtering.
• Postgrey uses Postfix's clever policy delegation mechanism. Rather than build in custom support for greylisting, one can create a "policy server" which is handed a sender/recipient/IP triplet at the start of an SMTP conversation, and this policy server tells Postfix what to do about it. This is really nice because you can handle policy any way you like.
• Postgrey comes with a small program which can scan the maillog and produce a report of blocked mailservers and all the various parameters: this can be reviewed periodically to look for systems to whitelist, but I've not found anything yet worth taking action on (nor have I looked much).

I perform backup mail relay for several domains, and a few of them have been the subject of relentless brute-force attacks via email. This was many thousands of messages a day which had to run through the blacklists, SpamAssassin, and be bounced due to bad recipients. It made for a very busy mailserver.

Once I implemented greylisting, this shut down to nothing. In the last six days, more than 70,000 connections have been turned away on my little mailserver, and I get just a few spams a day now.

One of my customer's users asked "What happened? Did you kill all the spammers?". If wishing made it so :-)

The spammers will adapt, of course, but I don't think I've ever done anything which made as big a difference in my receives spam as greylisting (the previous winner was using the Spamhaus blocklist).

This is highly recommended for anybody running a real mailserver.

Posted by steve at 09:17 AM | Comments (2) | TrackBack

January 02, 2006

Speaking at SCALE 4X

Next month I'll be speaking at the Fourth Annual Southern California Linux Expo, and my topic is actually mostly non-technical: I'll be talking about my paper So you want to be a consultant...?. This has been a very popular paper, and I hope I can guide some aspiring consultants who have the technical chops but need to kick up their customer-service skills a notch.

I'm speaking on Sunday, Feb 12 from 1:30-2:30, and this is my speaker bio.

I'll be wearing both my Linux Tux pin, and my Microsoft MVP pin :-)

Posted by steve at 11:27 AM | Comments (0) | TrackBack