Why is spam filtering such a challenge?

This week in Security Levity, I'm digging further into the topic of spam filtering technology. Today, I want to talk about why spam filtering seems to be such a continual challenge for filter vendors. I mean, we've been filtering spam automatically since the mid-80s -- how come we haven't fully mastered it yet?

Surely, by now, the art of spam filtering should be well understood. So how come there's still room for new and innovative techniques to improve the accuracy of spam filtering?

There are even dedicated academic conferences, talking about new ideas in spam filtering, such as CEAS and the MIT Spam Conference. What's going on?

The first automatic spam filters were probably the USENET cancelbots, which used mathematical techniques such as the Breidbart Index (incidentally, we just passed the 16th anniversary on April 12 of the first major USENET spam). Since then, more and more techniques have been invented -- some have fallen into disuse, but others have found their way into common usage in some kind of Darwinian fight for the survival of the fittest  techniques.

So why can't these clever spam filter technologists go off and do something more interesting? Well, the obvious answer is that many spammers are clever, too. Spam filters need to adapt to new techniques employed by spammers. The example that usually springs to mind is the rise of image spam a few years ago.

But that's not the whole story. Not at all.

Spam is quite different from other types of security threats. It may at first appear like it's more of an irritant than a threat, but it has an immediate and tangible effect. Plus, the volume of spam is such that small variations in spam filter accuracy are extremely visible.

For example, your anti-virus protection might block 95% of malware. Of the 5% that's remains, you probably wouldn't be misguided enough to open most of them. Those that you do open wouldn't normally have an immediate, direct impact on you -- and those that you do open would work silently, so you'd not be aware of their existence.

So customer satisfaction for a "95%" AV solution is usually high, assuming it doesn't noticeably slow your PC down. But spam is different.

A 95%-accurate spam filter might leave dozens of spam messages cluttering up your inbox. That's lousy accuracy, and has a very real and immediate effect on customer satisfaction.

What was state-of-the-art accuracy a few years ago is completely inadequate today. It's not simply that people's expectations have been raised, but there's so much more spam than there was back then. That's the hidden reason why spam filter accuracies need to continually improve.

Also, no discussion of accuracy or effectiveness would be complete without also thinking about false positives. When users discover legitimate messages hiding in their spam folder, they rightly get infuriated.

Just another reason why competition is intense, with spam filter vendors working hard to continually improve their technologies. Some are more successful than others, of course!

 
Amir Lev worries about spam filter customer satisfaction, because he's the CTO, President, and co-founder of Commtouch (NASDAQ:CTCH), an e-mail and Web defense technology provider. MORE...