Ask Amir #4: What's a Web reputation service?

This week in Security Levity, I want to talk about 'Web reputation' and how it's used to protect users from malicious Web sites, or sites with malicious content for some other reason.

 
History

Just as with email sender reputation, the history of Web reputation services grew out of blocklists. These were simple, static lists of bad sites, which could be imported into a Web gateway or proxy. The basic idea was that users inside the firewall would be prevented from inadvertently accessing these known-bad sites.

As Web-based infections became more sophisticated, bad actors would hack into non-malicious Web sites and plant malware that unsuspecting users would access. So it became necessary for blocklists to be able to block only certain pages or sections of a Web sites, rather than the entire site. This is especially important with sites that contain user generated content (or Web 2.0 sites, if you insist): most of the site would be OK, except for the occasional instance of malicious content.

But in the same way that simple email blocklists became too unwieldy, Web blocklists needed to evolve. The answer: Web reputation services.

 
What is it?

As with email sender reputation, a Web reputation service is a cloud-based database. Web filters can query it in real-time. This allows the reputation service to be far more reactive to quickly-emerging or fast-moving threats. Much better than trying to push regular signature updates out to all the filters.

This is particularly important as the number of Web sites and pages continues to grow at a frightening rate. For example, the growth of user-generated content is hugely significant -- bad actors love to take over Facebook users' accounts: and suddenly a trusted friend is posting links to malware on her wall!

Web reputation services should not be confused with URIBLs (Uniform Request Indicator Block Lists). URIBLs are focused on filtering spam where a particular Web site or Web page has been referenced in previous spam messages. Spam and other malicious email are useful indicators for newly emergent malicious sites -- but it is just one data source out of many.

 
Shades of gray

Again, as with email sender reputation reputation, Web reputation needs to be more than just binary. It's not good enough to simply say whether a link is 'good' or 'bad': we need shades of gray. Different reputations may require different, nuanced actions: not just block or allow.

Here are a few example of reputation levels and their associated actions:

1. Bad: Block all access to these pages
2. Highly suspicious: Allow only filtered text
3. Suspicious: Disallow JavaScript, Java, Flash, and other embedded or scripted content
4. Slightly suspicious: Do not download executable files
5. Good: No restrictions

Also, some reputation responses are going to be based on heuristics, especially if this is a newly-emergent site. For example, if we know nothing about a site, but it's in a 'bad neighborhood' we might want to exercise caution. Similarly if it's a recently-registered domain.

 
In a future post, I'll talk about how the size of the today's Web makes it incredibly difficult to run a good reputation service, and what the industry is doing to solve the problem.

 
I want to make this an interactive place: where I can answer questions and cover topics that you suggest. Feel free to add comments and ask Amir!

 
When he's not measuring Web sites' reputations, Amir Lev is the CTO, President, and co-founder of Commtouch (NASDAQ:CTCH), an e-mail and Web defense technology provider. MORE...