Industry


Ads by TechWords

See your link here


Michael R. Farnum's picture
Michael R. Farnum

Hitting the Security Nerve

SSL offloading / accelerating / load-balancing is scary

I have been taking some training recently on web accelerators, WAN optimization, proxies, and other security devices, and one of the trends I have been noticing is the increase in devices that can decrypt SSL traffic for scanning and acceleration.  This really has me concerned for a few reasons.

 

One is the privacy issue.  There are a few devices that are performing scanning of traffic at the ingress point of the network, and they have the ability (basically through session-hijacking) to decrypt and scan incoming SSL traffic.  While I see the security reasons behind scanning SSL traffic (it is fairly easy for bad guys to hide what they are doing via SSL), I also see a HUGE privacy concern for users.  What if there is some personal data being decrypted?  What about credit card numbers?  While the vendor showed me how these concerns could be reduced (by different policies that stop certain traffic from being decrypted), it still has me worried about abuse by some disgruntled or just plain evil IT guy.

 

Second is the issue of a very tempting hacking target for the baddies out there.  If this thing is decrypting traffic, it makes sense that the bad guys would want to see what it is decrypting.  Also, many of these devices do SSL offloading for web farms.  Basically, they do the decryption and encryption of the traffic to take the load off of the servers on the farm.  Good idea.  But it also means that the device has the keys to the kingdom.  While it is arguably easier to lock down a device than a server, it still gives the bad guy a target to focus on (some use these devices for decrypting traffic to make load-balancing decisions, then they re-encrypt and send it to the server - either way, the problems are the same).

 

And tonight, as I was browsing through my info-junkie list on Blogbridge, I came across this post, and I saw some of the same concerns I have had.  Let's look at this quote:

There are a number of weaknesses in this design beyond the obvious one that the data is in the clear while being processed by the SSL appliance. If you keep your private key in an HSM [hardware service module], you’ve just downgraded its security to “stored in flash and RAM of a front-end network device.” Also, now there are two places to compromise the key. Even if both offered equal security but different implementations (i.e., different bugs), you’ve reduced the private key’s security by 50% because an attacker can choose between the two targets based on their bag of exploits.

This is a fantastic point.  Not only have you given the attacker another place to focus, you have also given him MORE choices for his attack.  That is not a good thing.  And another issue the author brought up to one commenter was that it is likely that the device is using proprietary (A.K.A. non-tested) methods, hence less secure.  He said made his point sarcastically by saying this:

...because no appliances use OpenSSL like the server does. And appliances that use a home-grown implementation are less likely to have flaws than a publicly-reviewed codebase that is 10x older and used in 1000x more systems.

One point I want to make on these devices is that they are quite often being adopted by the network team.  If the company does it right, the security team is going to have to sign off on this puppy.  But that is too often not the case, and this is a huge concern because these devices can go in without a true test of their claims of being secure.  I am not saying network people are stupid.  I am saying that their main concern is not usually security (though everyone should be concerned with security first, but I digress).

 

So basically, I have always been a little wary of these technologies, even if they are supposedly locked down and the vendor gives me ten thousand promises that they are encrypted with AES.  But it also a reality that these devices are being adopted, so the security gurus need to be testing these things to make sure the promises made are promises kept.  The first time someone's ID gets jacked because one of these devices has a crack in it, someone is going to pay.

What People Are Saying

Good points, I thank you for

Good points, I thank you for them. I thank the comments even more though. While I don't think storing private keys for multiple identities on one gateway is a good security practice, setting up a gateway to connect to back end web servers with it's own ssl connection isn't so bad. Sure, you lose the performance gain, but you gain the ability to inspect payloads and perform other operations like transport bridging.

OpenSSL is used by a large

OpenSSL is used by a large and growing community , one that seems be to getting large, from a code base point of view then non-open source commercial products. For example, we use XRoads Networks internet load balancing products and I believe it uses embedded linux and OpenSSL. With such a large group of organizations now deloying this code base, I would think that security is even stronger than other non-open solutions.

Sorry Mr. Farnum, I say

Sorry Mr. Farnum, I say you're way off base on essentially all points.

Privacy: any person or device which has the private keys can read the encrypted data. That's a fact, and security folks know this. Whether it's your webserver, a network device, or John Smith using ssldump, if you have the private key, you do have the keys to the kingdom. That's not a privacy concern, it's just a fact.

Hi-jacking: there is no hi-jacking involved in any cases of standard acceleration or content filtering. In all cases it's using well-established "standard" SSL public key cryptography principles.

Credit card data: Every company decides that at some point, the devices or network is secure. Some companies decide that everything behind their firewall is secure, so encryption is no longer required. Others require that encryption is "end to end" to the web server, but then the web server itself must of course deal in plaintext unencrypted data in memory, in logs, in debugging. Many companies store data in databases unencrypted. In all cases it doesn't matter which is chosen per-se: security folks understand these obvious trade-offs and make decisions based on business requirements. Acceleration/filtering devices don't change anything.

Keys in network devices: security principles don't change based on the devices involved. Servers have SSL private keys stored on hard disks and in memory. Network devices have the same. It doesn't change anything. If your switch isn't as secure as your server: you fail. If I shutdown your core switch, or compromise it in such that I can capture packets going accross the wire, it doesn't matter what's on your server. Network devices are as secure as and possibly more secure than servers, and are typically much more critical.

Home-grown vs. openssl: Perhaps this will be a surprise to you, but many and I might even say most acceleration/filtering solutions use OpenSSL or a hardware-enhanced version of OpenSSL. For example, Cavium (a supplier of crypto acceleration hardware) provides hardware-accelerated OpenSSL. Search Cisco or Juniper's site or any other popular networking vendor for "OpenSSL" and you'll see how extensively it's used. In any case, this is a weak argument.

"I have always been a little wary of these technologies": I suggest your lack of comfort with the technologies stems from your lack of understanding of the technologies and security principles involved.

There are advantages and disadvantages to having SSL acceleration and filtering technologies in place. Properly considered and implemented, I suggest they are more secure than the alternative. Improperly considered and implemented, I suggest they are no worse off than direct-to-server models that are thought out with equally little care.

Security is a mindset, a commitment, a way of doing business. It's not a feature, it's not a product. Security is not as simple as encryption boundaries (server vs. "network").

If the "end" is your

If the "end" is your protected network does it really matter? Inspecting the traffic after a load balancer or offloading SSL device is a must especially as you look at trends of blackhats using SSL to hide malicious activities.

I happen to deploy just such

I happen to deploy just such a device and initially had identical concerns. However, when I thought more about it I realized it isn’t that big a deal for several reasons:

1. The accelerator should be 'classified' equal to the most sensitive data it decrypts and that should determine who manages the accelerator.

2. Management access to the device itself should be restricted to the internal LAN only. This reduces the attack service of the device. By restricting public access to only the VIPS, the only way to target the accelerator is through an attack that actually targets the VIP but gets executed during the offloading process.

3. Some of these accelerators also incorporate firewall technology as well as 'replace text' features that can be leveraged in your favor, thus adding to your security posture.

I have more details as well on my blog here. I tried to do a track-back but I don't think this blog honors them.

Michael- excellent article!

Michael- excellent article! I agree with you. However, to Michael's comment, I disagree. Not all ends are created equal. The SSL data was supposed to be encrypted all the way to the real "end". When it is decrypted before that, you are introducing risk into the equation. I have written more about this on my blog here.

Interesting read, but I

Interesting read, but I guess a large number of appliances run an internal embedded linux version. Why reinventing the wheel?

Sure, it increases the attack vector, but an SSL accelerator only load balances traffic, and that's it. Hence, less room.. i guess, for services to be exploited. Often, the SSL is even terminated on that device, and is passed on clear text afterwards.

A more potential downsite i'd see, is the fact that a webserver gets compromised, and a sniffeer is installed, which would capture sensitive data. Though, this data will be processed on the server anyhow, and probably stored in databases. What would stop an attacker from rewriting the ASP/JSP/whatever pages on it?

It's a tradeoff between speed and security, and the main objective of SSL was to ensure end to end encryption, they just replaced the "end" :)

just my 2 cents,
michael