Industry


Ads by TechWords

See your link here


Linux security idiots

There are some Linux system administrators out there who should be glad, very glad, they don't work for me because I'd be firing them today.

Why? Because the U.S. CERT (Computer Emergency Readiness Team) is reporting that Linux systems are being successfully attacked by crackers using compromised SSH (Secure Shell) keys. Once a system has been cracked with an illegitimate SSH key, the cracker uses local kernel exploits to gain root access and a rootkit called "phalanx2" is installed.

Once in place, phalanx2 steals more SSH keys from the compromised system. These keys are then, of course, sent to the attackers, who will use them to try to compromise other sites and... Well you get the idea.

So, why am I ticked off? Because to get those SSH keys in the first place they had to be vulnerable to capture. And, guess what? In the last few months there have been two occasions when it's been revealed that certain Linux distributions were wide open to attack.

The first time was when Debian, thanks to some really fouled up development thinking left OpenSSL on Debian, and related distributions like Ubuntu, wide open for attacks from September 17th 2006 until May 13th 2008. OpenSSL provides SSL (Secure Socket Layer) and TLS (Transport Layer Security) protection. It's used through Linux internally and in network communications for 'secure' transactions.

Then, much more recently, Red Hat's RHEL (Red Hat Enterprise Linux) and Fedora were briefly compromised. In these cases, Red Hat says some, not many, but some OpenSSH packages had been messed around with.

"No problems," said Red Hat. Funny that a few days later we're seeing successful SSH attacks on Linux servers isn't it?

OK, at this point, we don't know which, if any of these publicly acknowledged security problems, has lead to the current rash of attacks. I do know one thing though. First, that if system administrators had been awake at the switch they would have already set things right on both their Debian/Ubuntu and their Red Hat/Fedora boxes.

It's not like there was anything obscure about either of these security flops. You'd need to be deaf, dumb, and blind to have missed these stories and negligent to have not fixed any potential problems.

OK, so let's say you did manage to blow it. What do you do now?

Well, first you make sure that your systems have had the necessary fixes installed if you're not sure they have been. Next, according to CERT, you can see if you have a case of phalanx2 with the following steps:

" "ls" does not show a directory "/etc/khubd.p2/", but it can be entered with "cd /etc/khubd.p2".

" "/dev/shm/" may contain files from the attack.

" Any directory named "khubd.p2" is hidden from "ls", but may be entered by using "cd".

" Changes in the configuration of the rootkit might change the attack indicators listed above. Other detection methods may include searching for hidden processes and checking the reference count in "/etc" against the number of directories shown by "ls".

If, lucky you, you have a case, CERT recommends that you disable key-based SSH authentication on the affected systems; perform an audit of all SSH keys on the affected systems, and, oh boy won't this be a lot of fun, notify all key owners of the potential compromise of their keys.

What's lucky about this? You're lucky that I'm not your boss or your next step would be finding another job. Hey try this phrase out: "Would you like some fries with that?" If you can say that, you may have a challenging and exciting future in the fast-food industry.

Linux really is more secure than most operating systems, but, as the security mantra goes, "security is a process, not a product."

Some security breaches you can't stop. If your CEO insists on keeping his passwords on a post-it note on his display, there's not much you can do. But, for attacks like phalanx2, where simply being aware of recent major security problems and updating systems would have stopped the assault in its tracks, there is no excuse.

What People Are Saying

Security by obscurity....with private keys?

hey steve.
i had to laugh about your sarcasm here.
so, linux-admins are lucky, coz you are not their boss? well maybe. but maybe not.

First of all this root-kit phalanx2 is nothing special at all. sure it compromises a ssh-based system with just this issue of authentication.

this rootkit has then a the chance to get on more private keys of other servers, which may me based on just the authentication of private keys, and so on.

well, my opinion is where one weak server may be, the next weak server is not far. and weak admins either.

It is sometimes not understandable why some "roots" out there don't harden their system for example via software-middleware like pam.

advanteges are:
1. granulating the system authentication very briefly.
2. you can disable root authentication via non-local ip's. Root authentication via "exampleUnRxW2.!".example-user, and then via authenticated su /PAM.
In Combination with a good private ssh-key on such a host this is very powerful.
3. enable ssh/pam on a special tty
4. group all other users to the group "luser", the rest like root and the special example user above to the wheel group. Users who are not part of wheel are automatically part of the "lusers"-group and will not have the chance to get root...even if they would have the root password in clear-text.

The above example is just a question of hardening the linux-system, from the ground to the top....before you get online for everyone.
I guess that most game-servers on linux-basis might be quite endangered through this root-kit out there.
But i can bet most linux-admins out there might not use a standard-linux system for a important server.
Big Distributions like RedHat or SuSe have hardened Server-Software anyway. You can't be admin on such systems if you are a beginner.

I am using Debian for the last 6 years now. Since 1994 i am into linux, first into SuSe then RH. From time-to-time maybe a ubuntu-live or Knoppix-Live-DVD. I have friends who have been working for the zOS/OS390 linux-kernel in germany. I have other friends who are managing in major companies linux-systems. I am very sure they make all a very good job. coz Security is a state of mind, and not a matter of ssh alone.

If you would fire all those guys, who have been participating and developing linux, you could fire everybody who is working with computer systems in general.

The private ssh-key is just a plain hiding method by "obscurity". But everyone should know that security by obscurity is stupid.
I know there are people out thre that my user-name example might be obscurity. This is just for providing a name-dictionary attack. The special username, which can be root via su/Pam is a part of a chain of a security process. This process is never ending.
The problem is to automate this via scripts and so on. There might be admins out there who are just too lazy to treat every linux server as a important server. So they leave their system in a almost standard installation.

Sure there might be sysadmins out there who have no glue of security at all. You know how many Users coming from windows have a root-server out there?
Try to fire them.

best regards
mbv

Fixed Fast

Yeah, it sucks that this happened. But, The great thing about open source is that it got fix right away.

No system is 100% secure. But, at lest Linux and most other Unix like systems are 99% secure.

If you rename a plain text file to .bat in windows it can take down the whole system. There is no security there.

All you need is...

fwknop

Better buy a BIG Rolodex

OK, your Linux system got hacked. Big deal. Windows systems have been getting hacked for years. Did you fire all your Windows admins?
Here's a tip: Fire your entire IT staff, go buy a BIG rolodex and put your data there. Place it in a Fireproof/Waterproof safe at night, and make photocopies of all your rolodex cards, and mail them to your 'hot' site.
That way, you'll be all safe and sound.

Agreed and etc

So would I, I mean fire a person who doesn't do his/her job. No questions about that! But would it always be system administrator or such - maybe not always?

Now, unfortunately as usually, it's not always lazy or incompetent person but the company which has problems. First, of course, maybe they hired an incompetent person - duh! At least two people to fire or have same problems all over again. Second, worse, there is micromanagement or "micropolicies" - no changes without an approval, which, as we all know can take a long time if they ever are even replied.

Just had a consulting job in one of those places, the personnel knew what the problems were and were fully capable to handle the situation, after my report to top management to straighten that out, the CIO left and the current problems were fixed in two days. Previous plan was to fire two administrators and to hire two more managers?

So, the "root cause" is not always as simple as it looks first.

Making a lot of assumptions here

The author is making a lot of assumptions here. These include:

1. That the systems being compromised in the SSH attacks are RedHat/Fedora systems.

2. That the reason RedHat released new OpenSSH packages was because the old ones allowed these types of attacks.

To the first assumption, I would say that unless he has some other source than those he provided in the article that says that the targeted systems were RedHat/Fedora systems, then he is talking out of his hat. The CERT information says nothing about a particular distribution being targeted. And unless you have evidence that RedHat's claim that RHN wasn't compromised is also false, this makes your contentions even more specious. If it turns out that the targeted systems are CentOS systems, Oracle Enterprise Linux, or other RHEL clones that don't use RHN, then you might have a case, but until CERT is more forthcoming about the distribution running on the compromised systems, then I say that your conclusions are simply speculation.

To the second assumption, again, I say that unless you have evidence that the replaced OpenSSH packages made systems vunerable to this particular type of SSH attack, then you are just speculating. Given that RedHat's servers that were compromised contained the OpenSSH packages, which need to be above suspicion, then it only makes sense that RedHat release new versions of them so that any potential tampering that may have occurred could be thwarted.

I am not saying that RedHat's handling of this particular incident was perfect, but I think that your assumptions that the SSH key attacks and the compromise of RedHat's servers are connected is not based on hard fact.

I agree that the system admins are lucky that they don't work for you, since you apparently like to make knee jerk reactions based on speculation.

If the system admins were indeed to blame for compromised systems, then they should be fired, but only if it was really their incompetence and not pressure from others in the company to do something that goes against good security practices.

I have been in a situation like this before, where I decided that I could not in good conscious continue to work at a company where management was making security decisions against the recommendation of the system admin staff. And you can be sure that if I had stayed with that company and something had gone wrong, that it would have been my head on the chopping block instead of the VP whose argument for making his decision was, "I am best friends with the founder of the company and we are going to do it my way because I said so."

I normally enjoy reading your articles, but I think that in this case, that you are off base.

You are from India right

You are from India right ;-)?

I think you missed the

I think you missed the point. The machines getting subjected to attacks now aren't necessarily debian or red hat based. An ssh key to these systems could have been obtained from a system with a compromised openssh. So regardless of the distro (even the OS) of the machines being attacked on a wide scale now it most likely began with compromised machines not being fixed.

SSH

I don't understand why people leave SSH running all the time anyway. People attack the heck out of SSH.

I only run SSH when I need it. Most of my server run Webmin and when I need SSH I turn it on.

Or I use something like fail2ban.

The key here to take the focus off your machines. When hackers and scripted apps know your machine is running SSH your machine will be the focus of attack.

Once I turned off SSH on my servers outside scans and focused brute force attacks went down by over 50%

Linux SSH hack

You know I never really liked SSH keys. I know it make life easier to ssh to a bunch of boxes. But what amazes me about this article is that the lazy admin's refused to turn it off. This happens a lot. I have seen it on both sides of the fences,Windows and Linux admin's. It is just stubbornness that causes more work in the end. One day someone will learn one ounce of prevention is worth a lot.