Industry


Ads by TechWords

See your link here


Subscribe to our e-mail newsletters
For more info on a specific newsletter, click the title. Details will be displayed in a new window.
Computerworld Daily News (First Look and Wrap-Up)
Computerworld Blogs Newsletter
The Weekly Top 10
More E-Mail Newsletters 

DBA's not to blame for Oracle patch application failures?

Paul Vallee, CEO of The Pythian Group, has a problem with a story I did recently based on a survey from Sentrigo Inc., which showed that two-thirds of Oracle database administrators had never installed the company's quarterly Critical Patch Updates.

According to Vallee, whose company provides database management services to some pretty big organizations, the article makes it sound as if Oracle DBAs are mostly a lazy, irresponsible bunch who don’t care too much about security. He seemed especially nettled about a quote from Sentrigo CTO Slavik Markovich saying that some database administrators don’t even monitor for Oracle’s CPUs or know when they come out.

“It made my jaw drop. They are making it sound as if it’s the DBAs' fault that these patches are not being applied,” Vallee said. Nothing, headded, could be further from the truth. This idea that DBAs would be running scared of the effort involved in applying the fixes or are plain lazy is just ludicrous, Vallee said.

“The real story that needs to be told is that DBAs don’t have the leverage they need for security best practices to be followed,” when it comesto database patching, he said. According to Vallee, less than 10% of the customers that his company does work for have applied Oracle CPUs to their databases. “It’s not because we don’t advocate it. And it’s certainly not because we don’t know how to do it,” he claimed. Rather, it’s because no policies exist within these companies that requiretheir DBAs to installthe patch sets, he said.

“There is no standard of care. There’s never anything written that says they need to patch it,”Vallee said. In the absence of such leverage, it can be hard mustering the resources and the time needed to implement Oracle patches. So DBAs tend to do the obvious thing, which is to focus their attention on other more pressing projects, according to Vallee.

There are other reasons as well. Commenting on the story, a reader who identified himself as Markand said he has been an Oracle DBA for the past six years, admitted that he hadn’t been applying the patches either. But the reason was not because of a blatant disregard for safety, he said. Rather it was because his company was running an older version of Oracle that is no longer supported. “The other part of the equation is risk tolerance,” Markwrote. The fact is that for some companies, the risk of someone breaching a database outweighs the months “of testing, profiling, resource consumption for test environments, scheduled downtime and patch repercussions”that are involved in an Oracle patch install, he said. For others, it quite simply does not.

And then of course there’s the whole patch reliability issue that Windows administrators run into all the time. A reader who said he (or she) was from a university notedthat IT staffers there had become “gun shy” about installing Oracle patches because of problems they had with one previous patch set. Despite installing the CPU only after extensive testing, a problem manifested itself “under the load and usage profile of our production instance,” the readerwrote. “If Oracle cannot release these patches so they don't break things, who will want to take the risk of causing even more trouble than the patch is supposed to fix in the first place?”

OK, so maybe there are some perfectly valid reasons why DBAs should not be faulted for not patching their Oracle databases. But it doesn’t change the fact that despite all the data breaches of the past few years and requirements such as HIPAA and PCI, there still are an awfullarge number of pretty vulnerable databases out there that fewpeople are doing a thing to patch. One thing no one seems to be disputing a whole lot is Sentrigo’s results which showed that just one-third of all the DBA’s they polled have ever installed an Oracle patch.

Ifit isn't the DBA's responsibility, whose is it? And if it is indeed the DBA that the buck stops with, what needs to be done to ensure they can do the job in the mostefficient manner?

 

Related Article:

Update: Two-thirds of Oracle DBAs don't apply security patches

 

What People Are Saying

I second some of that

I do agree that DBA's are NOT lazy. However, I do not buy the "no policies exist within these companies". Yes, my policy (I work for the CSO) doesn't say "DBA's must install Oracle patches", but it does require that we run supported applications and that vendor patches are applied within a timely manner after release and testing.

Do I have 50 people I can lend for that? No, I'm on a team of 7. There are 3-5 times that many DBA's at my company, plus dedicated test teams. Planning for maintenance is a business process that must be supported by the business. It's not security or the DBA's fault when that doesn't happen, but most DBA's are closer to the process and should "remind" business owners of that need.

Ignorance is bliss for management

I have to concur that failure to apply patches often isn't the fault, or even in the hands, of the DBA. I'm a database consultant, and for some of my clients, security is simply not a priority.

Despite my best efforts, warnings, and pleading, there are some organizations that just won't listen. Unfortunately, if (or when) a breach is detected, there will be hell to pay.

And it's not just patches. I have clients that use packaged apps. The well-known application accounts, that the vendor requires have DBA-level super-privileges, have the password set to the account name. So, anyone with a client on their desktop and a little knowledge can own the database.

I recently fired a government (military) account because they flat refused to do anything about security. Default passwords in the database, default listener ports, no patches, easy to guess passwords, no real backup policy, no qualified sysadmin, unpatched OS, and the servers are web-facing. The database has personal information, and though we're not talking SSN, it does have enough info that a spammer would have a field day with the several million valid names and email addresses. For two years I battled with them to lock things down, but it just wasn't a priority. Rather than facing potential liability when (not if) it gets compromised, I walked away.

Often, Oracle DBA's face a tough sell to management. They seem to think that they paid so much for Oracle licenses that it'll just work out of the box. Whether it's selling them on the need for security, or performance tuning, or the need to upgrade to a supported platform, it's an uphill battle for many of us.

There are just too many cases out there where management believes that their network or firewall will protect them, users are happy sheep that aren't motivated or knowledgeable enough to do any damage, and that adding cheap memory or faster processors will fix performance, while saving money on database administration.

DBAs are definitely not lazy

Hi Jaikumar,

As a former DBA for many years I definitely do not insinuate that DBAs are lazy. The simple fact is that they just have toם much working against them when trying to apply the CPUs:
1. The need to test all applications using the database is a heavy burden
2. Oracle supports only the latest patchsets
3. The lack of application vendor certification of the CPUs
4. The simple fact that it takes a huge amount of work to manually shutdown the database and apply the patch in an organization running hundreds if not thousands of instances
5. For production critical databases you have to consider maintenance windows which might come once a year
6. The lack of understanding by some IT security personnel of the severity of the problem simply does not generate enough pressure in the organization - please see Rich Mogull's excellent post on this topic - http://securosis.com/2007/11/20/who-owns-database-security/
All in all, I know of companies that analyze and deploy CPUs as soon as three months after release but those companies are very few and usually have budgets in the millions for such things…

Slavik

I Blogged On Oracle's Security Paradox

Key issues: patches pose catastrophic operational impact risks; many DBAs not responsible for security; software vendors are incentivized to migrate users to new version; security pros dependent upon general purpose netsec appliances...

www.archimedius.net

G