Big, Big Storage Systems - Fact or Fiction?

Storage Anarchist is wrong

Storage Anarchist is wrong about HDS not supporting DIB. We support an 8 byte DIB and have for over two decades for FC drives and from the beginning with our support for SATA drives. EMC needs to do their homework if they are going to make such comments. In fact HDS does both DIB and RAW and now enhanced RAW. According to Storage Anarchist, EMC only supports DIB which is a big mistake on EMC's part since using only DIB can lead to long-term corruption of data.

I stand corrected

Sorry, I apparently missed the documented use of DIBs in Hitachi kit. My bad.

As to Read After Write, that seems to add a rather hefty performance penalty. Do you have any statistics on how frequently RAW actually detects errors?

Instead of RAW, EMC arrays do continous background integrity checks of all disk (and flash) data. We believe that this approach has less impact on performance and provides greater protection than merely verifying once as does RAW. (Perhaps you've addressed this with "enhanced RAW".)

The real point here is that CERN wasn't using an array that had ANY of this sort of protection, and if they were, it is highly unlikely they would have suffered from their reported data corruption. Storage devices (disks, flash and RAM) are not infallible, and good storage arrays will protect against all kinds of data integrity risks that home-brew storage kits won't.

Response to Storage Anarchist

Anyone interested in Storage Anarchist’s comments I think it is important to know he is a VP at EMC - thus his references to EMC solutions.

Barry – aka Storage Anarchist – I actually recommend that you re-evaluate your analysis of the CERN report. The CERN report stated that 80% of the errors were due to 64k regions of corrupted data and appears to be SATA disk drive firmware problems. Are you saying that you believe that DIB addresses this?

I did a bit of research and as I understand DIB is at the block level on the physical disk and therefore a checksum could be validated but the data can still be incorrect.

What I like about Data Direct and how they deal with silent data corruption on SATA drives is they perform command retries, drive resets, individual drive power recycling and if needed perform a rebuild based on journaling. In this regard they don't manage disk drives individually but as a community - which provides integrity above and beyond checksums at the individual drive level. Also their process doesn't require any additional capacity consumption AND it does not slow down system performance AND is done instantly.

You do make one good point about the total aggregate capacity of lots of “smaller” storage systems present the same challenges as a single large one. However, you did miss the point I was making - there is value in having a single storage system supporting lots of capacity and therefore a discussion of reliability and scalability with these types of systems is an important topic.

DIBs

DIBs may or may not be managed by the drive itself; for maximum robustness, DIBs and checksums should be driven by the storage controller - this helps protect against buggy drive firmware causing data corruption.

And for the record, Data Direct is not unique in how they manage drives. Many other arrays (including those from EMC) provide similar features as those you list (retry, reset, recycle). Additionally, some arrays (like the Symmetrix) actually generate and validate checksums to ensure end-to-end data integrity for data blocks as they move within the array. Importantly, integrity is assured by implementing multiple overlapping error detection domains, ensuring that the bytes received from the host stay the same all the way out to the disk and back. Although CERN attributed errors to the SATA firmware itself, it is possible that some percentage of the corruptions they found had actually occurred further upstream in the I/O path.

That said, while clearly not infallible, Data Integrity Bytes of sufficient size can indeed be expected to detect the errors found by CERN, if indeed they were caused by corruption localized to the drive or drive firmware.

This is because a SATA drive doesn't store data in 64K byte increments; the native block size is (by current SATA definition) 512 bytes. Using T10-DIF, for example, an additional 8 bytes of DIB would be more than sufficient to detect even multi-bit errors in any of the 512 byte blocks used to store the 64K host writes. Each 64K logical block would be actually stored as 128 512-byte blocks with 128 8-byte DIBs, more than sufficient to identify multiple errors spread across the entire 64K byte block.

Note that on SATA drives with a fixed block size, DIBs are typically stored in separate 512-byte blocks. On SAS and FC drives that support it, the drives are usually formatted with 520 byte blocks and the DIB is stored alongside the data.

Whether rebuild from a journal or from the partners in a RAID set, the potential for bit errors within the data blocks used to rebuild the data always exists...journalling is not necessarily better or worse than RAID-based rebuild, it's just another approach.

And yes, there is definite value in consolidating lots of little arrays into a single big one - especially if that consolidated array can cost effectively support multiple different tiers of capacity in the same footprint (e.g. Flash, Fibre and SATA storage). This is a key component of EMC's strategy for both Symmetrix and CLARiiON, and both arrays currently support more spindles and usable capacity than any of their direct competitor's offerings.