Obligatory Stack Wars Post

Two days ago, Hitachi announced a preconfigured stack that is perceived to compete with VMware/Cisco/EMC's vBlock.  On one hand, this is pretty significant since it is the second enterprise announcement as far as I can remember since the current USP-V was announced in 2007 (itself a speeds-and-feeds announcement).  The first was, of course, the over-hyped announcement that you could mirror USP-Vs and failover between them via MPIO.

On the other hand, being a pretty serious Block-Head, I have difficulty seeing much significance in this new announcement.  HDS bloggers are posting more on virtualization and general IT topics and less on storage.  Even this press release focuses more on Hyper-V and Hitachi Blades, and not so much on the storage offering.

Historically Did Storage, indeed.

However, mostly due to the recent GestaltIT Tech Field Day, VCE's vBlock is getting a lot more attention.  There are a lot of really good posts on the architecture and what it provides:

• GestaltIT Tech Field Day blog roundup - a lot of detailed vBlock links
• GestaltIT's StackWars post - links to most of the StackWars posts
• @3PARFarley's StackWars Post
• Chuck Hollis's Lengthy vBlock Rationalization
• StorageBod's Take
• My recent vBlock post.

When it comes to vBlock, there appears to be approximately three standard responses:

1. I don't see the point.  Many excellent technical minds share this response.  After all, vBlock is an extremely inflexible product that isn't customizable to the nth degree.  It isn't meant to be, but from a technical point of view, I can see this argument having a lot of validity.
2. This could cause me to lose my job.  I haven't seen this online, but typically this response is from people who see their entire value proposition as being able to develop and support heterogeneous configurations.  Honestly, if that is someone's only value proposition, they'll be in trouble eventually regardless of technology shifts.
3. VCE made the wrong configuration decisions/is too expensive.  If I was developed it, I would do the following differently.  Basically a variant on response 1, this pushes on the details of the implementation and why different choices could be better/cheaper in certain situations.

The consistent argument being made by VCE proponents (typically employed by one of the companies) is that this isn't a product for technical people... the target market is at the C-Level executive role.  This typically follows with analogies about cars and garages and why people don't build their own cars anymore.

First off, that is slightly insulting... all good technical people should be able to show the business benefit that what they support provide.  I'd say a more apt way of putting it would be "focus less on the technology, and more on the benefits."

What benefits?

• It frees IT staff time from building this infrastructure.  While I'm sure there are people who really enjoy analyzing compatibility matrices and getting different vendor technologies working together, there is very little business benefit to these hours (and, I'd argue, most high-level technical minds don't particularly enjoy this... anything that saves me from having to do the tedious portions of my job is a good thing).  Additionally, for this scale (1000s of VMs), the setup time for the VMware clusters, storage provisioning, networking is not a small amount.
• One vendor POC to work with when things don't go well.
• The technology trade-offs are typically made to ensure the vBlock is balanced regardless of what is running on it.  Most of the arguments around the amount of memory per blade is this trade-off in effect. vBlock strives to be a bottleneck-free architecture.
• Any cost benefits are built on the massive reduction of IT hours installing and configuring virtualization environments on this scale.  The argument of "too expensive" may go away in many cases after an honest assessment of what a 500 VM farm costs in time and materials.
• This product is targeted towards large enterprises.  It definitely doesn't make sense at smaller scales (yet).

Does vBlock have a place?  Definitely.  Is it perfect?  No.  I'd argue that it is likely VCE released the current version more to generate the 'stack' conversation with customers and convert a lot of those customers with version 2 than to necessarily sell a ton of v1 vBlocks.  When you're trying to change a fairly deeply rooted mindset, it is going to take some time.

What could v2Block look like?  If the rumors of a converged CLARiiON / Celerra offering running on a customized VMware kernel are true, there is quite a bit of potential for extending that into the vBlock and gaining efficiencies... especially depending on what happens with the CLARiiON RecoverPoint splitter after the convergence.    

The two questions I had after reading this Register article were:

1. When will Enginuity be rolled into the converged platform?
2. Would that allow tighter integration (non-switch based) between Recoverpoint and VMax?

XIV Final Thoughts- Drive Failures a Red Herring?

UPDATE [8/6/2010]:  If you're interested in more current XIV information, I recommend reading Tony Pearson's recent posts here and here.  He also provided additional information in the comments to one of my posts here.

--

Over the past few weeks, between the Wave and the blog posts, I've been thinking about XIV quite a bit. It has taken IBM quite a while to attempt to explain the impact and risk of double drive failures on XIV.

IBM definitely has an explanation, one that could have been told quite a bit ago.  In fact, I'd assume that this is the same explanation they've been giving customers who pushed the point; that the risk is less that it seems due to quick rebuilds and the way parity is distributed between interface and data nodes.  I realize that UREs are a very large concern, but to be honest, I bet less than 5% of customers even think about storage at that level.  Perhaps the double drive failure issue is just a red herring that draws attention away from other issues.

One thing that continues to stick out in my mind is the ratio of interface nodes to data nodes.  On the Google Wave, on of the IBM VARs made the following statement:

Remember there is more capacity in the data modules than in the interface modules.  (9 data, 6 interface)  Why they couldn't make this easy and have an equal number of both module types, I'll never know!  :) 

The interface nodes are only 40% of the array.  Even IBM VARs can't explain why this is a 40:60 ratio rather than 50:50.  It increases the probability of double drive faults causing data loss at high capacity and it is a pretty specific design decision.

I wonder if it is related to the Gig-E interconnect and driving out "acceptable" performance from non-interface nodes.  Jesse over at 50 Micron shares similar thoughts.  Thinking this through (and this is all simply a hypothesis)... perhaps the latency and other limitations of the Gig-E interconnect are somewhat offset by having additional spindles (IOPS+throughput) on the "remote data nodes."  I'd like to load a XIV frame to 50% utilization, run a 100% read workload at it, and see if the interface nodes are hit much harder than the data nodes (in effect, performing like a RAID 0, not a RAID 1).  If that were true, for optimal performance you'd never want to load a frame past the point where new volumes would be allocated solely from data nodes.

I am not claiming this is true (no way for me to test it), but if XIV changes the interconnect to a different type (Infiniband, for example), I will find it interesting if "suddenly" there is a 50:50 ratio of interface to data nodes.

XIV Recap

UPDATE [8/6/2010]:  If you're interested in more current XIV information, I recommend reading Tony Pearson's recent posts here and here.  He also provided additional information in the comments to one of my posts here.

--

A few weeks ago, I created a Google Wave to discuss the architecture surrounding XIV and the related FUD (some of it fact-based) that this architecture attracted.  I intended to post a recap after the wave had died down.

This is not that recap.  The recap was about 80% complete, but more reputable resources have posted much of the same information.  For anyone interested in the actual Wave information, contact me and I'll send a PDF (provided there is some mechanism to decently print the Wave).  There was a podcast Nigel hosted last week that I participated in available on his podcast archives.

New Zealand IBMer the Storage Buddhist wrote this post discussing the disk layout and points of failure in IBM's XIV array... which generated this response by NetApp's Alex McDonald.  Both posts, especially the comments, are interesting and show both sides of the argument around disk reliability for XIV.

This post is meant to bridge a few gaps on both sides, and requires a little disclaimer.  Most of the technical information below came from the Google Wave, primarily from IBM badged employees and VARs.  I have been unable to independently guarantee accuracy- even the IBM RedBook on XIV has diagrams of data layout that contradict these explanations, but with disclaimers that basically say the diagrams are for illustrative purposes and don't actually show how it really works.  So, caveat emptor - make sure you go over the architecture's tradeoffs with your sales team.

Hosts are connected to the XIV through interface nodes.  Interface nodes are 6 of the 15 servers in an XIV system have FC and iSCSI Ethernet interfaces providing host connectivity.  Prior to an unspecified capacity threshold, each incoming write is written to an interface node (most likely the one it came in on) and mirrored to a data node (one of the 9 other servers in an XIV system).

At this point, you can have drive failures in multiple interface nodes without data loss.  In fact, one person claimed that you could lose all of the interface nodes without losing any data (of course, this would halt the array).  The "data-loss" risk in this case is losing one drive in an interface module (40% of the disks) followed by one drive in a data module (60% of the disks) prior to a rebuild being complete (approximated at, worst case, 30-40 minutes).  Or, as it was put in the wave:

"If I lose a drive from any of a pool of 72 drives, and then I lose a second disk from a separate pool of 108 drives before the rebuild completes for the first drive, I'm going to have a pretty huge problem." 

Past a certain unknown threshold, incoming writes start getting mirrored between two data nodes rather than an interface node and a data node.  At that point, double disk failures between different data nodes can also cause a pretty huge problem.

From a 'hot spare' perspective, the XIV has space capacity to cover 15 drive failures.  When you hear XIV resources discuss "sequential failures," they typically mean drive failures that occur after the previous one has rebuilt, but prior to the replacement of the failed drive.  This is an important statistic from the perspective of double drive failures that occur because the failed drive was never detected (have you verified YOUR phone home lately?).

A couple of final thoughts.  First off, the effect of a uncorrectable error during a rebuild was never fully explained.  I have heard in passing that "the lab" can tell you what the affected volume is and that it shouldn't cause the same impact as two failed drives.  Secondly, Hector Servadac mentions the following on the StorageBuddhist's post:

2 disk failures in specific nodes each one, during a 30 min windows, is likely as 2 controller failure

Unless I'm not understanding the impact of a 2 controller failure, there is no data loss with that type of 'unlikely' failure... with the double drive failure, there is significant data loss.  But as a yardstick of "how likely does XIV/IBM feel this outage scenario is," it serves as a decent yardstick.

I tried to make this as unbiased as possible.  I am positive I will be brutally corrected in the comments :-).