- The Hot Aisle has an article showing the mathematical inevitability of storage arrays moving to Flash and SATA (AKA Flash and Trash). While SSD adoption was slow initially, almost every vendor is offering it in some fashion. I agree that to reap the full benefits, it will eventually have to stop looking like a standard "spindle."
- Storagezilla had a nice post on Oracle's declaration of war on NetApp. It is the second time Oracle has declared war on an established vendor in recent memory, the first time being their release of rebranded RedHat. It doesn't look like it affected RedHat in the long term, and I doubt it'll affect NetApp much. During storage purchases, you're relying on the vendor's ability to deliver as much as what their delivering, and it'll be some time before Oracle has proven itself in the storage realm.
- EMC obliterates the competition in SPECsfs_cifs and posts extremely competitive numbers in SPECsfs_nfs. The cifs benchmark originally looked like the result of some bored engineer in an EMC lab trying to see how much he could destroy the existing rankings - the benchmark was ran on all SSDs (well, 4 FC disks for Celerra information). I wonder if this will cause some of the other vendors to post updated cifs numbers. Storagezilla claims they won't because of how bad their implementation is. It could be due to the few vendors that can offer that amount of SSD storage. I have to ask, does CIFs really make sense for this type of a workload?
- Storagebod posted on the cloud-angle of Apple's iPad announcement. I thought something very similar when I saw the announcement, except for a few things. First of all, the bulk of storage on most consumers computers is media. iTunes already has most of that content available, so pushing that storage into the iTunes cloud is more a function of scaling IO/access rather than 'having sufficient storage'. In fact, if Apple could talk Big Content into allowing them to detect non-iTunes media and offer free-of-charge the equivalent iTunes media, this would be even easier. Pirates won't buy songs they already 'have', so there isn't a lot of money left on the table AND it reduces the availability of completely wide-open music/movie files.
- Cleversafe posted a good primer on silent errors. This is the main reason why details matter when it comes to RAID implementation, and why you need more than 1 piece of parity for large drives.
- If you have the chance to try it, New Belgium's Spring Seasonal, the Mighty Arrow, is quite tasty. A nice pale ale that is light on hops and extremely drinkable.
Sunday, January 31, 2010
Odds and Ends - 01/31/10
Thursday, January 28, 2010
vBlock and Private Cloud
I'll be honest... when EMC announced the vBlock architecture alongside the VCE initiative, I didn't quite get its importance. In my mind, there was very little benefit to these preconfigured stacks, especially at the price points that I heard rumored.
After a few weeks, I think I've got a handle on it and why this is potentially a big deal.
When technologies first come out, implementations are fairly complex and require quite a bit of trial and error alongside a fairly good breadth of skillsets. VMware was no different... it required people who understood virtualization technology, networking, storage, Windows/Guest OSes, and security. As time passed, the implementations become easier due to the toolsets becoming better and the availability of knowledge increasing.
After that, the difficulty shifted. Beyond the politics of getting signoff for virtualizing as much of your environment as possible, the next challenge was taking the architecture and scaling it big. As the technology progressed, this became easier (it is still not what I'd call "simple") and things start shifting towards "how do we backup/recover these large environments, and how do we leverage the technologies in play for Disaster Recovery/Business Continuity?"
But what strikes me is, as the challenges become greater, the importance of a good fundamental implementation remains. Compatibility matrices still need to be kept up to date, documented, and tested. Research needs to be done on new server hardware models and processor models, alongside updating any documentation/procedures that change as new VM farms are built on the new hardware.
What is really the kicker, though, is typically the people who originally brought in VMware are still at that level, making sure the implementations are solid rather than spending the time on the more difficult "next big thing."
So how does vBlock fit into this? Simple. If you are an organization where there is a large virtualized environment and you aren't allowed headcount increases, vBlock offers an opportunity... namely to take some of your best technical employees and allow them to be repositioned to where they can provide the most value. Lets face it, an architect with 4+ years of experience is wasted on validating firmware levels. Similarly, in these large environments where vBlock makes sense, churning VMware farms to stay supported isn't a great use of highly skilled resources time.
If you notice, the vBlock architecture does not cover the current cutting edge portions of leveraging the virtual infrastructure as much as possible to benefit DR/BC.
How does this play into Private Clouds? Simple. There are a lot of private cloud definitions floating around, but for the purpose of this, I'm going to drastically simplify it.
A private cloud is "self service IT."
A lot of people get edgy when you start discussing private clouds. The foremost argument typically runs along these lines: A private cloud, implemented properly, greatly reduces the time to deploy a server/system, increases the accuracy of the build process, and dramatically reduces the "friction" of implementation. By "friction," what I'm really getting at are those things that take a 1 day server build and turn it into a 2 week process. By reducing the friction and difficulty of implementing new systems, the total cost will go up because there will be a lower barrier of entry (easier/quicker build process = more systems being built = more money goes to VMware, Microsoft, and your storage vendor).
Not quite. A good private cloud still requires strong processes. Systems have to be sized, priced, signed off on, and someone with a budget has to agree to fund it. None of that really changes (granted, depending on many factors, sizing exercises may be reduced in many cases). All that changes is, once all of the appropriate approvals are attained, systems can be deployed in days instead of weeks. Money still should not be spent without a good business and cost justification - but in either model, if someone gets the appropriate signoffs and funding, the environment is going to get built.
The second argument is fairly simple: In order to implement a good private cloud, it takes automation, standardization, and virtualization. If things get to the point where it is extremely simple to deploy systems, then people could lose jobs. Lets be honest with ourselves... if the only value IT resources provide is the ability to install a server, then their days of employment are probably numbered anyway. If their only function is to ask "small, medium, large, or super-sized server," they are probably in the wrong profession.
The final bit is this. The vBlock provides a couple things:
After a few weeks, I think I've got a handle on it and why this is potentially a big deal.
When technologies first come out, implementations are fairly complex and require quite a bit of trial and error alongside a fairly good breadth of skillsets. VMware was no different... it required people who understood virtualization technology, networking, storage, Windows/Guest OSes, and security. As time passed, the implementations become easier due to the toolsets becoming better and the availability of knowledge increasing.
After that, the difficulty shifted. Beyond the politics of getting signoff for virtualizing as much of your environment as possible, the next challenge was taking the architecture and scaling it big. As the technology progressed, this became easier (it is still not what I'd call "simple") and things start shifting towards "how do we backup/recover these large environments, and how do we leverage the technologies in play for Disaster Recovery/Business Continuity?"
But what strikes me is, as the challenges become greater, the importance of a good fundamental implementation remains. Compatibility matrices still need to be kept up to date, documented, and tested. Research needs to be done on new server hardware models and processor models, alongside updating any documentation/procedures that change as new VM farms are built on the new hardware.
What is really the kicker, though, is typically the people who originally brought in VMware are still at that level, making sure the implementations are solid rather than spending the time on the more difficult "next big thing."
So how does vBlock fit into this? Simple. If you are an organization where there is a large virtualized environment and you aren't allowed headcount increases, vBlock offers an opportunity... namely to take some of your best technical employees and allow them to be repositioned to where they can provide the most value. Lets face it, an architect with 4+ years of experience is wasted on validating firmware levels. Similarly, in these large environments where vBlock makes sense, churning VMware farms to stay supported isn't a great use of highly skilled resources time.
If you notice, the vBlock architecture does not cover the current cutting edge portions of leveraging the virtual infrastructure as much as possible to benefit DR/BC.
How does this play into Private Clouds? Simple. There are a lot of private cloud definitions floating around, but for the purpose of this, I'm going to drastically simplify it.
A private cloud is "self service IT."
A lot of people get edgy when you start discussing private clouds. The foremost argument typically runs along these lines: A private cloud, implemented properly, greatly reduces the time to deploy a server/system, increases the accuracy of the build process, and dramatically reduces the "friction" of implementation. By "friction," what I'm really getting at are those things that take a 1 day server build and turn it into a 2 week process. By reducing the friction and difficulty of implementing new systems, the total cost will go up because there will be a lower barrier of entry (easier/quicker build process = more systems being built = more money goes to VMware, Microsoft, and your storage vendor).
Not quite. A good private cloud still requires strong processes. Systems have to be sized, priced, signed off on, and someone with a budget has to agree to fund it. None of that really changes (granted, depending on many factors, sizing exercises may be reduced in many cases). All that changes is, once all of the appropriate approvals are attained, systems can be deployed in days instead of weeks. Money still should not be spent without a good business and cost justification - but in either model, if someone gets the appropriate signoffs and funding, the environment is going to get built.
The second argument is fairly simple: In order to implement a good private cloud, it takes automation, standardization, and virtualization. If things get to the point where it is extremely simple to deploy systems, then people could lose jobs. Lets be honest with ourselves... if the only value IT resources provide is the ability to install a server, then their days of employment are probably numbered anyway. If their only function is to ask "small, medium, large, or super-sized server," they are probably in the wrong profession.
The final bit is this. The vBlock provides a couple things:
- A standardized, compliant "plug and play" architecture for virtualized environments
- The ability to free up valuable time to work on areas that provide true business value.
- A decent building block for private clouds, alongside software to (supposedly) streamline the administration of the cloud architecture and increase the number of systems a given admin can support.
Monday, January 25, 2010
XIV Disk Fault Questions [Updated]
UPDATE [8/6/2010]: If you're interested in more current XIV information, I recommend reading Tony Pearson's recent posts here and here. He also provided additional information in the comments to one of my posts here.
--
Today, I came across an XIV RAID-X post by IBMer KT Stevenson: RAID in the 21st Century. It is a good overview of the XIV disk layout/RAID algorithm. I have limited my questions in this post to ones raised by KT’s post since this post is already a bit lengthy.
The following is based off of what I've read online... allocations are divided up into 1 MB partitions, which are the distributed across the frame. For the purpose of this question, I will assume 100% of all disks are available for distribution (which is untrue, but it is the absolute best case scenario) and that the data is perfectly evenly distributed.
In a fully loaded XIV frame, there are 180 physical disks. What I'm interested in is the number of chunks that can be mirrored among the 180 disks without repeating the pair – once a ‘unique’ pair is repeated, you are vulnerable to a double disk failure with every allocation past that point. So, 180 C 2 = 16110. With 1MB per chunk, that is 16 GB. From an array perspective, you run out of uniqueness after 16GB of utilization. From an allocation perspective, any allocation larger than 16GB would be impacted by a double disk fault. I assume XIV doesn’t “double up” on 1MB allocations (going for a less wide stripe for reducing the chances of a double fault) simply because I've always heard that hotspots aren't an issue. This is best case assuming a perfect distribution, as near as I can reason - I'm sure any XIVers out there will correct this if I'm making an invalid assumption.
If you look closer, though, it’s a little worse than that. Every single disk is not a candidate as a mirror target – as noted above, XIV does not mirror data within a module. With 15 modules in a 180 disk system, that means for each mirror position there are 11 disks that can not be used. The math gets beyond me at this point, so if anyone wants to comment on what that actually equates to, I’d be interested.
I realize that the math part of this post is a little hard to understand, especially with 180 spindles in play, so I went ahead and drew it out with only 5 spindles (5 C 2 = 10).
This shows the 5 spindle example half utilized. In this diagram it is possible to lose 2 spindles without data loss... for example, you can lose spindle 2 and spindle 4 - since neither of them have both copies of a mirror, no data is lost.
This shows the 5 spindle example with all unique positions utilized. In this diagram, it is impossible to lose 2 spindles without losing both sides of one of the mirrors.
The 16GB number quoted above is based off of a 1MB chunk size, which is what has been documented online. If the chunk size was larger, then that amount would be higher before guaranteed loss. Of course, if you lose the wrong two drives prior to 16GB, you'll still lose data. The percentage chance of data loss increases as you get closer to 16GB.
I know KT is working on a response to this, I'm looking forward to being shown where the logic above is faulty (or where my assumptions went south).
--
Today, I came across an XIV RAID-X post by IBMer KT Stevenson: RAID in the 21st Century. It is a good overview of the XIV disk layout/RAID algorithm. I have limited my questions in this post to ones raised by KT’s post since this post is already a bit lengthy.
In fact, DS8000 Intelligent Write Caching makes RAID-6 arrays on the DS8000 perform almost as well as pre-Intelligent Write Caching RAID-5 arrays.Any array that does caching for all incoming writes should be able to claim the same (from a host perspective). For large writes where the entire stripe is resident in memory for parity computations, there should be almost NO performance degradation. It is great that the DS8000 performs well with RAID-6, but that is rapidly becoming “table stakes” if it isn’t already.
When data comes in to XIV, it is divided into 1 MB “partitions”. Each partition is allocated to a drive using a pseudo-random distribution algorithm. A duplicate copy of each partition is written to another drive with the requirement that the copy not reside within the same module as the original. This protects against a module failure. A global distribution table tracks the location of each partition and its associated duplicate.
[..]
The most common ways to mitigate the risk of data loss are to decrease the probability that a critical failure combination can occur, and/or decrease the window of time where there is insufficient redundancy to protect against a second failure. RAID-6 takes the former approach. RAID-10 and RAID-X take the combination approach. Both RAID-10 and RAID-X reduce the probability that a critical combination of failures will occur by keeping two copies of each bit of data. Unlike RAID-5, where the failure of any two drives in the array can cause data loss, RAID-10 and RAID-X require that a specific pair of drives fail. In the case of RAID-X, there is a much larger population of drives than a typical RAID-10 array can handle, so the probability of having the specific pair of failures required to lose data is even lower than RAID-10.While at first this paragraph made perfect sense to me, there was something that just didn't seem to sit right. Namely, this portion:
In the case of RAID-X, there is a much larger population of drives than a typical RAID-10 array can handle, so the probability of having the specific pair of failures required to lose data is even lower than RAID-10.
The following is based off of what I've read online... allocations are divided up into 1 MB partitions, which are the distributed across the frame. For the purpose of this question, I will assume 100% of all disks are available for distribution (which is untrue, but it is the absolute best case scenario) and that the data is perfectly evenly distributed.
In a fully loaded XIV frame, there are 180 physical disks. What I'm interested in is the number of chunks that can be mirrored among the 180 disks without repeating the pair – once a ‘unique’ pair is repeated, you are vulnerable to a double disk failure with every allocation past that point. So, 180 C 2 = 16110. With 1MB per chunk, that is 16 GB. From an array perspective, you run out of uniqueness after 16GB of utilization. From an allocation perspective, any allocation larger than 16GB would be impacted by a double disk fault. I assume XIV doesn’t “double up” on 1MB allocations (going for a less wide stripe for reducing the chances of a double fault) simply because I've always heard that hotspots aren't an issue. This is best case assuming a perfect distribution, as near as I can reason - I'm sure any XIVers out there will correct this if I'm making an invalid assumption.
If you look closer, though, it’s a little worse than that. Every single disk is not a candidate as a mirror target – as noted above, XIV does not mirror data within a module. With 15 modules in a 180 disk system, that means for each mirror position there are 11 disks that can not be used. The math gets beyond me at this point, so if anyone wants to comment on what that actually equates to, I’d be interested.
- What is the “blast radius” of a double drive fault with the two drives on different modules? Is it just the duplicate 1MB chunks that are shared between the two drives, or does it have broader impacts (at the array level)?
- At what size of allocation does a double drive fault guarantee data loss (computed as roughly 16GB above)?
- What is the impact of a read error during a rebuild of a faulted disk? How isolated is it?
- Does XIV varyoff the volumes that are affected by data loss incurred by a double drive issue, or is everything portrayed as “ok” until the invalid bits get read?
- If there is data loss due to a double drive issue, are there reports that can identify which volumes were affected?
I realize that the math part of this post is a little hard to understand, especially with 180 spindles in play, so I went ahead and drew it out with only 5 spindles (5 C 2 = 10).
This shows the 5 spindle example with all unique positions utilized. In this diagram, it is impossible to lose 2 spindles without losing both sides of one of the mirrors.
The 16GB number quoted above is based off of a 1MB chunk size, which is what has been documented online. If the chunk size was larger, then that amount would be higher before guaranteed loss. Of course, if you lose the wrong two drives prior to 16GB, you'll still lose data. The percentage chance of data loss increases as you get closer to 16GB.
I know KT is working on a response to this, I'm looking forward to being shown where the logic above is faulty (or where my assumptions went south).
Subscribe to:
Posts (Atom)