Pages

Tuesday, March 16, 2010

Breaking Datacenter Boundaries

Chuck Hollis (EMC) has an interesting post up regarding the future of workload optimization and fluid architectures.  First off, he has one of the clearest definitions of cloud architecture and private clouds I've seen recently:
"What makes a cloud a cloud is three things: technology (dynamic pools of virtual resources), operations (low-touch and zero-touch service delivery) and consumption (convenient consumption models, including pay as you go)... What makes a cloud "private" is that IT exerts control over the resources and associated service delivery."
Let's take a look at today's dynamic datacenter, especially in an organization where private cloud is being pursued.

  1. You have a very high virtualization rate.  Due to less friction for resource acquisition, you can assume that more and more systems will become virtualized on the private cloud as time goes on.
  2. You have a variable cost model, allowing for changing costs based off of actual consumption and performance utilization.
  3. You have an automation engine, to drive processes/systems through the private cloud.
  4. Regardless of technology, you're hopefully pursuing loosely coupled systems that do not have low latency requirements and provide rich web interfaces.
From a technology play, you have at least most of these in play:
  1. VMware - VMs are moving among hosts based on dynamic workload decisions - "where" something is running becomes less important.
  2. Intelligent Storage Optimization - placing the right data in the right place without sacrificing performance.
  3. Replication - ensuring production data is recoverable in a remote location.
Virtualization allows IT organizations to break down silos and drive utilization up while controlling costs.  Most large organizations maintain several data centers, and resources are not easily shared between them.  That's the next silo to be knocked down... by leveraging the investment in virtualization and storage technologies, it could be possible in the near future.
  1. You have extremely high visibility into utilization, data traffic, response times, frequency... basically, what drives the physical location of a VM.  The main reason not to move a workload to a different data center typically has to do with latency between users and the application layer, or between the application layer and the data.  By hooking into the hypervisor, you could determine likely candidates that can be moved without massively disrupting the user experience.
  2. The "heavy lifting" of migrating large portions of production data is already taken care of.  You have an asynchronous mirror of the data at the remote site, probably hooked up to an existing VMware Cluster.  The remain "system state" information could be replicated with a brief outage at a predefined window and then promoted to production at the remote site (flipping the replication to maintain recoverability).
Given the end-to-end knowledge from #1, and the data proximity of #2, you can theoretically "warm" migrate a VM from one datacenter to another, keeping response times the same or better, and increase the flexibility of the environments.

So, in the end, it comes down to what percentage of applications are eligible for this type of workload distribution based on network and performance requirements.  By optimizing at that level, you can more evenly spread out your workload requirements geographically.  The notion of distributed cache coherence comes into play for applications that don't behave well in a higher latency location.  Finally, once you have that technology in place, disaster recovery becomes much simpler - instead of vMotioning between hosts, you vMotion to an alternate datacenter.

Sure, none of this is available right now... but looking forward, you can see how an entirely fluid, geographically dispersed IT infrastructure is possible.

Thursday, March 11, 2010

Odds and Ends - Tiering, and Performance Planning

A few articles I wanted to briefly highlight:
  • Storagebod posted a brief article on automated storage tiering.  To briefly summarize, imperfect automated storage tiering is better than nothing... it is an easy way to get value out of SSDs in an existing environment and it provides a mechanism to move less-used data off of FC drives and onto SATA drives.  One thing is certain... the importance of manual data layouts is decreasing.  Between array architectures that don't 'allow' it (XIV being the most notorious example), don't 'need' it (NetApp FAS), and traditional architectures getting performance-driven automated storage tiering, using Excel to mismanage storage layouts could finally be over.  Dimitris makes the point that due diligence still needs to be applied to allocations that require high performance a few times a month to ensure the volumes don't get migrated to the wrong tier (among other comments).  There are excellent comments on that post from EMC and NetApp discussing the two approaches.
  • Dimitris also has a good post on vendor competition and under-sizing proposals to get the sale.  It is worth reading just for the 'basics' explanation of performance-sizing small arrays - it also has some good information on Compellant's architecture.  My comments regarding this vendor comparison are attached to that post.  As always, prior to storage acquisitions, make sure you understand how the vendor determined their bid's sizing and get guarantees on performance/capacity if you are at all concerned about meeting your requirements.
  • Chuck Hollis (EMC) and Marc Farley (3PAR) have excellent posts up on storage caching.

Monday, March 1, 2010

FAST & PAM Contrasted

** Updates Appended Below **

Over the past few days, I've been thinking about storage tiering... both in general, and specifically FAST and PAM II.  Each takes a very different approach to providing better storage performance without highly specific tuning.  This is an outsider's view based off of publicly available information (so, in cases where I'm wrong, both vendors have shown that they aren't shy in correcting misconceptions).  First, some general definitions:

FASTv1:  Released in December, it is the first version of EMC's Fully Automated Storage Tiering.  It works at the LUN level, and requires identical LUN sizes across tiers.  It is not compatible with Thin Provisioned/Virtual Provisioned LUNs.

FASTv2:  Scheduled to be released in the second half of 2010, it is the next version of FAST that works at a sub-LUN level.  It requires Thin Provisioning/Virtual Provisioning to manage the allocations since it utilizes that functionality to provide the granularity of migration.

PAM II:  NetApp's Flash solution, Performance Acceleration Module.  It acts as a additional layer of cache memory and does not have specific layout requirements.

Architecture Differences
FAST runs as a task on the processors of the DMX/VMAX.  At user specified windows, it will determine volumes/sub-volumes that would benefit from relocation and perform a migration to a different tier.  This requires some IO capacity to migrate the data, so offhours/weekends are ideal window candidates.  It does a semi-permanent relocation so all reads/writes are serviced by the new location post-migration (semi-permanent since FAST can relocate an allocation back to the prior tier if the performance data indicates it is a good swap).  Since RAID protection is maintained throughout the migration, the loss of components do not substantially affect response time.
PAM II is treated as an extremely large read cache.  Basically, as a given read-block is expired in memory, it trickles down to PAM until it is finally flushed and resides solely on disk.  This gives PAM II a few nice features.  First of all, there is no performance hit during the 'charging' of the PAM - since it is fed by expired 'tier-1' cache, there is no additional performance impact after the un-cached block is read.  Secondly, it does not cache any writes.  This is a giant assumption on my part, but I assume that due to the 'virtualization' WAFL provides, PAM does not need to track changed blocks on the disk.  Since everything is pointer based (think of NetApp snaps), when the track is changed on disk, future reads hit the new disk location then get migrated through the cache levels like 'new' reads since the location has changed (the old location/data gets expired fairly quickly).  The downside to this approach is that the loss of PAM requires all reads to be serviced by disk+tier 1 memory until it is replaced and recharged.

One thing that the NetApp resources on Twitter kept repeating was the benefits of PAM as an extension of cache.  I assume the main benefit of taking this approach to Flash is that it is accessed via memory routines (less layers/translation to execute through) rather than disk routines.  Whether or not this is a significant performance benefit, I really can't say.

From the initial implementation, PAM will provide almost immediate benefit as data expires from cache.  FAST will require a few iterations of the swap window before things have optimized.  Taking a longer view, FAST will work best with consistent workloads... after a few weeks, the migrations should hit an equilibrium and response times should be stable and fast.  Component failures should not adversely affect response time.  PAM, as an extension of cache, will continuously optimize whatever blocks are getting hit hardest at any given moment.  While this is more flexible day-to-day than data migrations, consistent performance could be an issue.  Additionally, the IO hit of losing PAM would decrease response times, but the impact of this is somewhat reduced by the fact that ramping up PAM is much faster than the data migrations that FAST requires.

Both solutions make various trade offs between performance, stability, and consistency.  Understanding these trade offs will benefit the customer as they choose which tools to leverage in their environment.  Following are a few considerations...

Considerations
  1. Many customers have performance testing environments.  Since both of these approaches optimize as tests run, what relationship can be drawn between the 3rd-5th week of integration/performance testing and the production implementation?  Theoretically, if the data is identical between performance testing and production, NetApp dedup could leverage performance testing optimizations during the production implementation.
  2. Can customers run both FASTv1 and FASTv2 simultaneously since they have mutually exclusive volume requirements?  Are both separately licensed?  There are implementations where LUN level optimization may be preferred over sub-LUN.
  3. NetApp can simulate the benefits of PAM II in an environment.  Can similar benefits be simulated for FAST prior to implementation?
  4. I assume that FAST will promote as much into SSD as possible to increase response time.  How can customers determine when to grow that tier of storage?
  5. If a customer is using PAM II to meet a performance requirement, what can they do to reduce the impact of a PAM II failure?
  6. For both FASTv2 and PAM II, how can a customer migrate to a new array while keeping the current performance intact?  With FASTv1, it is a simple LUN migration since it is determinable what tier a LUN is on.  With FASTv2 and PAM II, it gets tricky (please note, I'm not talking about migrating the data, which is a standard procedure, I'm talking about making sure you hit performance requirements post-migration).
** Updates - 03/02/2010 AM **
To be clear, this is an "apples to oranges" comparison.  Each solution takes a completely different approach to implementing flash into an array, and the two solutions behave very differently.

Additionally, since I was focusing on Flash in particular, I neglected to compare cache capacity directly.  DMX/VMAX has a much higher cache capacity than the NetApp arrays.  Per Storagezilla on Twitter:  "Symmetrix already has acres of globally accessible DRAM for read/write and doesn't need anything like PAM."

Finally, cost does play into comparing the two approaches, but I don't have access to any sort of real-world pricing.