There are problems that come with handling, moving, storing, or processing large volumes of anything—and across totally unrelated domains, those problems can look highly similar. Consider the postal service: It takes a complex network of airplanes, ships, semi-trucks, machines, delivery vehicles, and people to get letters and packages where they belong. And at each step of the way, there are huge risks related to efficiency, cost, delay, and error.
For example, imagine using a semi-truck to deliver mail door-to-door and how much wasted space would be in the truck. Or imagine a fleet of small, electric mail delivery vehicles driving mail from Alaska to Florida rather than using a plane.
These are simplistic examples, but they help explain why hierarchical storage management (HSM) exists. Organizations with large data footprints face constant problems of efficiency, and a lot of effort goes into planning how to store, move, and process all that information. Hierarchical storage management (HSM) is the historical method of making sure organizations essentially don't use semi-trucks as delivery vehicles for their digital data.
In this article, we'll explore HSM, the problems it aims to solve, and some of the modern alternatives to it.
What Is Hierarchical Storage Management?
Hierarchical storage management, or HSM, is a process for managing digital data that aims to use storage media in the most economical way possible while minimizing inefficiencies in using the data.
Two key facts underlie HSM: First, different methods of storing digital media have different characteristics. Second, not all data is treated equally. To the first point, the most obvious difference between different digital storage media is cost. The fastest, most available, most versatile storage media tends to be the most expensive. And to the second point, some data gets used daily, while some data is used much less frequently.
Many companies' bottom lines depend on fast access to their most essential data. But it would be hugely inefficient to pay as much to store and access the data they use a fraction of the time at the same level of speed and availability.
Different organizations implement HSM differently—there's no single rule book for doing HSM. But any time an organization splits its data storage into at least two tiers, HSM is the process of setting the rules for what gets stored where, and how it gets moved around.
Benefits of Hierarchical Storage Management
Organizations can see a number of benefits from implementing hierarchical storage management. Cost savings are the most obvious benefit of HSM: By moving less urgent data to cheaper storage media, companies can sacrifice availability for cost. Performance is also generally improved by HSM principles. When applications that need to access data don't have to waste time sifting through old, outdated, or irrelevant data, they can provide better results, faster.
And although HSM may sound complicated, good rules of where different categories of data belong, and automatic application of them, lead to simplified data management. HSM also optimizes storage utilization because it automatically migrates data to the appropriate storage tier based on rules established by IT professionals.
How Does Hierarchical Storage Management Work?
HSM can consist of many layers of storage medium, but the core of it is that at one end is a high-performance tier and at the other end is a slower, less expensive tier. The high-performance tier has historically been made up of storage-class memory, enterprise-grade flash solid-state drives (SSDs), and high-performance HDDs. At the lower end are devices like optical disks and even tape storage.
The actual implementation of HSM policies is quite complicated, but HSM essentially works by identifying the frequency with which a file is accessed, and as time passes, the system automatically moves infrequently used files to slower and less expensive storage. IT teams write rules for the parameters that define when data gets moved, what data is exempt from these rules, and other clarifications. But HSM is generally an automated process that optimizes data access and data storage costs.
What Are the HSM Tiers?
The key to understanding HSM and the use of storage media like tape drives is that there was a time when the differences between cost, performance, and speed were extreme. Although those gaps have shrunk and are continually shrinking, there was a time when the difference in cost between optical disks and solid-state drives was substantial enough to justify complicated methods of sorting data so that you were never wasting money.
The lowest-volume, highest-performance tier of HSM is usually referred to as Tier 0. This is mission-critical data that cannot afford delays or disruptions in service. Tier 1 is often referred to as "hot data," data that is used continuously for day-to-day business operations and whose immediacy can be balanced against storage costs. Tier 2 is "warm data," where cost considerations are given substantial priority, and where data that doesn't get accessed frequently gets put. Last, Tier 3 usually refers to "cold data," or data that is rarely accessed or updated, if at all.
All-flash Alternatives to HSM Tiers
In a perfect world, a company could have fast, highly available storage at all tiers, with subtle demarcations between them. For decades, the dream has been all-flash data storage at the enterprise level. But until quite recently, it simply wasn't feasible. However, in the last few decades, the cost of all-flash storage at the enterprise level isn't just possible, it's being done regularly by companies as large as Meta.
Pure Storage is a notable champion of all-flash enterprise storage, and as early as 2012, developed flash storage solutions that could cover an enterprise network's Tier 0 needs. When FlashArray//C™ was released, Pure Storage could promise consistent single-millisecond latency for enterprise networks' business-critical workloads and data, with 99.9999% availability and non-disruptive upgrades.
That was groundbreaking in its own right, and then FlashArray//X™ and FlashArray//XL™ made it possible to run everything from massive databases to cloud-native applications on flash memory. Even with those advances, popular wisdom still believed that Tiers 2 and 3 would never make their way to flash memory due to cost.
But in 2023, Pure Storage released FlashArray//E™ and FlashBlade//E™, challenging low-end spinning disk and tapes. Designed for long-term retention, FlashArray//E provides 1PB-4PB of raw, unified file and block data storage capacity. FlashBlade//E can save companies tons of money when storing unstructured and object workloads.
Pure Storage is setting the example of all-flash alternatives to HSM tiers at organizations of any size. While we aren't eliminating HSM per se, we are radically reshaping the lines that separate the tiers of HSM.
Conclusion
Hierarchical storage management emerged as a necessary response to the vast differences in cost that existed between the fastest, most expensive, and slowest, most affordable forms of data storage. Technological limitations have led to an entire career field dedicated to automatically categorizing, moving, and optimizing data storage solutions.
HSM is still an important cost-saving process, but all-flash enterprise-scale solutions like the ones offered by Pure Storage are rapidly doing away with the strict delineations between them while they provide huge volumes of fast data at fractions of the cost they used to require.
