What factors make HPC possible?
In particular, there are four factors driving the use of HPC:
Processing power
Put simply, the bandwidth required to process huge volumes of data can’t be delivered by a single processor. Instead, in an HPC model, multiple processing centers work in parallel to deliver results. Recall that within this model:
- The collection of individual computers that are networked together is called a cluster.
- Each individual processing unit in the cluster is called a node.
- Each processor in a node will have multiple cores.
As an example, a cluster with 16 nodes with four cores each is a very small cluster, representing a total of 64 cores operating in parallel.
Most HPC use cases today involve thousands of cores working in parallel to complete designated processes in a shorter amount of time. Infrastructure-as-a-service (IaaS) providers offer users the ability to leverage large numbers of nodes when required and then wind down the workload when the requirement is complete. Users only pay for the processing power required, without the capital expenditure (CAPEX) costs associated with building out infrastructure. With IaaS, users also typically have the ability to prescribe node layouts for specific applications, if required.
Operating system
Operating systems act as the interface between the hardware and software used in HPC. The two major operating systems used in HPC environments are Linux and Windows. Linux is generally used for HPC, whereas Windows is used only when Windows-specific applications are required.
Network
In HPC, the network connects the computing hardware, the required storage, and the user. The computing hardware is connected through networks that can handle a large bandwidth of data. The networks should also have a low latency to facilitate faster data transfers. Data transmissions and the management of clusters are handled by cluster managers, management services, or schedulers.
The cluster manager runs the workload among the distributed computational resources, such as CPUs, FPGAs, GPUs, and disk drives. All the resources have to be connected to the same network for the cluster manager to manage resources. When using the services of an IaaS provider, all of the facilities required to manage the infrastructure will be automatically applied by the provider.
Storage
Finally, the data to be processed by HPC has to be stored in a large data repository. Since that data can take different forms—structured, semistructured, and unstructured—different types of databases may be required to store the data.
Data in its raw format(s) is stored within a data lake. It can be difficult to process this data because it doesn’t have a purpose assigned to it yet. Data warehouses store data after processing once it’s been cleaned up to suit its specific purpose.
Storage: The missing link in HPC
In many HPC use cases, storage—a critical part of the architecture—is often overlooked. HPC is used when a vast quantity of data has to be processed in parallel, yet its performance depends on whether all of the components in its architecture can keep up with each other.
Traditional legacy storage solutions may not be able to handle the needs of HPC, creating bottlenecks in the process and potentially hampering its performance. Data storage must be able to keep up with the speed of the processing power of the setup, which is why many HPC architectures make use of unified fast file and object (UFFO) storage.
Evergreen//One™ offers fast and reliable UFFO storage with the convenience of the pay-as-you-go (PaYG) model. It can be used in on-premises and hybrid-cloud models and is ideal for HPC environments, which require the ability to scale operations without compromising performance.
Get started with Evergreen//One today. New customers get the first three months of service free.
