A round-up of the latest storage products and technology available for scientists using HPC as a tool for their research
The storage market is in a unique position, in that there is demand from both the traditional HPC and enterprise storage markets, such as media and entertainment, alongside new growing markets for AI and machine learning. This has created huge potential to increase market share for storage vendors, as long as they can deliver the necessary performance required by HPC and AI users.
While storage volumes continue to increase dramatically, storage providers are trying to meet demand by increasing storage performance, while introducing more efficient methods of managing data across large multi-petabyte storage platforms.
Choosing the right system for a particular workflow is critical to getting the most out of storage technology. The traditional products associated with parallel file systems still persist, but now there is increasing competition from cloud and all-flash storage arrays, which are becoming increasingly attractive to users at opposite ends of the hardware spectrum.
_______
Featured product: Quantum Corporation
Addressing massive data growth challenges in life sciences with a smart archive
The pace of innovation in life sciences has resulted in enormous data growth, often overwhelming the storage systems required to support it. As genomics moves from the laboratory into precision medicine, the research data is too vital to be archived on inefficient, unsecure storage solutions.
HPC environments commonly produce extreme volumes of unstructured data, which need to be stored in durable, forever archives, enabling researchers and engineers to move seamlessly from raw data to analysis to actionable insights.
For unstructured data archives to fulfil their promise and help accelerate research, they need to get smart.
Quantum, the leader in unstructured data storage and management, offers the easiest path to a ‘smart archive’ with ATFS and ActiveScale, and multi-tiered storage offerings that are optimised to meet the data-intensive requirements of today’s genomics workflows – and to preserve and protect life science data for decades.
Find out more: https://www.quantum.com/en/solutions/build-smart-archives/
Try ATFS free for 30 days
https://www.quantum.com/en/products/file-system/atfs/trial/
More in store
Atipa’s Capella storage solutions are scalable, affordable, reliable and high performance. The Capella LZE series storage solution is its flagship Lustre appliance, including high-availability and advanced monitoring features. The starting configuration consists of one Lustre metadata/management module and one Lustre object storage module.
BullSequana Xstor is a single family of storage appliances from Atos. With machine learning and data analytics creating many of the most challenging HPC workloads in the move towards the exascale era, Atos participates in initiatives to strengthen industrial control of digital simulation and big data technologies.
BeeGFS is a parallel file system, developed and optimised for high-performance computing. It includes a distributed metadata architecture for scalability and flexibility, and is available free for end-users. For enterprise systems, professional commercial support is available, usually in co-operation with international turn-key solution partners.
Caringo’s Swarm software-defined object storage provides a complete platform for data protection, management, organisation and search at massive scale for all unstructured data (e.g., photos, videos and files). Users no longer need to migrate data to separate solutions for long-term preservation, delivery and analysis. Consolidate all files in Swarm, then easily find the data.
ClusterVision has 16 years of knowledge and expertise working with different storage solutions, with straight-forward to very complex installations. It can offer classic NAS systems, (hybrid) cloud solutions, parallel HPC storage installations with BeeGFS, IBM Spectrum Scale or Lustre, and many more configurations.
DDN is one of the premier providers of high-performance storage solutions, voted top in high-performance storage for AI and HPC computing in 2020 (HPCwire Readers’ and Editors’ Choice Awards). With DDN A3I and EXAScaler data management solutions, organisations can analyse petascale datasets, with capacities up to 500+ PB, and throughput up to 2TB/second.
Excelero’s NVMesh software-defined storage deploys distributed data protection over public cloud instances with ephemeral NVMe drives, creating a unique, virtual, high-performance and low-latency storage pool. Excelero delivers low-latency distributed block storage for web-scale applications. NVMesh enables shared NVMe across any network and supports any local or distributed file system.
HPC using Fungible solutions eliminates lengthy delays and lost output typically related to legacy HPC clusters. Elastic, high-performance configurations that support practically unrestricted scalability allow you to grow and contract infrastructure as applications require.
HPE provides HPC storage solutions that span the whole storage hierarchy to accelerate time-to-insights, while managing and protecting valuable data in parallel file systems in a cost-effective way. Its Cray ClusterStor E1000 storage system embeds Lustre and is ideal for attachment with HPE Slingshot, InfiniBand HDR and 100/200 GbE to HPC Cray EX supercomputers, and large clusters of HPE Apollo systems.
IBM Spectrum Scale is a parallel high-performance solution with global file and object data access for managing data at scale, and can perform archive and analytics in place.
MemoScale specialises at developing compression and erasure coding software for data storage solutions. It has provided technical solutions and has extensive experience in software and hardware optimisations.
MooseFS allows the combination of data storage and data processing in a single unit using commodity hardware, providing an extremely high ROI. Through this approach, it provides professional services and expert advisory for storage solutions.
NEC’s Storage HS Series platform enables long-term data retention through scalability of performance and maximised capacity without the complexity and inherent limitations of legacy storage solutions.
NetApp has a proven track record when it comes to modernising and simplifying storage environment. From simple, smart, trusted storage for shared NAS and SAN environments, to arrays built for dedicated, high-bandwidth applications like data analytics and disk-based backup, it can design the perfect solution.
Open-E DSS V7 is a fully featured NAS and SAN software platform. NAS offers support for multiple file protocols which enable you to build cross-platform solutions and SAN offers iSCSI Target and Fibre Channel Target functionalities.
The flexible design and excellent scalability of Open-E DSS V7 enables companies and institutions of any size to create effective storage solutions to meet and adapt to the simplest or most complex data management needs.
Panasas supports industry and research innovation around the world with PanFS Dynamic Data Acceleration that automatically adapts to evolving workloads to deliver a consistently fast, total-performance HPC storage solution. PanFS is delivered on the ActiveStor Ultra turnkey appliance, to maximise simplicity, boost reliability and provide the lowest TCO.
Pure Storage aims to empower innovators by simplifying how people consume and interact with data. FlashArray//C delivers NVMe performance, hyper-consolidation, and simplified management. FlashArray//X is designed for entry-level to enterprise applications.
QCT HPC/DL solution integrates with open source software to offer a software stack that includes resource management for a fully end-to-end integrated HPC/DL solution. It also features a modularised architecture that can be easily tailored to meet customer demands, simplifying the implementation journey and accelerating time-to-market.
Qualstar magnetic tape and data storage solutions are known for their high quality, ease of use and reliability. Tape has long been used as a digital data storage medium, but it is the rapid expansion of big data in recent years that has really made its value apparent.
Qumulo Core is a high-performance file data platform designed to help you store, manage and build workflows and applications with data in its native file form, at massive scale, across on-prem and cloud environments.
Quobyte is a software storage system that delivers unlimited performance and consistent low latency to applications and users through scale-out, without limits. Quobyte is designed around the core principle of unconditional simplicity, so anyone can easily manage hundreds of petabytes with many applications and users.
Scality helps scientists manage file and object data from cloud-to-core-to-edge. Scality Ring is a scale-out file and object storage, while Scality Artesca provides lightweight, cloud-native object storage.
Consolidate and manage massive datasets with ease using high-throughput, scalable storage architecture for mass capacity. Built for rapid throughput, Seagate storage arrays affordably scale out petabytes of storage to meet data-intensive HPC workloads, while maximising capacity over time.
SoftIron’s HyperDrive Performance+ leverages the AMD EPYC 3000 processor, the Performance+ family debuts the firm’s first x86 CPU and NVMe SSD-based designs in its HyperDrive series.
Spectra Logic deliver disk, object storage, tape and data management software for archive, backup and cloud. Their storage solutions give cost-effective storage to meet performance, growth and environmental needs.
Suse offers an adaptable Linux operating system and the only open Kubernetes management platform.
Weka offers a modern storage architecture that can handle demanding I/O intensive workloads and latency-sensitive applications at Exabyte scale to help firms improve business outcomes wherever their data resides – on-premises, the cloud or across both.