Panasas, a storage provider has announced a collaboration with MLCommons, the consortium behind MLPerf, to create industry-wide benchmarks for machine learning (ML) storage. Panasas will work with MLCommons to help steer these benchmarks by establishing best practices for measuring ML storage performance and ultimately helping to develop the next generation storage systems for AI/ML.
MLCommons invited Panasas to attend the foundational meetings, after which Curtis Anderson, software architect at Panasas, was named co-chair. ‘It is an honour to be a co-chair of the MLPerf Storage working group, and I look forward to the meaningful progress that this team will accomplish,’ said Anderson. ‘AI/ML practitioners are doing some of the most revolutionary work today, and I am excited to help develop the benchmarks that will enable them to determine the storage systems they need to carry out their projects.’
David Kanter, founder and executive director of MLCommons comments: ‘The end goal of the MLPerf Storage working group is to create a storage benchmark for the full ML pipeline which is compatible with diverse software frameworks and hardware accelerators. I’d like to thank Panasas for contributing their extensive storage knowledge, and Curtis specifically for the leadership he is providing as a co-chair of this working group.’
Panasas builds industry-leading price/performance data storage solutions that drive innovation across a vast range of computing environments. The Panasas ActiveStor portfolio features the all-NVMe ActiveStor Flash system, which enables outstanding small and random file performance and enhanced support for AI/ML projects. All Panasas solutions are powered by the company’s flagship PanFS® parallel file system, a reliable and autonomic data engine that orchestrates networked servers into a single file system serving data to clients at up to hundreds of gigabytes per second.
MLCommons, the open and global engineering consortium dedicated to making machine learning better for everyone, promotes widespread ML adoption and democratisation through benchmarks, large-scale public datasets, and best practices. Panasas approached MLCommons to discuss the storage challenge in the ETL (extract, transform, and load) process and its impact on the overall performance of the ML pipeline. The discussion was timely, as MLCommons had been in the early stages of forming the MLPerf Storage working group to develop a storage benchmark that evaluates performance for ML workloads including data ingestion, training, and inference phases.
