The Standard Performance Evaluation Corporation (SPEC) has released a new version of its storage solution benchmark. The software includes new features for AI and genomics workloads, expanded customisation, better scaling, and statistical visualisation.
SPEC is a non-profit organization that establishes, maintains and endorses standardised benchmarks and tools to evaluate performance and energy consumption for the newest generation of computing systems. Its members are comprised or more than 120 computer hardware and software vendors, educational institutions, research organisations, and government agencies
The SPECstorage Solution 2020 benchmark measures performance of an entire storage configuration as it interacts with application-based workloads. The latest version includes new workloads for artificial intelligence (AI) and genomics, expanded custom workload capabilities, massively better scaling, and a statistical visualisation mechanism for displaying benchmark results.
‘The new SPEC storage benchmark is a valuable tool to help IT managers make better purchasing, configuration and optimisation decisions based on actual applications and real-world scenarios,’ says Don Capps, chair of the SPEC storage subcommittee.
The new AI workload in the SPECstorage Solution 2020 benchmark represents AI Tensorflow image processing environments, with traces collected from systems running COCO, Resnet50 and CityScape datasets. The new genomics workload comes from commercial and research facilities that perform genetic analysis. Other workloads in the benchmark address video data acquisition (VDA), electronic design automation (EDA), and software build environments.
New capabilities in the benchmark enable users to customise existing workloads or create new ones and test them to gain insight into storage performance issues.
New internal IPC mechanisms in SPECstorage Solution 2020 significantly reduce the number of TCP ports and DNS lookups, improving scalability, reliability, and start-up and run-time performance. Scaling has been increased from 60,000 load-generating processes in previous versions to more than 4 million processes that can be distributed throughout the world. The new efficiency is especially significant for testing cloud-based storage solutions.
‘Among cloud providers, TCP ports are a scarce resource, with most virtual machines able to accommodate only a few thousand at best,’ says Capps. ‘The new benchmark never requires more than 1,000 TCP ports. This enables users to scale up to 4 million processes without running into cloud constraints.’
The new statistical visualisation mechanism within SPECstorage Solution 2020 allows users to extract runtime counters and connect them to a database for visualisation. This provides greater insight into the behaviour and operation details of the system under test.
The SPECstorage Solution 2020 benchmark is available for immediate download on the SPEC website for $2,000, with discounts for qualified research and academic institutions. Results from the new benchmark are not comparable to those from past SPEC storage benchmarks.
