Cray has announced a new framework designed for ‘big data’ that gives Cray customers the ability to implement and run Apache Hadoop easily on their Cray XC30 supercomputers.
The Cray Framework for Hadoop package includes documented best practices and performance enhancements designed to optimise Hadoop for the Cray XC30 line of supercomputers. It is aimed at giving users the utility of the Java-based MapReduce Hadoop programming model on the Cray XC30 system, complementing, HPC-optimized languages and tools of the Cray Programming Environment.
The initial release of the Cray Framework for Hadoop and an optimised Cray Performance Pack for Hadoop will be available as free downloads. They include validated and documented Apache Hadoop configurations. This performance pack includes Lustre-Aware Shuffle to optimise Hadoop performance on the Cray XC30 supercomputer.
Further enhancements to the performance pack, which will include a native Lustre file system library and a plug-in to further accelerate Hadoop performance using the Aries system interconnect, will be available in the first half of 2014.
