Storage Lessons from HPC: Extreme Scale Computing Driving Economical Storage Solutions into Your IT Environments

High Performance Computing (HPC) at the extremes pushes technology development, and the technology and concepts produced have and continue to trickle down into economical commercial IT products for decades. Products like Lustre and GPFS owe their existence to the HPC community. This talk will give a background on the current challenges faced by extreme HPC storage, will describe several products that have come from the HPC storage lessons learned, and will introduce a current development effort called MarFS, an R&D100 finalist storage technology that leverages cloud scale storage systems to provide inexpensive massive scale storage for legacy storage needs.

Massively Scalable Near POSIX Name Space over Cloud Style Erasure based Storage
Extreme Scale Data Management using Storage Tiers
Dealing with trillions of files in a storage system to billions of files in a single folder
Handling single files that span from 1 byte to 10s of petabytes in size
Multi-tiered erasure to provide protection for extremely large failure domains

Gary Grider is the Leader of the High Performance Computing (HPC) Division at Los Alamos National Laboratory. As Division Leader, Gary is responsible for all aspects of High Performance Computing technologies and deployment at Los Alamos. Additionally, Gary is responsible for managing the R&D portfolio for keeping the new technology pipeline full to provide solutions to problems in the Lab’s HPC environment, through funding of university and industry partners. Gary is also the US Department of Energy Exascale Storage, IO, and Data Management National Co-Coordinator. In this role, Gary helps manage the US government investments in Data Management, Mass Storage, and IO. Gary has 30 active patents/applications in the data storage area and has been working in HPC and HPC related storage since 1984. Gary is known recently for inventing the Burst Buffer concept for HPC, co-invented the parallel file system, and showed the economic need for a campaign storage tier for HPC centers.