2nd International Workshop on The Lustre Ecosystem:
Enhancing Lustre Support for Diverse Workloads

Keynote Speaker - March 8, 2016 @ 8:45am

Brent_Gorda

Brent Gorda, General Manager for HPC Storage - High Performance Computing Group, Intel

HPC Storage Futures - a 5 year outlook

Abstract: Changes in storage hardware are coming and HPC is going to benefit in a major way. Intel and others are working on solid state solutions that will disrupt how we view the storage hierarchy and bring massive changes in both performance and capacity nearer the cpu. As the compute-centric world of HPC starts to care about energy use, data movement and efficiency, the storage community will see more of the limelight (read: resources) to help the overall system design. Lustre is still a critical technology for storage in this future, but it will be augmented with on-system hardware and a number of (object-oriented) software interfaces. The future of storage for HPC is about to undergo change for the better.

Biography: Brent Gorda is the General Manager for HPC Storage in the High Performance Computing Group within Intel. Brent co-founded and led Whamcloud, a startup focused on the Lustre technology which was acquired by Intel in 2012. A long time member of the HPC community, Brent was at the Lawrence Livermore National Laboratory and responsible for the BlueGene P/Q architectures as well as many of the large IB-based clusters in use at the Tri-Labs. Previously Brent ran the Future Technologies Group at the Lawrence Berkeley National Laboratory, and has a long history with the National Energy Research Scientific Computing Center (NERSC).

Keynote Speaker - March 9, 2016 @ 9:00am

Cory_Spitz

Cory Spitz - Cray, Lead Developer of Storage R&D

Making Lustre Data-Aware

Abstract: Technologies that will breathe new life into storage systems are coming soon, but it isn't clear how they can best be leveraged. In fact, it isn't entirely evident that Lustre will be the software technology that will best take advantage of these new technologies. There are several problems that need careful consideration. One basic problem is the hardware organization of new systems. It is obvious that simply dropping in solid-state components with the same approach to deployment and software isn't going to work. Concepts such as burst buffers, which typically distribute storage components across and throughout a system, are being tested, but the ideas and execution of such methods are far from perfected.

Other obstacles must also be taken into account. Perhaps more importantly, we also need to consider the new requirements of shared storage for diverse workloads. If our data is more broadly distributed across these components, we need to think about enabling the seamless movement of and access to that data. Power HPC users could make the investment to carefully marshal their data to optimize their own personal workflows. However, I doubt that we should expect technical computing users, data scientists, or others to make the same investment. Today, Lustre doesn't provide the framework, the tools, or the technology to easily access broadly distributed data. To meet these kinds of expectations, Lustre must move on from so-called scratch storage and become both data aware and data placement aware. Permanence and provenance of data will then be absolute requirements.

While it is not yet clear how we can begin to solve these problems in the Lustre ecosystem, it is evident that we must begin to find solutions. If we abandon Lustre altogether we risk starting over and setting ourselves further back. Our codes and workflows will likely be less portable as different providers and different vendors begin trying different emerging solutions. However, Lustre is battle tested and it will take new technology years to catch up. Consequently, a better path forward is to adopt solutions that have been successfully demonstrated, evolve Lustre to adopt the best solutions, and thus ready Lustre to meet the needs of the future. If Lustre is a (sledge) hammer we need to evolve it into a Swiss army knife (with a hammer).

Biography: Cory Spitz leads a development team in Cray's Storage R&D group. Cory is
a graduate of Michigan State University and has been with Cray Inc. for fifteen years, working with Lustre and storage for roughly half that time. He has been an advocate of Lustre and involved member of OpenSFS from its outset and is particularly involved in OpenSFS working groups, recently serving as the interim lead of the Lustre Working Group. Cory is working at Cray and through OpenSFS to create a strong Lustre development community that will contribute to successful Lustre community releases. Cory works in St. Paul and lives in Minneapolis with his wife and two boys.