- published: 19 Apr 2014
- views: 4592
Lustre is a type of parallel distributed file system, generally used for large-scale cluster computing. The name Lustre is a portmanteau word derived from Linux and cluster. Lustre file system software is available under the GNU General Public License (version 2 only) and provides high performance file systems for computer clusters ranging in size from small workgroup clusters to large-scale, multi-site clusters.
Because Lustre file systems have high performance capabilities and open licensing, it is often used in supercomputers. Since June 2005, it has consistently been used by at least half of the top ten, and more than 60 of the top 100 fastest supercomputers in the world, including the world's No. 2 and No. 3 ranked TOP500 supercomputers in 2014, Titan and Sequoia.
Lustre file systems are scalable and can be part of multiple computer clusters with tens of thousands of client nodes, tens of petabytes (PB) of storage on hundreds of servers, and more than a terabyte per second (TB/s) of aggregate I/O throughput. This makes Lustre file systems a popular choice for businesses with large data centers, including those in industries such as meteorology, simulation, oil and gas, life science, rich media, and finance.
In computing, a file system (or filesystem) is used to control how data is stored and retrieved. Without a file system, information placed in a storage area would be one large body of data with no way to tell where one piece of information stops and the next begins. By separating the data into individual pieces, and giving each piece a name, the information is easily separated and identified. Taking its name from the way paper-based information systems are named, each group of data is called a "file". The structure and logic rules used to manage the groups of information and their names is called a "file system".
There are many different kinds of file systems. Each one has different structure and logic, properties of speed, flexibility, security, size and more. Some file systems have been designed to be used for specific applications. For example, the ISO 9660 file system is designed specifically for optical discs.
File systems can be used on many different kinds of storage devices. Each storage device uses a different kind of media. The most common storage device in use today is a hard drive whose media is a disc that has been coated with a magnetic film. The film has ones and zeros 'written' on it sending electrical pulses to a magnetic "read-write" head. Other media that are used are magnetic tape, optical disc, and flash memory. In some cases, such as with tmpfs, the computer's main memory (RAM) is used to create a temporary file system for short-term use.
Lustre or Luster may refer to:
A system is a set of interacting or interdependent component parts forming a complex/intricate whole. Every system is delineated by its spatial and temporal boundaries, surrounded and influenced by its environment, described by its structure and purpose and expressed in its functioning.
The term system may also refer to a set of rules that governs structure and/or behavior. Alternatively, and usually in the context of complex social systems, the term is used to describe the set of rules that govern structure and/or behavior.
The term "system" comes from the Latin word systēma, in turn from Greek σύστημα systēma: "whole compounded of several parts or members, system", literary "composition".
According to Marshall McLuhan,
"System" means "something to look at". You must have a very high visual gradient to have systematization. In philosophy, before Descartes, there was no "system". Plato had no "system". Aristotle had no "system".
In the 19th century the French physicist Nicolas Léonard Sadi Carnot, who studied thermodynamics, pioneered the development of the concept of a "system" in the natural sciences. In 1824 he studied the system which he called the working substance (typically a body of water vapor) in steam engines, in regards to the system's ability to do work when heat is applied to it. The working substance could be put in contact with either a boiler, a cold reservoir (a stream of cold water), or a piston (to which the working body could do work by pushing on it). In 1850, the German physicist Rudolf Clausius generalized this picture to include the concept of the surroundings and began to use the term "working body" when referring to the system.
File or filing may refer to:
In this video from the OpenFabrics International Developer Workshop 2014, Doug Oucharek from Intel presents: Lustre File System. Learn more: https://www.openfabrics.org/index.php/press-room/2014-international-developer-workshop.html
LUG 2016: The 1 Million IOPS Lustre File System at TU Dresden Speakers: Michael Kluge, Technische Universität Dresden Johann Peyrard, Atos
In this video, Whamcloud's Andreas Dilger presents an introduction to the Lustre file system. Recorded at the HPC Advisory Council Switzerland Workshop 2011 in Lugano. Slides available at: http://www.hpcadvisorycouncil.com/events/2011/switzerland_workshop/agenda.php
Presented by Peter Braam, VP of Lustre at Sun Microsystems, this presentation updates the roadmap for Lustre. This presentation was part of the Sun HPC Consortium held Nov 10, 2007 in Reno, Nevada. More Consortium presentations are available at https://events-at-sun.com/hpcreno/presentations.html
Google Tech Talks June 23, 2007 ABSTRACT 2007 Google Seattle Conference on Scalability: Lustre File System Speaker: Peter Braam, Cluster File Systems, Inc. Lustre is a scalable open source Linux cluster file system that powers 6 of the top 10 computers in the world. It is resold by HP, SUN, Dell and many other OEM and storage companies, yet produced by a small powerful technology company, Cluster File Systems, Inc. This lecture will explain the Lustre architecture and then focus on how scalability was achieved. We will address many aspects of scalability mostly from the field and some from future requirements, from having 25,000 clients in the Red Storm computer to offering exabytes of storage....
In this video, Brian Behlendorf from Lawrence Livermore National Laboratory presents: Sequoia's 55PB Lustre+ZFS Filesystem. Recorded at LUG 2012 in Austin. Learn more at: http://www.opensfs.org/lug/program Keep up with daily supercomputing news: http://insidehpc.com
In this video from the 2014 Lustre Administrators and Developers Conference, Gabriele Paciucci from Intel describes how the company has enabled Hadoop users to maximize their performance using the Lustre File System. "When organizations operate both Lustre and Apache Hadoop* within a shared HPC infrastructure, there is a compelling use case for using Lustre as the file system for Hadoop analytics, as well as HPC storage. Intel Enterprise Edition for Lustre includes an Intel-developed adapter which allows users to run MapReduce* applications directly on Lustre. This optimizes the performance of MapReduce operations while delivering faster, more scalable, and easier to manage storage." Learn more: http://insidehpc.com/lad14-video-gallery/ and http://www.intel.com/content/www/us/en/software...
In this video, Sarp Oral from ORNL presents: A Technical Overview of the OLCF's Next-generation Center-wide Lustre File System. Recorded at LUG 2012 in Austin. Learn more at: http://www.opensfs.org/lug/program
In this video, Sun's Peter Bojanic presents on the roadmap for the Lustre file system. Recorded at the Sun HPC Consortium in Dresden, Germany on June 16, 2008. Sun Microsystems
In this video from the OpenFabrics International Developer Workshop 2014, Doug Oucharek from Intel presents: Lustre File System. Learn more: https://www.openfabrics.org/index.php/press-room/2014-international-developer-workshop.html
LUG 2016: The 1 Million IOPS Lustre File System at TU Dresden Speakers: Michael Kluge, Technische Universität Dresden Johann Peyrard, Atos
In this video, Whamcloud's Andreas Dilger presents an introduction to the Lustre file system. Recorded at the HPC Advisory Council Switzerland Workshop 2011 in Lugano. Slides available at: http://www.hpcadvisorycouncil.com/events/2011/switzerland_workshop/agenda.php
Presented by Peter Braam, VP of Lustre at Sun Microsystems, this presentation updates the roadmap for Lustre. This presentation was part of the Sun HPC Consortium held Nov 10, 2007 in Reno, Nevada. More Consortium presentations are available at https://events-at-sun.com/hpcreno/presentations.html
Google Tech Talks June 23, 2007 ABSTRACT 2007 Google Seattle Conference on Scalability: Lustre File System Speaker: Peter Braam, Cluster File Systems, Inc. Lustre is a scalable open source Linux cluster file system that powers 6 of the top 10 computers in the world. It is resold by HP, SUN, Dell and many other OEM and storage companies, yet produced by a small powerful technology company, Cluster File Systems, Inc. This lecture will explain the Lustre architecture and then focus on how scalability was achieved. We will address many aspects of scalability mostly from the field and some from future requirements, from having 25,000 clients in the Red Storm computer to offering exabytes of storage....
In this video, Brian Behlendorf from Lawrence Livermore National Laboratory presents: Sequoia's 55PB Lustre+ZFS Filesystem. Recorded at LUG 2012 in Austin. Learn more at: http://www.opensfs.org/lug/program Keep up with daily supercomputing news: http://insidehpc.com
In this video from the 2014 Lustre Administrators and Developers Conference, Gabriele Paciucci from Intel describes how the company has enabled Hadoop users to maximize their performance using the Lustre File System. "When organizations operate both Lustre and Apache Hadoop* within a shared HPC infrastructure, there is a compelling use case for using Lustre as the file system for Hadoop analytics, as well as HPC storage. Intel Enterprise Edition for Lustre includes an Intel-developed adapter which allows users to run MapReduce* applications directly on Lustre. This optimizes the performance of MapReduce operations while delivering faster, more scalable, and easier to manage storage." Learn more: http://insidehpc.com/lad14-video-gallery/ and http://www.intel.com/content/www/us/en/software...
In this video, Sarp Oral from ORNL presents: A Technical Overview of the OLCF's Next-generation Center-wide Lustre File System. Recorded at LUG 2012 in Austin. Learn more at: http://www.opensfs.org/lug/program
In this video, Sun's Peter Bojanic presents on the roadmap for the Lustre file system. Recorded at the Sun HPC Consortium in Dresden, Germany on June 16, 2008. Sun Microsystems
In this video from the LAD'14 Conference in Reims, Gabriele Paciucci from Intel presents: Current Status of the Adoption of ZFS as Backend File System for Lustre. Learn more: http://insidehpc.com/lad14-video-gallery/
In this video from LUG 2015 in Denver, Shuichi Ihara and Li Xi from DDN present: OSD-Btrfs - a Novel Lustre OSD based on Btrfs. "As the base file system, ldiskfs, now based on ext4, has been a crucial component of the Lustre file system, guaranteeing both high performance and stability. In Lustre 2.4, the Object Storage Device (OSD) layer has been restructured and a new OSD API, added on top of the OST/obdfilter layer, has been introduced. This change has made it relatively easy to add new backend file system and OpenZFS has recently been introduced as a new type of OSD. We have built an novel OSD prototype, termed OSD-Btrfs, that is based on Btrfs. Similar to ZFS, Btrfs is Copy on Write (COW) style next generation Linux file system with a number of interesting features, such as snapsho...
Google Tech Talks June 23, 2007 ABSTRACT 2007 Google Seattle Conference on Scalability: Lustre File System Speaker: Peter Braam, Cluster File Systems, Inc. Lustre is a scalable open source Linux cluster file system that powers 6 of the top 10 computers in the world. It is resold by HP, SUN, Dell and many other OEM and storage companies, yet produced by a small powerful technology company, Cluster File Systems, Inc. This lecture will explain the Lustre architecture and then focus on how scalability was achieved. We will address many aspects of scalability mostly from the field and some from future requirements, from having 25,000 clients in the Red Storm computer to offering exabytes of storage. Performance is an important focus and we will discuss how Lustr...
File input and output often become a severe bottleneck when parallel applications are scaled up to large numbers of processors. In this tutorial, I will outline the methods commonly used for parallel IO and discuss how well each of these is likely to perform on ARCHER. The work space on ARCHER uses the parallel Lustre file system, and I will explain how the design and configuration of Lustre affects IO performance. The tutorial will end with a brief overview of ways to measure the IO behaviour of your own applications, and what steps to take if the performance is not as good as expected. The main aim of the 45-minute presentation is to initiate online discussions between users and staff on parallel IO, or any other aspects of the ARCHER service. ARCHER Virtual Tutorial page: http://www.a...
Irby Thompson and Mathew Monroe: Advanced File System Hiding and Detection Abstract: The ability to both conceal and detect hidden data on the hard drive of a compromised computer represents an important arms-race between hackers and forensic analysts. While rootkits and other kernel manipulation tools make hiding on live systems fairly easy, the trick of hiding data from forensic tools and offline drive analysis is much more difficult. In this presentation, we will review traditional data hiding techniques, examine their strengths and weaknesses, and then explore more advanced methods of data hiding which go beyond the detection capabilities of current forensics tools. Further attention will be given to enabling transparent access to hidden file systems while also minimizing detection, e...
In this video from LUG 2015 in Denver, Gabriele Paciucci from Intel presents: ZFS as Backend File System for Lustre: the Current Status, How to Optimize, Where to Improve. "Lustre has supported ZFS as a backend filesystem since version 2.4. Lawrence Livermore National Lab (LLNL) is using this filesystem in production, and has for years. Many sites are following their lead. System integrators are actively evaluating inexpensive Lustre storage solution using ZFS and JBODs. Intel supports users, system integrators, and OEMs using ZFS with Intel Lustre. In this presentation, we summarize the results of proof-of-concept (PoC) on a variety of the ZFS configurations. We cover sequential and metadata performance, data Integrity, manageability, availability and reliability. The work identifies th...
Presented by Shane Canon, ORNL. Recorded at the Lustre User Group Meeting, April 28, 2008 in Sonoma, California. http://lustre.org Sun Microsystems