The latest release of Lustre* software unleashes the performance and scalability of the Lustre parallel file system as a platform for organizations large and small. It allows businesses that need large-scale, high-bandwidth storage to tap into the power and scalability of Lustre—which can be supplemented by worldwide 24x7 technical support from the Lustre experts at Intel.
The open-source advantage Lustre provides standard interfaces that promote easy integration and interoperability with existing infrastructures, and because Lustre is open-source, IT can grow datacenter storage systems over time and on budget, employing a variety of networking and hardware options without vendor lock-in.
Intel Support for Lustre Software
Enterprise-grade technical support and services
Intel is the primary developer, maintainer, and technical support provider of Lustre software. As the recognized experts in the development of Lustre software, Intel has the technical expertise and experience you need to support even the most complex Lustre configurations.
Key Features of Storage Systems Built Using Lustre Storage Software
Optimized for performance at scale
Lustre software was designed to enable fully parallel I/O across clients, servers, and storage devices. Metadata and data are stored separately to allow each to be optimized yielding maximum performance.
- Massive data flows efficiently use a high percentage of the underlying storage and network bandwidth for low latency, high-throughput storage performance
- Improved innovation and precision as enterprises are able to run larger and more complex simulations faster and more easily
- Native Lustre client optimized for Intel® Xeon Phi™ coprocessors delivers data 10 times faster than a Network File System (NFS)—boosting application performance
Intel® Xeon Phi™ processor optimization
Lustre software provides support for the Intel® Xeon Phi™ Processor in both self-boot and leveraged-boot modes to provide compute nodes direct access to high-performance Lustre storage.
Most Scalable POSIX Filesystem
Lustre clients can read and write directly from storage servers, striping files across multiple targets. This allows both performance and volume to increase up as storage servers are added, and because Lustre provides extremely efficient hardware utilization, storage systems can scale up to thousands of storage nodes.
Scalable metadata performance
Remote filesystem directories can now be spread across multiple metadata servers to provide increased metadata performance as the number of metadata servers is increased.
Storage vendor neutral
Lustre is delivered as software-defined storage (SDS) meaning that it is completely vendor neutral. Existing storage clusters can be expanded without having to purchase new hardware from the original vendor, or trying to match existing HW specifications.
Broad OS support
Lustre software provides broad support for enterprise-class operating systems such as RedHat Enterprise Linux, SUSE Linux Enterprise Server, and Ubuntu Server.
Windows client support
Lustre software now provides a high availability framework for providing SMB file services to Windows clients. This framework allows Lustre to be integrated into Windows enterprise networks and provide consolidated storage and identity management between Windows and Linux client systems.
Kerberos authentication and encryption
Lustre software provides enhanced authentication and encryption by leveraging Kerberos to provide customers with secure authentication and encryption of information sent over the network.
Mandatory access controls with SELinux*
Lustre now supports the use of SELinux allowing file access control to be moved from user space to kernel space, and providing enforcement of client access to allowed files.
Lustre software provides modular interfaces for tiered storage, and intelligently manages the filesystem through policy driven data movement. This allows storage administrators to migrate “hot” and “cold” data between Lustre and value-optimized storage dynamically to achieve the ideal balance of performance and cost.
Integrated support for snapshots allow storage administrators to leverage this data protection feature to checkpoint filesystems. Snapshots can be mounted as a new namespace allowing access to previously modified or deleted files in the storage system.
Lustre and OpenZFS provide on-the-fly data compression maximizing available storage and increasing overall performance by reducing data I/O requests.
Clients can mount an individual directory, limiting clients from accidentally modifying or deleting files outside of the subdirectory.
Software defined networking
Lustre Networking (LNET) is abstracted from the physical hardware allowing Lustre to run on top of heterogeneous networks comprised of InfiniBand, Omni-Path, Ethernet, and other interconnects.
Optimized for Intel® Omni-Path Fabric
Intel® Omni-Path Architecture (Intel® OPA) delivers high performance fabric allowing scalability to tens of thousands of nodes and is built on enhanced IP and Intel® technology.
Dynamic network configuration
This powerful extension of the Lustre Networking (LNET) stack simplifies administrative tasks by allowing routes and parameters to be modified on the fly without requiring LNET to be stopped or restarted.
OpenZFS is a 128-bit file system allowing it to address 1.84 x 1019 time more data than 64-bit filesystems such as Btrfs. The actual size limitations of OpenZFS filesystems are so large that they are unlikely to ever be encountered in actual usage.
OpenZFS provides copy-on-write mechanism, meaning when data is overwritten a new copy is written to a separate block. In the case that power is lost before the operation completes, the existing metadata will still point to the old block minimizing the impact of data loss.
Hybrid Storage Pools
OpenZFS Hybrid Storage Pools allow SSDs to be used as an Adaptive Read Cache (L2ARC) for HDD-based storage. This cache can greatly improve overall read performance of HDD-based storage pools.
RAID-Z protection against write holes
Tradition RAID suffers from “Write Holes” - multiple disks can’t be updated atomically, causing RAID volumes to become damaged during a crash or power outage. OpenZFS writes are full stripes and leverage the Copy-on-Write mechanism, eliminating “Write Holes” and providing superior data protection when compared to traditional RAID.
Enhanced small file performance
Small file performance is being enhanced in Lustre by allowing small files to be stored directly on fast SSD-based metadata targets. This reduces RPCs and improves overall performance of both small files and larger sequential IO.
This feature allows administrators to set and enforce quotas based on individual projects providing more flexible quota use when compared with user/group quotas.
Parity declustered RAID
Parity declustered RAID for Lustre allows for a massive improvement in rebuild performance of an array, reducing the time it takes to recover from a drive fault. This improvement comes from the parallelization of rebuild operations and is especially impactful when using extremely large HDD.
Integrated storage encryption
Native data encryption for storage at rest is coming to Lustre when used with OpenZFS backed storage.
Integrated Storage Encryption will also provide the mechanism for secure delete by removing encryption keys from the system. This is more effective than purely overwriting data as modern filesystems and SSDs can obscure the blocks being written to, making other implementations of secure erase ineffective.
Progressive File Layouts
Progressive File Layouts in Lustre allow for file striping to expand from narrow to wide as the size grows. This enables maximum file performance to be maintained regardless of future file size.
Start Using Lustre Today
Lustre is Software-Defined Storage (SDS) capable of handling extremely large volumes of data shared concurrently across clustered servers. Storage powered by Lustre software is a breakthrough technology for addressing the Exascale and emerging high-performance data analytics challenge.
Now, with Lustre software and support from Intel, organizations have a complete solution to meet their storage and data requirements.