Today, Microsoft is saying the general public preview of Azure Managed Lustre, a brand new addition to the storage choices in our Azure HPC + AI resolution. Lustre is an open-source parallel file system famend for high-performance computing (HPC) and is adept at large-scale cluster computing. Azure Managed Lustre offers the high-performance storage of Lustre with the management and consistency of Microsoft’s cloud. As a consequence, prospects can concentrate on their enterprise targets, whether or not that’s constructing a fraud detection system primarily based on SAS analytics, decoding the human genome to create the subsequent breakthrough in medication, main the frontiers of vitality exploration with seismic information processing and sustainable vitality resolution, or predicting the local weather and climate patterns impacting human life on planet earth.
Lustre as an Azure Managed Service
With Lustre as a managed service, customers can now concentrate on constructing and working their HPC and AI functions with out managing a complicated parallel file system. HPC and AI workloads can seamlessly migrate to the cloud, retaining their compatibility with Lustre and defending present automation and platform investments developed when beforehand run on premises. Azure Managed Lustre permits a quick on-demand deployment of clusters serving world areas, assuaging advance planning whereas assembly the compliance and information residency necessities. With a palette of efficiency choices, Azure Managed Lustre delivers an elastic resolution the place customers can deploy unbiased and unique clusters with predictable efficiency eliminating the noisy-neighbor downside generally skilled within the on-premises shared infrastructures.
The public preview consists of two sturdy SSD-based SKUs which ship a alternative in efficiency choices for mission important workloads: 125 MB/s and 250 MB/s for each provisioned TiB of capability. During the preview, you possibly can create cluster sizes as much as 128 TiB by default, with an choice to scale as much as 768 TiB upon request. Azure Managed Lustre is constructed on the extremely sturdy Azure managed disks with locally-redundant storage consisting of three replicas so even when one or two replicas expertise points you continue to have tolerance in opposition to failures. Azure Managed Lustre delivers POSIX-compliant Lustre model 2.15 (LTS), which gives a number of efficiency enhancements. We centered on capabilities that allow customers to eat Azure Managed Lustre simply with out worrying about the best way to construct and deploy the Lustre purchasers on their compute VMs and containers. With the Azure Managed Lustre preview, prospects can select from:
- Use Azure HPC pictures prebuilt with Lustre shopper packages for Ubuntu 18.04 and 20.04 (or)
- Download Lustre shopper packages from packages.microsoft.com for Linux distros – Ubuntu 18.04, 20.04 & 22.04; RHEL 7.8 & RHEL 8.0
Data tiering utilizing hierarchical storage administration permits customers to import and export information between Azure Managed Lustre and Azure Blob. This functionality permits customers to outline information archival, retention and safety primarily based on pre-defined insurance policies. Users of enormous information units can import the info which is scorching and related for his or her energetic information processing in Azure Managed Lustre clusters and archive/retain the remaining in Azure Blob. This permits them to maintain their run-time prices low. Additionally, information tiering to Azure Blob permits the customers to leverage its world presence and availability to instantiate Azure Managed Lustre clusters in a number of areas on demand. Integration with Azure Blob moreover facilitates multiprotocol information entry through NFS, HDFS and REST. Modern containerized functions in AI/machine studying (ML) and analytics can now run on Azure Kubernetes Service leveraging the CSI driver for Azure Managed Lustre. For instance, analytics functions constructed on high of SAS Viya can seamlessly leverage the excessive per-core storage efficiency that they require by integrating with the Azure Managed Lustre CSI driver.
Purpose-built for HPC and AI workloads
We are grateful to our personal preview prospects who helped us construct and excellent the product whereas addressing their key wants. We are dedicated to constructing and supporting Azure Managed Lustre because it helps you together with your journey of working your high-performance functions in Azure.
“We work with customers who have many different types of HPC workloads, and we frequently hit storage performance constraints because their projects can involve anything from thousands to tens of thousands of files, all of which need to be read and written in parallel. When we encounter storage bottlenecks or throttling errors on our HPC platforms, we know it is time to turn to Azure Managed Lustre. With Azure Managed Lustre we can add extremely performant storage that can readily keep up with the HPC compute, and automatically sync input and results data with persistent blob storage. Because it can be started and stopped on demand, within a few minutes, Azure Managed Lustre is quite cost efficient and pays for itself by giving us the ability to scale up or scale out our HPC compute, generating results much faster.”—Felipe Ayora, Director—Research and Advanced Computing, BizData.
“The new Azure Managed Lustre file system is a real game changer. It accelerates HPC deployment of our software and eliminates the need to run persistent storage servers. Setup is simple, and parallel i/o performance is fast enough to support very high-throughput workloads.”—Bill Shea, CEO Sharp Reflections.
Next Steps
To trial Azure Managed Lustre at no cost, full the registration kind. Learn extra about the best way to use Azure Managed Lustre and its varied supported options from our documentation.
Learn extra about HPC and AI options