This website uses cookies so that we can provide you with the best user experience possible. Cookie information is stored in your browser and performs functions such as recognising you when you return to our website and helping our team to understand which sections of the website you find most interesting and useful.
Related Category Blog
- Leveraging Artificial Intelligence and Machine Learning in Cloud Solutions
- Leveraging Cloud Analytics for Sales Forecasting and Decision Making
- Unleash the Potential of Cloud Modernization via Containerization with AWS & Redington
- Redington’s object storage-based network share mounting solution on AWS Cloud
- Overcoming issues organizations face with the hybrid data storage model
Archives by Month:
- October 2024
- September 2024
- April 2024
- December 2023
- November 2023
- August 2023
- June 2021
- April 2021
- August 2020
- February 2020
- August 2019
- June 2019
- May 2019
- March 2019
- February 2019
- January 2019
- December 2018
- November 2018
- June 2018
- May 2018
- April 2018
- February 2018
- November 2017
- September 2017
- August 2017
Categories
Redington’s solution on persistent storage for high-performance workloads using Amazon FSx for Lustre
Challenges
Let’s start with a high-level overview of some of the problems that many firms have in a typical on-premises high-performance Lustre-based persistent files system solution or even any Hybrid Cloud services:
- High data backup cost
- Complex backup solution
- CAPEX based storage solution
Solution overview
Before we go into the solution overview, let’s define what a Lustre file system is and how it’s used in different technologies. A Lustre file system is an open-source parallel file system that covers many of the criteria of leadership class HPC simulation environments. This is mostly utilized for workloads on HPC clusters. Amazon FSx for Lustre is a fully managed storage service for compute workloads that is cost-effective, high-performance, and scalable.
Workloads that utilize Lustre storage were listed below:
- Machine learning
- High-performance computing (HPC)
- Video rendering and simulations
We’ll use an Amazon Elastic Compute Cloud (Amazon EC2) instance to run a storage-intensive job, which will be backed by Amazon FSx for Lustre (persistent type) as a high-performance, scalable storage solution. In the next parts, we’ll go through the architecture in detail as well as the deployment method. Aside from that, this storage solution may be used for a variety of use cases, including HPC workloads, Persistent storage for Containers, and Data lakes on Amazon Simple Storage Service (Amazon S3).
Architecture
Deployment
- For a new setup, the complete networking components will be deployed in the first place as per the above architecture design
- Amazon Virtual Private Cloud (Amazon VPC) security group needs to be created which will be specified during the Amazon FSx for Lustre persistent deployment process
- The required storage space is configured from the specific size availability of either 1.2 TiB or increments of 2.4 TiB
- We will analyze the client storage processing requirement and based on this the performance tier will be selected where the details on it are specified below:
- Amazon FSx for Lustre persistent file system allows you to choose from three deployment options, PERSISTENT-50, PERSISTENT-100, and PERSISTENT-200. Each of these deployment options comes with 50-MB/s, 100-MB/s, or 200- MB/s baseline disk throughput per TiB of file system storage, respectively
- Configuring LZ4 compression if required
- Post this the file system can be mounted on the Linux instances which are running the storage-intensive rendering systems with the help of Amazon FSx for Lustre client
Benefits
Better redundancy
Even if a file server (Amazon FSx for Lustre) becomes unavailable on a persistent file system, it is replaced automatically within minutes of failure.
Ease of accessibility
Amazon FSx for Lustre provides a centralized user interface on the AWS console for complete operational tasks and even the operations can be quickly carried out from the AWS Command Line Interface (AWS CLI).
Cost Optimization
A wide range of storage options and features were offered from Amazon FSx for Lustre to optimize the price and performance. Storage on Solid-State Drives (SSDs) is optimized for latency-sensitive applications, whereas storage on Hard Disk Drives (HDDs) is intended for throughput-focused workloads.
Security
Amazon FSx for Lustre is highly secured because of its ability to encrypt data at rest using customer managed Customer Master Key (CMK) with the persistent file system. Even if there is no CMK in place, AWS Managed CMK can be used for encryption at rest.
Redington can help mitigate the challenges of any data storage model. Our cloud experts help design solution that is unique to your requirements. Get in touch with us to know more about our services and how we can help you.