ParallelClusterMaker: An Open Source Toolkit for AWS ParallelCluster

ParallelClusterMaker is a powerful command line interface (CLI) toolkit designed to automate the creation and destruction of AWS ParallelCluster v3 stacks. This tool enables researchers and engineers to efficiently deploy High-Performance Computing (HPC) clusters on AWS without requiring extensive knowledge of infrastructure management.

Key Features

• Cloud-based HPC Clusters: Simplifies the process of setting up secure and scalable HPC environments suitable for various workloads, from machine learning to scientific computing.
• Multiple Use Cases:
  • Training ML Models at Scale: Supports multi-node distributed training using GPU instances and optimized inter-node communication with Elastic Fabric Adapter (EFA) and FSx for Lustre for effective data handling.
  • HPC and Scientific Computing: Ideal for tightly-coupled workloads such as Computational Fluid Dynamics (CFD), molecular dynamics, and genomics leveraging low-latency interconnects.
  • Cost-sensitive Batch Workloads: Perfect for jobs that can tolerate interruptions, allowing users to utilize spot instances effectively.
  • Research Flexibility: Enables on-demand cluster setups that can be easily brought down post-execution to minimize overhead costs.

Robust Functionality

• Compute Scheduling:

  • Integrates Slurm job scheduling with options for separate CPU and GPU queues.
  • Multi-instance-type and spot capacity configurations, optimizing cost and resource utilization.

• Shared Storage Solutions: Automatically provisions shared storage across clusters, including EBS, EFS, and FSx for Lustre, tailored to specific workload needs.

• Comprehensive Monitoring and Diagnostics: Utilizes tools like Grafana and Prometheus for performance monitoring. Health checks and diagnostic scripts provide detailed insights into cluster status, facilitating proactive management.

• Streamlined Operations: Includes a variety of operational scripts for managing cluster health, costs, and resource allocation, ensuring efficient cluster management without manual overhead.

Example Commands

Here's a glimpse of how easy it is to set up your HPC cluster:

# Create a basic cluster with defaults
./make_pcluster.py -N my-cluster -O owner -E owner@example.com -A us-east-1a

# Enable EFS with encryption
./make_pcluster.py -N secure-cluster -O user -E user@example.com -A us-east-1a --enable_efs=true --efs_encryption=true

# Fixed-size compute pool with specific configurations
./make_pcluster.py -N optimized-cluster -O user -E user@example.com -A eu-central-1a --initial_cpu_queue_size=4 --max_cpu_queue_size=10

Ease of Access and Management

With ParallelClusterMaker, accessing your HPC cluster is just as simple:

./access_cluster.py -N my-cluster

To stop or start your compute fleet without disrupting the head node, use:

./stop_pcluster.py -N my-cluster
./start_pcluster.py -N my-cluster

Conclusion

ParallelClusterMaker stands out as a versatile toolkit for cloud-based HPC management, making it an invaluable resource for scientific research and data analysis. Its user-friendly interface and comprehensive features cater to a wide array of computational tasks, maximizing efficiency in AWS environments.