Create an HPC-ready VM instance

Tightly coupled high performance computing (HPC) workloads often use the Message Passing Interface (MPI) to communicate between processes and Compute Engine instances. But building your own OS image that is tuned for optimal MPI performance requires systems expertise, Cloud de Confiance by S3NS knowledge, and extra time for maintenance. To quickly set up an optimized environment for your HPC workloads, use an HPC VM image.

An HPC VM image is based on Rocky Linux 8 or Rocky Linux 9, and is optimized for tightly coupled HPC workloads. It includes pre-configured kernel and network tuning parameters required to create compute instances that achieve optimal MPI performance on Cloud de Confiance by S3NS.

For best results, deploy the HPC VM image on a machine type from the compute-optimized machine series. These machine types are designed for tightly-coupled HPC applications. To use Cloud RDMA with H4D, you must use version 20250917 or later of the HPC VM Rocky 8 image.

You can create an HPC VM instance by using any of the methods available to create a compute instance. You can also use SchedMD's Slurm workload manager, to deploy a HPC cluster. Slurm support isn't yet available for HPC VM images based on Rocky Linux 9.

Before you begin

• If you haven't already, set up authentication. Authentication verifies your identity for access to Cloud de Confiance by S3NS services and APIs. To run code or samples from a local development environment, you can authenticate to Compute Engine by selecting one of the following options:

  Select the tab for how you plan to use the samples on this page:

  Console

  When you use the Cloud de Confiance console to access Cloud de Confiance by S3NS services and APIs, you don't need to set up authentication.

  gcloud

  1. Install the Google Cloud CLI, and then sign in to the gcloud CLI with your federated identity. After signing in, initialize the Google Cloud CLI by running the following command:

     gcloud init

  2. Set a default region and zone.

Benefits

The HPC VM image provides the following benefits:

1. Compute instances ready for HPC workloads out-of-the-box. There is no need to manually tune performance, manage compute instance reboots, or stay up to date with the latest Cloud de Confiance updates for tightly-coupled HPC workloads.
2. Networking optimizations for tightly-coupled workloads. Optimizations that reduce latency for small messages are included, which benefits applications that are heavily dependent on point-to-point and collective communications. If using the H4D machine series, the HPC VM Rocky 8 image contains the necessary Cloud RDMA drivers.
3. Compute optimizations for HPC workloads. Optimizations that reduce system jitter are included, which makes single-node high performance more predictable.
4. Consistent, reproducible performance. OS image standardization gives you consistent, reproducible application-level performance.
5. Improved application compatibility. Alignment with the node-level requirements of the Intel HPC platform specification enables a high degree of interoperability between systems.

HPC VM image features

The HPC VM image offers several features that are designed to optimize performance for high performance computing (HPC) workloads:

• Automatic updates disabled
• MPI collective tunings
• Pre-installed RPMs

Automatic updates disabled

Automatic updates can have a negative impact on the performance of HPC workloads. Automatic updates can be disabled when using HPC VM images by setting the google_disable_automatic_updates metadata entry to TRUE when creating a compute instance. How this metadata entry is set during instance creation depends on the tool that you use to create the instance.

For example, when using the gcloud compute instances create command to create a compute instance, provide the --metadata argument. For more information, see About VM metadata.

The google_disable_automatic_updates metadata entry is available as follows:

• HPC VM Rocky 9 images: all versions.
• HPC VM Rocky 8 images: only in images built on or after July 12, 2024. The build date (formatted as YYYYMMDD) is used as the version number at the end of image names. For example, an HPC VM Rocky 8 image built on the 21st of July, 2025 has the name hpc-rocky-linux-8-v20250721.

If you disable automatic updates on your H4D instances, then you should regularly run the dnf update command on the instance to keep the Cloud RDMA driver up to date.

Intel MPI Library

Google recommends that you use the google_install_intelmpi script to install the message passing interface (MPI) library before running MPI jobs on Cloud de Confiance by S3NS. For more information, see Use Intel MPI 2021.

Pre-installed RPMs

The HPC VM image comes with the following RPM packages pre-installed:

• daos-client
• gcc-gfortran
• gcc-toolset-12
• Lmod
• dkms
• htop
• hwloc
• hwloc-devel
• infiniband-diags
• kernel-devel
• kmod-idpf-irdma
• libfabric
• librdmacm-utils
• libibverbs-utils
• libXt
• ltrace
• nfs-utils
• numactl
• numactl-devel
• papi
• pciutils
• pdsh
• perf
• perftest
• rdma-core
• redhat-lsb-core (HPC VM Rocky 8 only)
• redhat-lsb-cxx (HPC VM Rocky 8 only)
• rsh
• screen
• strace
• wget
• zsh
• "Development Tools" package group

Quickstarts

These tutorials describe the steps to set up an HPC-optimized VM instance. The procedures describe how to:

• Create an HPC VM instance (without configuring Cloud RDMA)
• Specify a compact placement policy when creating HPC VM instances
• Create an HPC VM instance that uses Cloud RDMA

You can't add Cloud RDMA to an existing HPC VM instance, so follow the appropriate steps when creating your instance.

Before you begin

1. To use the Google Cloud CLI for this quickstart, you must first install and initialize the Google Cloud CLI:
   • Download and install the Google Cloud CLI.
   • Initialize the Google Cloud CLI.
2. In the Cloud de Confiance console, on the project selector page, select or create a Cloud de Confiance project.

Create an HPC VM instance

We strongly recommend using the HPC VM image with your HPC workloads using the following machine series:

The machine types in these series have fixed virtual-to-physical core mappings and expose NUMA cell architecture to the guest OS, both of which are critical for performance of tightly-coupled HPC applications.

If you are creating multiple, interconnected HPC VM instances, then follow the instructions at create HPC VM instances with compact placement policies to achieve low network latency.

gcloud compute instances create INSTANCE_NAME \
        --zone=ZONE \
        --image-family=IMAGE_FAMILY \
        --image-project=cloud-hpc-image-public \
        --maintenance-policy=TERMINATE \
        --machine-type=MACHINE_TYPE

gcloud compute instances describe INSTANCE_NAME

POST https://compute.googleapis.com/compute/v1/projects/PROJECT_ID/zones/ZONE/instances

{
   "machineType":"zones/ZONE/machineTypes/MACHINE_TYPE",
   "name":"VM_NAME",
   "disks":[
      {
         "initializeParams":{
            "sourceImage":"projects/cloud-hpc-image-public/global/images/IMAGE"
         },
         "boot":true
      }
   ],
   "networkInterfaces":[
      {
         "network":"global/networks/NETWORK_NAME"
      }
   ]
}

Create HPC VM instances with compact placement policies

You can reduce the latency between compute instances by creating a compact placement policy. A compact placement policy ensures that instances in the same availability zone are located close to each other.

If you need more compute instances than can fit in a single compact placement policy, then divide your instances into multiple placement policies. Use the minimum number of placement policies that can fit all your instances.

To create HPC VM instances that specify a compact placement policy, follow these steps:

1. Create a compact placement policy.

2. Do one of the following:

   • Apply the compact placement policy to an existing HPC VM instance.

   • Create an instance that specifies the compact placement policy.

     To create an instance that specifies a compact placement policy, use the --maintenance-policy and --resource-policies flags with the gcloud compute instances create command.

Create a HPC VM instance that uses Cloud RDMA

To create a HPC VM instance that uses Cloud RDMA, you must first create at least one regular Virtual Private Cloud (VPC) network and one Falcon VPC network. The Falcon VPC network uses an RDMA network profile that enables RDMA traffic between compute instances. This network is separate from the regular VPC network that carries non-RDMA traffic to other Cloud de Confiance services or the internet.

To create a HPC VM instance that uses Cloud RDMA, the tasks to complete are as follows:

1. Identify or create at least two VPC networks:

   • A regular VPC network for the traffic that goes through the gVNIC network interface
   • A Falcon VPC network for the RDMA network traffic

2. Create a HPC VM instance.

   1. For the source image, use a HPC VM image. This image includes the drivers needed for Cloud RDMA.

   2. During the instance creation, configure at least two network interfaces—one that uses the gVNIC driver and one that uses the IRDMA driver.

      For a more complete explanation, see Create an instance that uses Cloud RDMA.

3. If you plan to run MPI applications on the HPC VM instances that use Cloud RDMA, then following the MPI configuration steps in Set up and scale MPI applications on H4D VMs with Cloud RDMA.

Access the HPC VM instance

After you create the HPC VM instance, it starts automatically. To access the instance, do one of the following:

gcloud compute ssh INSTANCE_NAME --zone ZONE

Clean up

To avoid incurring charges to your Cloud de Confiance by S3NS account for the resources used in these quickstarts, delete any HPC VM instances and all attached resources that you created.

gcloud compute instances delete INSTANCE_NAMES \
        --delete-disks=DELETE_DISK_TYPE \
        --zone=ZONE

Configure your HPC VM instance according to best practices

To get better and more predictable performance for your HPC VM instance, we recommend that you use the following best practices.

Disable simultaneous multithreading

The HPC VM image enables simultaneous multithreading (SMT), also known as Hyper-Threading on Intel processors, by default. Disabling SMT can make your performance more predictable and can decrease job times.

You can use the following methods to disable SMT:

• To disable SMT while creating a new HPC VM instance, follow the steps to create an HPC VM instance and include the flag
  --threads-per-core=1.

• To disable SMT on an existing HPC VM instance, connect to the instance and run the following command from the guest OS:

  sudo google_mpi_tuning --nosmt
  

For more information, see Set the number of threads per core.

Configure gVNIC as the network interface type for C2 and C2D instances

The HPC VM image supports both Virtio-net and Google Virtual NIC (gVNIC) as virtual network interfaces. Using gVNIC instead of Virtio-net can improve the scalability of MPI applications by providing better communication performance and higher throughput. Additionally, gVNIC is a prerequisite for per VM Tier_1 networking performance, which provides higher bandwidth and allows for higher throughput.

If you create a new C2 or C2D instance, then, by default, Virtio-net is used for the virtual network interface. To use gVNIC, follow the steps to create a HPC VM instance and do one of the following:

The HPC VM image includes the gVNIC driver as a Dynamic Kernel Module Support (DKMS). For more information, see Using Google Virtual NIC.

Turn off Meltdown and Spectre mitigations

The HPC VM image enables the Meltdown and Spectre mitigations by default. In some cases, these mitigations might result in workload-specific performance degradation. To disable these mitigations and incur the associated security risks, do the following:

1. Run the following command on your HPC instance:

   sudo google_mpi_tuning --nomitigation
   

2. Reboot the instance.

Improve network performance

To improve the network performance of your instance, set up one or more of the following configurations:

• Configure a higher bandwidth. To configure Tier_1 networking for C2 or C2D compute instances, use the gcloud compute instances create command to create the instance. Specify the --network-performance-configs flag when creating the instance. For more information, see Create instances and containers that use Tier_1 networking.

• Use jumbo frames. To help minimize the processing overhead for network packets, we recommend using a larger packet size. You need to validate larger packet sizes for the specifics of your application. For information about the use of jumbo frames and packet sizes, see Maximum transmission unit guide.

• Increase the TCP memory limits. Higher bandwidth requires larger TCP memory. Follow the steps to increase tcp_*mem settings.

• Use the network-latency profile. Evaluate your application's latency and enable busy polling that reduces latency in the network receive path. Adjust the net.core.busy_poll and net.core.busy_read settings in /etc/sysctl.conf, or use tuned-adm.

Use Intel MPI 2021

Google recommends to use the Intel MPI 2021 library for running MPI jobs on Cloud de Confiance.

MPI implementations have many internal configuration parameters that can affect communication performance. These parameters are especially relevant for MPI Collective communication, which lets you specify algorithms and configuration parameters that can perform very differently in the Cloud de Confiance environment.

The HPC VM image includes a utility, google-hpc-compute, that installs the recommended MPI libraries and uses Cloud de Confiance tailored libfabric providers over the TCP transport.

Limitations

The benefits of tuning vary from application to application. In some cases, a particular tuning might have a negative effect on performance. Consider benchmarking your applications to find the most efficient or cost-effective configuration.

Use google-hpc-compute utility for Intel MPI 2021 support

The google_install_intelmpi script is the MPI related tool in the google-hpc-compute utility. It helps to install and configure Intel MPI.

The google-hpc-compute utility is included in the HPC VM image.

Install Intel MPI 2021

To install the Intel MPI library while creating a new HPC VM instance, follow the steps to create an HPC VM instance and include the following when creating the compute instance:

--metadata=google_install_intelmpi="--impi_2021"

To install the library on an existing HPC VM instance, run the following command on that compute instance:

sudo google_install_intelmpi --impi_2021 --install_dir=PATH_INSTALL_MPI

The default location for install_dir is set to /opt/intel.

Intel MPI Library and MPI collective tunings

The HPC VM image includes generic tunings which disable hyperthreading, optimize memory, modify system resource limits and apply custom tuned profiles. The google_hpc_firstrun script is designed to run automatically on the first boot of a HPC VM instance.

As a part of google-hpc-firstrun utility, the mpi-tuning script is run to improve the performance and latency of the HPC VM instance. You can run the mpi-tuning.sh script to apply tunings on individual compute instances.

Usage:
  Verify tuning steps: $ mpi_tuning OPTIONS --dryrun
  Apply tunings: $ mpi_tuning OPTIONS

Options:
  --hpcprofile       Install and apply google-hpc-compute tuned profile
                     Also applies: --tcpmem, --limits
  --hpcthroughput    Install and apply google-hpc-compute-throughput profile
                     Also applies: --tcpmem, --limits
  --tcpmem           Increase memory for TCP
  --limits           Change the system ulimits
  --nosmt            Disable simultaneous multi threading
  --nofirewalld      Disable firewalld
  --noselinux        Disable SE Linux (reboot required)
  --nomitigation     Disable CPU vulnerabilities mitigations (reboot required)
  --reboot           Reboot system after tunings if required
  --dryrun           Do not execute commands
  --verbose          Print verbose messages
  --help             Show help message

Create a custom image using the HPC VM image

To implement the best practices at scale, create a custom OS image to use with your HPC VM instances. Complete the following tasks to create a custom image to use when creating HPC VM instances:

1. Create a compute instance that uses the HPC VM image.

2. Customize the compute instance with MPI tunings.

3. Create a custom image using the boot disk of your HPC VM image as the source disk. You can do so using the Cloud de Confiance console or the Google Cloud CLI.

gcloud compute images create IMAGE_NAME \
         --source-disk=INSTANCE_NAME \
         --source-disk-zone=ZONE \
         --family=IMAGE_FAMILY \
         --storage-location=LOCATION

Pricing

The HPC VM image is available at no additional cost. Because the HPC VM image runs on Compute Engine, you might incur charges for Compute Engine resources such as vCPUs, disks, and memory.

To learn more, see Compute Engine pricing.

What's next

• Review Best practices for running HPC workloads.
• Learn how to Set up and scale MPI applications on H4D VMs with Cloud RDMA on Cloud de Confiance by S3NS.
• Learn more about the Compute-optimized machine family.
• Learn how to create H4D HPC clusters with enhanced cluster management capabilities.
• If you have feedback or require support, email hpc-image-feedback@google.com.