Create a multi-zone cluster

This document describes how to create a Managed Service for Apache Spark multi-zone cluster. In a multi-zone cluster, Managed Service for Apache Spark provisions and distributes the Compute Engine VM instances of the cluster across multiple zones in a single region, instead of placing all cluster nodes in one zone.

Use a multi-zone cluster when you need to create or scale a large cluster and the capacity of a single zone might not be enough to meet your request.

How multi-zone clusters work

When you create a multi-zone cluster, Managed Service for Apache Spark provisions the cluster nodes as follows:

  • Master nodes: Managed Service for Apache Spark places the master instance group in a single zone in the cluster region by using Auto Zone placement.
  • Worker nodes: Managed Service for Apache Spark configures the primary and secondary worker groups as regional managed instance groups (MIGs). These groups provision and distribute worker VMs across any eligible zone in the region that has available capacity.

Because Managed Service for Apache Spark distributes worker VM requests across multiple zones, capacity constraints in a single zone are less likely to block your request. This helps prevent ZONE_RESOURCE_POOL_EXHAUSTED errors when you create or scale large clusters, clusters with Spot VM or preemptible secondary workers, or clusters that use high-demand machine types.

Requirements and limitations

To create a multi-zone cluster, you must use one of the following image versions or a later version in the same image version track:

Image version track Minimum supported image version
2.2.x 2.2.88
2.3.x 2.3.37
3.0.x 3.0.3

Multi-zone clusters have the following limitations:

  • You can't specify both a single zone and a multi-zone configuration in the same cluster creation request.
  • You can't create a multi-zone cluster with more than 1,000 primary workers in the initial cluster creation request. If your workload requires more than 1,000 primary workers, create the cluster with 1,000 or fewer primary workers, and then scale the cluster after it is created.

Network data transfer costs

In a single-zone cluster, all VMs are in the same zone, and network traffic that uses internal IP addresses between VMs in the same zone is offered at no charge. In a multi-zone cluster, master and worker VMs can be in different zones in the same region. Network traffic between VMs in different zones in the same region incurs inter-zone data transfer charges.

For pricing details, see VM-VM data transfer pricing within Cloud de Confiance by S3NS.

Create a multi-zone cluster

You can create a multi-zone cluster by using the Google Cloud CLI or the Dataproc API.

gcloud

To create a multi-zone cluster, run the gcloud dataproc clusters create command with the --multizone flag. Don't specify the --zone flag.

gcloud dataproc clusters create CLUSTER_NAME \
    --region=REGION \
    --multizone \
    --image-version=IMAGE_VERSION \
    --master-machine-type=MASTER_MACHINE_TYPE \
    --worker-machine-type=WORKER_MACHINE_TYPE \
    --num-workers=NUM_WORKERS

Replace the following:

  • CLUSTER_NAME: the name of the cluster.
  • REGION: an available Compute Engine region for the cluster, such as us-central1.
  • IMAGE_VERSION: an image version that supports multi-zone clusters, such as 2.3.37-debian12. For more information, see Requirements and limitations.
  • MASTER_MACHINE_TYPE: the Compute Engine machine type of the master node, such as n2-standard-4.
  • WORKER_MACHINE_TYPE: the Compute Engine machine type of the primary worker nodes, such as n2-standard-4.
  • NUM_WORKERS: the number of primary workers in the cluster. This value can't be greater than 1000.

REST

To create a multi-zone cluster, send a clusters.create request. In the gceClusterConfig object of the request, leave the zoneUri field empty, and set the multiZoneConfig.targetShape field to ANY.

HTTP method and URL:

POST https://dataproc.googleapis.com/v1/projects/PROJECT_ID/regions/REGION/clusters

Request JSON body:

{
  "projectId": "PROJECT_ID",
  "clusterName": "CLUSTER_NAME",
  "config": {
    "gceClusterConfig": {
      "multiZoneConfig": {
        "targetShape": "ANY"
      }
    },
    "softwareConfig": {
      "imageVersion": "IMAGE_VERSION"
    },
    "masterConfig": {
      "numInstances": 1,
      "machineTypeUri": "MASTER_MACHINE_TYPE"
    },
    "workerConfig": {
      "numInstances": NUM_WORKERS,
      "machineTypeUri": "WORKER_MACHINE_TYPE"
    }
  }
}

Replace the following:

  • PROJECT_ID: your Cloud de Confiance project ID.
  • REGION: an available Compute Engine region for the cluster, such as us-central1.
  • CLUSTER_NAME: the name of the cluster.
  • IMAGE_VERSION: an image version that supports multi-zone clusters, such as 2.3.37-debian12. For more information, see Requirements and limitations.
  • MASTER_MACHINE_TYPE: the Compute Engine machine type of the master node, such as n2-standard-4.
  • NUM_WORKERS: the number of primary workers in the cluster. This value can't be greater than 1000.
  • WORKER_MACHINE_TYPE: the Compute Engine machine type of the primary worker nodes, such as n2-standard-4.

What's next