REST Resource: projects.regions.clusters

Resource: Cluster

Describes the identifying information, config, and status of a cluster

JSON representation
{
  "projectId": string,
  "clusterName": string,
  "config": {
    object (ClusterConfig)
  },
  "labels": {
    string: string,
    ...
  },
  "status": {
    object (ClusterStatus)
  },
  "statusHistory": [
    {
      object (ClusterStatus)
    }
  ],
  "clusterUuid": string,
  "metrics": {
    object (ClusterMetrics)
  }
}
Fields
projectId

string

Required. The Cloud de Confiance Platform project ID that the cluster belongs to.

clusterName

string

Required. The cluster name, which must be unique within a project. The name must start with a lowercase letter, and can contain up to 51 lowercase letters, numbers, and hyphens. It cannot end with a hyphen. The name of a deleted cluster can be reused.

config

object (ClusterConfig)

Optional. The cluster config for a cluster of Compute Engine Instances. Note that the service may set default values, and values may change when clusters are updated.

Exactly one of ClusterConfig or VirtualClusterConfig must be specified.

labels

map (key: string, value: string)

Optional. The labels to associate with this cluster. Label keys must contain 1 to 63 characters, and must conform to RFC 1035. Label values may be empty, but, if present, must contain 1 to 63 characters, and must conform to RFC 1035. No more than 32 labels can be associated with a cluster.

An object containing a list of "key": value pairs. Example: { "name": "wrench", "mass": "1.3kg", "count": "3" }.

status

object (ClusterStatus)

Output only. Cluster status.

statusHistory[]

object (ClusterStatus)

Output only. The previous cluster status.

clusterUuid

string

Output only. A cluster UUID (Unique Universal Identifier). The service generates this value when it creates the cluster.

metrics

object (ClusterMetrics)

Output only. Contains cluster daemon metrics such as HDFS and YARN stats.

Beta Feature: This report is available for testing purposes only. It may be changed before final release.

ClusterConfig

The cluster config.

JSON representation
{
  "clusterType": enum (ClusterType),
  "engine": enum (Engine),
  "configBucket": string,
  "tempBucket": string,
  "gceClusterConfig": {
    object (GceClusterConfig)
  },
  "masterConfig": {
    object (InstanceGroupConfig)
  },
  "workerConfig": {
    object (InstanceGroupConfig)
  },
  "secondaryWorkerConfig": {
    object (InstanceGroupConfig)
  },
  "softwareConfig": {
    object (SoftwareConfig)
  },
  "initializationActions": [
    {
      object (NodeInitializationAction)
    }
  ],
  "encryptionConfig": {
    object (EncryptionConfig)
  },
  "securityConfig": {
    object (SecurityConfig)
  },
  "lifecycleConfig": {
    object (LifecycleConfig)
  },
  "metastoreConfig": {
    object (MetastoreConfig)
  },
  "dataprocMetricConfig": {
    object (DataprocMetricConfig)
  },
  "auxiliaryNodeGroups": [
    {
      object (AuxiliaryNodeGroup)
    }
  ],
  "masterFlexibleConfig": {
    object (FlexibleInstanceGroupConfig)
  },
  "workerFlexibleConfig": {
    object (FlexibleInstanceGroupConfig)
  },
  "secondaryWorkerFlexibleConfig": {
    object (FlexibleInstanceGroupConfig)
  },

  // The following is a list of mutually exclusive fields. At most one of the
  // fields will be set in a response:
  "schedulingConfig": {
    object (SchedulingConfig)
  }
  // End of mutually exclusive fields.
}
Fields
clusterType

enum (ClusterType)

Optional. The type of the cluster.

engine

enum (Engine)

Optional. The cluster engine.

configBucket

string

Optional. A Cloud Storage bucket used to stage job dependencies, config files, and job driver console output. If you do not specify a staging bucket, Cloud Dataproc will determine a Cloud Storage location (US, ASIA, or EU) for your cluster's staging bucket according to the Compute Engine zone where your cluster is deployed, and then create and manage this project-level, per-location bucket (see Dataproc staging and temp buckets). This field requires a Cloud Storage bucket name, not a gs://... URI to a Cloud Storage bucket.

tempBucket

string

Optional. A Cloud Storage bucket used to store ephemeral cluster and jobs data, such as Spark and MapReduce history files. If you do not specify a temp bucket, Dataproc will determine a Cloud Storage location (US, ASIA, or EU) for your cluster's temp bucket according to the Compute Engine zone where your cluster is deployed, and then create and manage this project-level, per-location bucket. The default bucket has a TTL of 90 days, but you can use any TTL (or none) if you specify a bucket (see Dataproc staging and temp buckets). This field requires a Cloud Storage bucket name, not a gs://... URI to a Cloud Storage bucket.

gceClusterConfig

object (GceClusterConfig)

Optional. The shared Compute Engine config settings for all instances in a cluster.

masterConfig

object (InstanceGroupConfig)

Optional. The Compute Engine config settings for the cluster's master instance.

workerConfig

object (InstanceGroupConfig)

Optional. The Compute Engine config settings for the cluster's worker instances.

secondaryWorkerConfig

object (InstanceGroupConfig)

Optional. The Compute Engine config settings for a cluster's secondary worker instances

softwareConfig

object (SoftwareConfig)

Optional. The config settings for cluster software.

initializationActions[]

object (NodeInitializationAction)

Optional. Commands to execute on each node after config is completed. By default, executables are run on master and all worker nodes. You can test a node's role metadata to run an executable on a master or worker node, as shown below using curl (you can also use wget):

ROLE=$(curl -H Metadata-Flavor:Google
http://metadata/computeMetadata/v1/instance/attributes/dataproc-role)
if [[ "${ROLE}" == 'Master' ]]; then
  ... master specific actions ...
else
  ... worker specific actions ...
fi
encryptionConfig

object (EncryptionConfig)

Optional. Encryption settings for the cluster.

securityConfig

object (SecurityConfig)

Optional. Security settings for the cluster.

lifecycleConfig

object (LifecycleConfig)

Optional. Lifecycle setting for the cluster.

metastoreConfig

object (MetastoreConfig)

Optional. Metastore configuration.

dataprocMetricConfig

object (DataprocMetricConfig)

Optional. The config for metrics.

auxiliaryNodeGroups[]

object (AuxiliaryNodeGroup)

Optional. The node group settings.

masterFlexibleConfig

object (FlexibleInstanceGroupConfig)

Optional. The High Obtainability (HO) config settings for the cluster's master instances.

In High Obtainability (HO) mode, flexible configs are the single source of truth for desired cluster configuration across node groups: * For single-node clusters: masterFlexibleConfig must be specified, and no worker configs should be set. * For multi-node (standard) clusters: both masterFlexibleConfig and workerFlexibleConfig must be specified. Mixing standard and flexible configs across node groups is not allowed.

Only one of masterConfig or masterFlexibleConfig can be set.

workerFlexibleConfig

object (FlexibleInstanceGroupConfig)

Optional. The High Obtainability (HO) config settings for the cluster's worker instances.

In High Obtainability (HO) mode, flexible configs are the single source of truth for desired cluster configuration. numInstances represents the desired capacity of the worker group and is mutable via cluster updates and autoscaling.

Only one of workerConfig or workerFlexibleConfig can be set.

In High Obtainability (HO) mode, for multi-node (standard) clusters, both masterFlexibleConfig and workerFlexibleConfig must be specified. Mixing standard and flexible configs across node groups is not allowed. Single-node clusters do not specify worker configs.

secondaryWorkerFlexibleConfig

object (FlexibleInstanceGroupConfig)

Optional. The High Obtainability (HO) config settings for a cluster's secondary worker instances.

In High Obtainability (HO) mode, flexible configs are the single source of truth for desired cluster configuration. numInstances represents the desired capacity of the secondary worker group and is mutable via cluster updates and autoscaling.

Only one of secondaryWorkerConfig or secondaryWorkerFlexibleConfig can be set.

In High Obtainability (HO) mode, secondary workers are optional. If secondary workers are used in an HO cluster, they must be configured using secondaryWorkerFlexibleConfig (not secondaryWorkerConfig).

Config for provisioning the Compute Engine resources. The following is a list of mutually exclusive fields. At most one of the fields will be set in a response:
schedulingConfig

object (SchedulingConfig)

Optional. Config for scheduling the resources to be allocated when available.

End of mutually exclusive fields.

ClusterType

The type of the cluster.

Enums
CLUSTER_TYPE_UNSPECIFIED Not set.
STANDARD Standard dataproc cluster with a minimum of two primary workers.
SINGLE_NODE https://documentation.s3ns.fr/dataproc/docs/concepts/configuring-clusters/single-node-clusters
ZERO_SCALE Clusters that can use only secondary workers and be scaled down to zero secondary worker nodes.

Engine

The cluster engine.

Enums
ENGINE_UNSPECIFIED The engine is not specified. Works the same as ENGINE_DEFAULT.
DEFAULT The cluster is a default engine cluster.
LIGHTNING The cluster is a Lightning Engine cluster.

GceClusterConfig

Common config settings for resources of Compute Engine cluster instances, applicable to all instances in the cluster.

JSON representation
{
  "zoneUri": string,
  "autoZoneExcludeZoneUris": [
    string
  ],
  "multiZoneConfig": {
    object (MultiZoneConfig)
  },
  "networkUri": string,
  "subnetworkUri": string,
  "privateIpv6GoogleAccess": enum (PrivateIpv6GoogleAccess),
  "serviceAccount": string,
  "serviceAccountScopes": [
    string
  ],
  "tags": [
    string
  ],
  "metadata": {
    string: string,
    ...
  },
  "reservationAffinity": {
    object (ReservationAffinity)
  },
  "nodeGroupAffinity": {
    object (NodeGroupAffinity)
  },
  "shieldedInstanceConfig": {
    object (ShieldedInstanceConfig)
  },
  "confidentialInstanceConfig": {
    object (ConfidentialInstanceConfig)
  },
  "resourceManagerTags": {
    string: string,
    ...
  },
  "internalIpOnly": boolean
}
Fields
zoneUri

string

Optional. The Compute Engine zone where the cluster will be located. If omitted, the service will pick a zone in the cluster's Compute Engine region. On a get request, zone will always be present.

A full URL, partial URI, or short name are valid. Examples:

  • https://www.googleapis.com/compute/v1/projects/[projectId]/zones/[zone]
  • projects/[projectId]/zones/[zone]
  • [zone]
autoZoneExcludeZoneUris[]

string

Optional. An optional list of Compute Engine zones where the cluster will not be located when Auto Zone is enabled. Only one of zoneUri or autoZoneExcludeZoneUris can be set. If both are omitted, the service will pick a zone in the cluster Compute Engine region. If autoZoneExcludeZoneUris is set and there is more than one non-excluded zone, the service will pick one of the non-excluded zones. Otherwise, cluster creation will fail with INVALID_ARGUMENT error.

A full URL, partial URI, or short name are valid. Examples:

  • https://www.googleapis.com/compute/v1/projects/[projectId]/zones/[zone]
  • projects/[projectId]/zones/[zone]
  • [zone]
multiZoneConfig

object (MultiZoneConfig)

Optional. Controls how instances within this Cluster are allowed to exist in multiple Zones within the Region. Only one of zoneUri or multiZoneConfig must be set.

networkUri

string

Optional. The Compute Engine network to be used for machine communications. Cannot be specified with subnetworkUri. If neither networkUri nor subnetworkUri is specified, the "default" network of the project is used, if it exists. Cannot be a "Custom Subnet Network" (see Using Subnetworks for more information).

A full URL, partial URI, or short name are valid. Examples:

  • https://www.googleapis.com/compute/v1/projects/[projectId]/global/networks/default
  • projects/[projectId]/global/networks/default
  • default
subnetworkUri

string

Optional. The Compute Engine subnetwork to be used for machine communications. Cannot be specified with networkUri.

A full URL, partial URI, or short name are valid. Examples:

  • https://www.googleapis.com/compute/v1/projects/[projectId]/regions/[region]/subnetworks/sub0
  • projects/[projectId]/regions/[region]/subnetworks/sub0
  • sub0
privateIpv6GoogleAccess

enum (PrivateIpv6GoogleAccess)

Optional. The type of IPv6 access for a cluster.

serviceAccount

string

Optional. The Dataproc service account (also see VM Data Plane identity) used by Dataproc cluster VM instances to access Cloud de Confiance Platform services.

If not specified, the Compute Engine default service account is used.

serviceAccountScopes[]

string

Optional. The URIs of service account scopes to be included in Compute Engine instances. The following base set of scopes is always included:

If no scopes are specified, the following defaults are also provided:

tags[]

string

The Compute Engine network tags to add to all instances (see Tagging instances).

metadata

map (key: string, value: string)

Optional. The Compute Engine metadata entries to add to all instances (see Project and instance metadata).

An object containing a list of "key": value pairs. Example: { "name": "wrench", "mass": "1.3kg", "count": "3" }.

reservationAffinity

object (ReservationAffinity)

Optional. Reservation Affinity for consuming Zonal reservation.

nodeGroupAffinity

object (NodeGroupAffinity)

Optional. Node Group Affinity for sole-tenant clusters.

shieldedInstanceConfig

object (ShieldedInstanceConfig)

Optional. Shielded Instance Config for clusters using Compute Engine Shielded VMs.

confidentialInstanceConfig

object (ConfidentialInstanceConfig)

Optional. Confidential Instance Config for clusters using Confidential VMs.

resourceManagerTags

map (key: string, value: string)

Optional. Resource manager tags to add to all instances (see Use secure tags in Dataproc).

An object containing a list of "key": value pairs. Example: { "name": "wrench", "mass": "1.3kg", "count": "3" }.

internalIpOnly

boolean

Optional. This setting applies to subnetwork-enabled networks. It is set to true by default in clusters created with image versions 2.2.x.

When set to true:

  • All cluster VMs have internal IP addresses.
  • Cloud de Confiance Private Access must be enabled to access Dataproc and other Cloud de Confiance APIs.
  • Off-cluster dependencies must be configured to be accessible without external IP addresses.

When set to false:

  • Cluster VMs are not restricted to internal IP addresses.
  • Ephemeral external IP addresses are assigned to each cluster VM.

MultiZoneConfig

Configuration for multi-zonal clusters that can create instances across multiple Zones within the Region.

JSON representation
{
  "targetShape": enum (TargetShape)
}
Fields
targetShape

enum (TargetShape)

Optional. The distribution shape of the nodes in the multi-zonal cluster.

TargetShape

The distribution shape of the nodes in the multi-zonal cluster.

Enums
TARGET_SHAPE_UNSPECIFIED Target shape is unspecified. Setting this will cause error.
ANY Instances may exist in any Zones within the Region.

PrivateIpv6GoogleAccess

PrivateIpv6GoogleAccess controls whether and how Dataproc cluster nodes can communicate with Cloud de Confiance Services through gRPC over IPv6. These values are directly mapped to corresponding values in the Compute Engine Instance fields.

Enums
PRIVATE_IPV6_GOOGLE_ACCESS_UNSPECIFIED If unspecified, Compute Engine default behavior will apply, which is the same as INHERIT_FROM_SUBNETWORK.
INHERIT_FROM_SUBNETWORK Private access to and from Cloud de Confiance Services configuration inherited from the subnetwork configuration. This is the default Compute Engine behavior.
OUTBOUND Enables outbound private IPv6 access to Cloud de Confiance Services from the Dataproc cluster.
BIDIRECTIONAL Enables bidirectional private IPv6 access between Cloud de Confiance Services and the Dataproc cluster.

ReservationAffinity

Reservation Affinity for consuming Zonal reservation.

JSON representation
{
  "consumeReservationType": enum (Type),
  "key": string,
  "values": [
    string
  ]
}
Fields
consumeReservationType

enum (Type)

Optional. Type of reservation to consume

key

string

Optional. Corresponds to the label key of reservation resource.

values[]

string

Optional. Corresponds to the label values of reservation resource.

Type

Indicates whether to consume capacity from an reservation or not.

Enums
TYPE_UNSPECIFIED
NO_RESERVATION Do not consume from any allocated capacity.
ANY_RESERVATION Consume any reservation available.
SPECIFIC_RESERVATION Must consume from a specific reservation. Must specify key value fields for specifying the reservations.

NodeGroupAffinity

Node Group Affinity for clusters using sole-tenant node groups. The NodeGroupAffinity resource is not related to the NodeGroup resource.

JSON representation
{
  "nodeGroupUri": string
}
Fields
nodeGroupUri

string

Required. The URI of a sole-tenant node group resource that the cluster will be created on.

A full URL, partial URI, or node group name are valid. Examples:

  • https://www.googleapis.com/compute/v1/projects/[projectId]/zones/[zone]/nodeGroups/node-group-1
  • projects/[projectId]/zones/[zone]/nodeGroups/node-group-1
  • node-group-1

ShieldedInstanceConfig

Shielded Instance Config for clusters using Compute Engine Shielded VMs.

JSON representation
{
  "enableSecureBoot": boolean,
  "enableVtpm": boolean,
  "enableIntegrityMonitoring": boolean
}
Fields
enableSecureBoot

boolean

Optional. Defines whether instances have Secure Boot enabled.

enableVtpm

boolean

Optional. Defines whether instances have the vTPM enabled.

enableIntegrityMonitoring

boolean

Optional. Defines whether instances have integrity monitoring enabled.

ConfidentialInstanceConfig

Confidential Instance Config for clusters using Confidential VMs

JSON representation
{
  "enableConfidentialCompute": boolean
}
Fields
enableConfidentialCompute
(deprecated)

boolean

Optional. Deprecated: Use 'confidentialInstanceType' instead. Defines whether the instance should have confidential compute enabled.

SoftwareConfig

Specifies the selection and config of software inside the cluster.

JSON representation
{
  "imageVersion": string,
  "properties": {
    string: string,
    ...
  }
}
Fields
imageVersion

string

Optional. The version of software inside the cluster. It must be one of the supported Dataproc Versions, such as "1.2" (including a subminor version, such as "1.2.29"), or the "preview" version. If unspecified, it defaults to the latest Debian version.

properties

map (key: string, value: string)

Optional. The properties to set on daemon config files.

Property keys are specified in prefix:property format, for example core:hadoop.tmp.dir. The following are supported prefixes and their mappings:

  • capacity-scheduler: capacity-scheduler.xml
  • core: core-site.xml
  • distcp: distcp-default.xml
  • hdfs: hdfs-site.xml
  • hive: hive-site.xml
  • mapred: mapred-site.xml
  • pig: pig.properties
  • spark: spark-defaults.conf
  • yarn: yarn-site.xml

For more information, see Cluster properties.

An object containing a list of "key": value pairs. Example: { "name": "wrench", "mass": "1.3kg", "count": "3" }.

NodeInitializationAction

Specifies an executable to run on a fully configured node and a timeout period for executable completion.

JSON representation
{
  "executableFile": string,
  "executionTimeout": string
}
Fields
executableFile

string

Required. Cloud Storage URI of executable file.

executionTimeout

string (Duration format)

Optional. Amount of time executable has to complete. Default is 10 minutes (see JSON representation of Duration).

Cluster creation fails with an explanatory error message (the name of the executable that caused the error and the exceeded timeout period) if the executable is not completed at end of the timeout period.

EncryptionConfig

Encryption settings for the cluster.

JSON representation
{
  "gcePdKmsKeyName": string,
  "kmsKey": string
}
Fields
gcePdKmsKeyName

string

Optional. The Cloud KMS key resource name to use for persistent disk encryption for all instances in the cluster. See Use CMEK with cluster data for more information.

kmsKey

string

Optional. The Cloud KMS key resource name to use for cluster persistent disk and job argument encryption. See Use CMEK with cluster data for more information.

When this key resource name is provided, the following job arguments of the following job types submitted to the cluster are encrypted using CMEK:

SecurityConfig

Security related configuration, including encryption, Kerberos, etc.

JSON representation
{
  "kerberosConfig": {
    object (KerberosConfig)
  },
  "identityConfig": {
    object (IdentityConfig)
  }
}
Fields
kerberosConfig

object (KerberosConfig)

Optional. Kerberos related configuration.

identityConfig

object (IdentityConfig)

Optional. Identity related configuration, including service account based secure multi-tenancy user mappings.

KerberosConfig

Specifies Kerberos related configuration.

JSON representation
{
  "enableKerberos": boolean,
  "rootPrincipalPasswordUri": string,
  "kmsKeyUri": string,
  "keystoreUri": string,
  "truststoreUri": string,
  "keystorePasswordUri": string,
  "keyPasswordUri": string,
  "truststorePasswordUri": string,
  "crossRealmTrustRealm": string,
  "crossRealmTrustKdc": string,
  "crossRealmTrustAdminServer": string,
  "crossRealmTrustSharedPasswordUri": string,
  "kdcDbKeyUri": string,
  "tgtLifetimeHours": integer,
  "realm": string
}
Fields
enableKerberos

boolean

Optional. Flag to indicate whether to Kerberize the cluster (default: false). Set this field to true to enable Kerberos on a cluster.

rootPrincipalPasswordUri

string

Optional. The Cloud Storage URI of a KMS encrypted file containing the root principal password.

kmsKeyUri

string

Optional. The URI of the KMS key used to encrypt sensitive files.

keystoreUri

string

Optional. The Cloud Storage URI of the keystore file used for SSL encryption. If not provided, the service will provide a self-signed certificate.

truststoreUri

string

Optional. The Cloud Storage URI of the truststore file used for SSL encryption. If not provided, the service will provide a self-signed certificate.

keystorePasswordUri

string

Optional. The Cloud Storage URI of a KMS encrypted file containing the password to the user provided keystore. For the self-signed certificate, this password is generated by the service.

keyPasswordUri

string

Optional. The Cloud Storage URI of a KMS encrypted file containing the password to the user provided key. For the self-signed certificate, this password is generated by the service.

truststorePasswordUri

string

Optional. The Cloud Storage URI of a KMS encrypted file containing the password to the user provided truststore. For the self-signed certificate, this password is generated by the service.

crossRealmTrustRealm

string

Optional. The remote realm the on-cluster KDC will trust, should the user enable cross realm trust.

crossRealmTrustKdc

string

Optional. The KDC (IP or hostname) for the remote trusted realm in a cross realm trust relationship.

crossRealmTrustAdminServer

string

Optional. The admin server (IP or hostname) for the remote trusted realm in a cross realm trust relationship.

crossRealmTrustSharedPasswordUri

string

Optional. The Cloud Storage URI of a KMS encrypted file containing the shared password between the on-cluster Kerberos realm and the remote trusted realm, in a cross realm trust relationship.

kdcDbKeyUri

string

Optional. The Cloud Storage URI of a KMS encrypted file containing the master key of the KDC database.

tgtLifetimeHours

integer

Optional. The lifetime of the ticket granting ticket, in hours. If not specified, or user specifies 0, then default value 10 will be used.

realm

string

Optional. The name of the on-cluster Kerberos realm. If not specified, the uppercased domain of hostnames will be the realm.

IdentityConfig

Identity related configuration, including service account based secure multi-tenancy user mappings.

JSON representation
{
  "userServiceAccountMapping": {
    string: string,
    ...
  },
  "enableSsh": boolean
}
Fields
userServiceAccountMapping

map (key: string, value: string)

Required. Map of user to service account.

An object containing a list of "key": value pairs. Example: { "name": "wrench", "mass": "1.3kg", "count": "3" }.

enableSsh

boolean

Optional. Whether to enable SSH access for the cluster. The default is true for image versions prior to 3.1 and false for image versions 3.1 and later. The default behavior can be changed when creating clusters using image versions 2.3.30 and later.

LifecycleConfig

Specifies the cluster auto-delete schedule configuration.

JSON representation
{
  "idleDeleteTtl": string,
  "idleStopTtl": string,
  "idleStartTime": string,

  // The following is a list of mutually exclusive fields. At most one of the
  // fields will be set in a response:
  "autoDeleteTime": string,
  "autoDeleteTtl": string
  // End of mutually exclusive fields.

  // The following is a list of mutually exclusive fields. At most one of the
  // fields will be set in a response:
  "autoStopTime": string,
  "autoStopTtl": string
  // End of mutually exclusive fields.
}
Fields
idleDeleteTtl

string (Duration format)

Optional. The duration to keep the cluster alive while idling (when no jobs are running). Passing this threshold will cause the cluster to be deleted. Minimum value is 5 minutes; maximum value is 14 days (see JSON representation of Duration).

idleStopTtl

string (Duration format)

Optional. The duration to keep the cluster started while idling (when no jobs are running). Passing this threshold will cause the cluster to be stopped. Minimum value is 5 minutes; maximum value is 14 days (see JSON representation of Duration).

idleStartTime

string (Timestamp format)

Output only. The time when cluster became idle (most recent job finished) and became eligible for deletion due to idleness (see JSON representation of Timestamp).

Uses RFC 3339, where generated output will always be Z-normalized and use 0, 3, 6 or 9 fractional digits. Offsets other than "Z" are also accepted. Examples: "2014-10-02T15:01:23Z", "2014-10-02T15:01:23.045123456Z" or "2014-10-02T15:01:23+05:30".

Either the exact time the cluster should be deleted at or the cluster maximum age. The following is a list of mutually exclusive fields. At most one of the fields will be set in a response:
autoDeleteTime

string (Timestamp format)

Optional. The time when cluster will be auto-deleted (see JSON representation of Timestamp).

Uses RFC 3339, where generated output will always be Z-normalized and use 0, 3, 6 or 9 fractional digits. Offsets other than "Z" are also accepted. Examples: "2014-10-02T15:01:23Z", "2014-10-02T15:01:23.045123456Z" or "2014-10-02T15:01:23+05:30".

autoDeleteTtl

string (Duration format)

Optional. The lifetime duration of cluster. The cluster will be auto-deleted at the end of this period. Minimum value is 10 minutes; maximum value is 14 days (see JSON representation of Duration).

End of mutually exclusive fields.
Either the exact time the cluster should be stopped at or the cluster maximum age. The following is a list of mutually exclusive fields. At most one of the fields will be set in a response:
autoStopTime

string (Timestamp format)

Optional. The time when cluster will be auto-stopped (see JSON representation of Timestamp).

Uses RFC 3339, where generated output will always be Z-normalized and use 0, 3, 6 or 9 fractional digits. Offsets other than "Z" are also accepted. Examples: "2014-10-02T15:01:23Z", "2014-10-02T15:01:23.045123456Z" or "2014-10-02T15:01:23+05:30".

autoStopTtl

string (Duration format)

Optional. The lifetime duration of the cluster. The cluster will be auto-stopped at the end of this period, calculated from the time of submission of the create or update cluster request. Minimum value is 10 minutes; maximum value is 14 days (see JSON representation of Duration).

End of mutually exclusive fields.

MetastoreConfig

Specifies a Metastore configuration.

JSON representation
{
  "dataprocMetastoreService": string
}
Fields
dataprocMetastoreService

string

Required. Resource name of an existing Metastore service.

Example:

  • projects/[projectId]/locations/[dataproc_region]/services/[service-name]

DataprocMetricConfig

Metric config.

JSON representation
{
  "metrics": [
    {
      object (Metric)
    }
  ]
}
Fields
metrics[]

object (Metric)

Required. Metrics sources to enable.

Metric

A custom metric.

JSON representation
{
  "metricSource": enum (MetricSource),
  "metricOverrides": [
    string
  ]
}
Fields
metricSource

enum (MetricSource)

Required. A standard set of metrics is collected unless metricOverrides are specified for the metric source (see Custom metrics for more information).

metricOverrides[]

string

Optional. Specify one or more Custom metrics to collect for the metric course (for the SPARK metric source (any Spark metric can be specified).

Provide metrics in the following format: METRIC_SOURCE:INSTANCE:GROUP:METRIC Use camelcase as appropriate.

Examples:

yarn:ResourceManager:QueueMetrics:AppsCompleted
spark:driver:DAGScheduler:job.allJobs
sparkHistoryServer:JVM:Memory:NonHeapMemoryUsage.committed
hiveserver2:JVM:Memory:NonHeapMemoryUsage.used

Notes:

  • Only the specified overridden metrics are collected for the metric source. For example, if one or more spark:executive metrics are listed as metric overrides, other SPARK metrics are not collected. The collection of the metrics for other enabled custom metric sources is unaffected. For example, if both SPARK and YARN metric sources are enabled, and overrides are provided for Spark metrics only, all YARN metrics are collected.

MetricSource

A source for the collection of Dataproc custom metrics (see Custom metrics).

Enums
METRIC_SOURCE_UNSPECIFIED Required unspecified metric source.
MONITORING_AGENT_DEFAULTS Monitoring agent metrics. If this source is enabled, the service enables the monitoring agent in Compute Engine, and collects monitoring agent metrics, which are published with an agent.googleapis.com prefix.
HDFS HDFS metric source.
SPARK Spark metric source.
YARN YARN metric source.
SPARK_HISTORY_SERVER Spark History Server metric source.
HIVESERVER2 Hiveserver2 metric source.
HIVEMETASTORE hivemetastore metric source

AuxiliaryNodeGroup

Node group identification and configuration information.

JSON representation
{
  "nodeGroup": {
    object (NodeGroup)
  },
  "nodeGroupId": string
}
Fields
nodeGroup

object (NodeGroup)

Required. Node group configuration.

nodeGroupId

string

Optional. A node group ID. Generated if not specified.

The ID must contain only letters (a-z, A-Z), numbers (0-9), underscores (_), and hyphens (-). Cannot begin or end with underscore or hyphen. Must consist of from 3 to 33 characters.

SchedulingConfig

Config for scheduling the request to create Compute Engine resources for the cluster, when available.

JSON representation
{
  "schedulingTimeout": string,
  "requestedRunDuration": string
}
Fields
schedulingTimeout

string (Duration format)

Optional. How long to wait for worker resources to be allocated before failing the cluster creation request. Max/Default value: 13 days

requestedRunDuration

string (Duration format)

Optional. Required lifetime of the resources, once provisioned. Min 10 mins, Max/Default 7 days . Note that the cluster can still be deleted before reaching this time. This time is a maximum amount of time before the cluster is forcibly deleted. Lower times are more likely to start running sooner.

FlexibleInstanceGroupConfig

The configuration for High Obtainability (HO) flexible instance groups. Flexible instance group configs serve as the single source of truth for desired state.

JSON representation
{
  "numInstances": integer,
  "coreCount": integer,
  "bootDiskSizeGb": integer,
  "hardwareClass": enum (HardwareClass)
}
Fields
numInstances

integer

Optional. The number of VM instances requested.

For master instances, this must be 1. For worker and secondary worker instances, this represents the desired capacity of the group and is mutable via cluster update operations and autoscaling.

coreCount

integer

Optional. The number of vCPUs per instance. Must be a power of 2 (2, 4, 8, 16, 32). This is a create-time specification and is not currently mutable via update masks.

bootDiskSizeGb

integer

Optional. The boot disk size in GB. This is a create-time specification and is not currently mutable via update masks.

hardwareClass

enum (HardwareClass)

Optional. The hardware class of the instances. This is a create-time specification and is not currently mutable via update masks.

HardwareClass

Controls the hardware class for the flexible instance group.

Enums
HARDWARE_CLASS_UNSPECIFIED Unspecified hardware class.
STANDARD

Standard hardware class. Provides a balanced compute configuration with approximately 4GB of memory per vCPU and standard disk storage tiers.

Candidate machine types include n4-standard, n4d-standard, n2-standard, and n2d-standard (Rank 0), and c4-standard and c4d-standard (Rank 1) (for requested core counts 2, 4, 8, 16, 32).

Warning: The specific machine families and types selected for this hardware class are subject to change. Users should not take a dependency on the underlying machine type selection.

PERFORMANCE

Performance hardware class. Provides a high-performance configuration with approximately 8GB of memory per vCPU (e.g. high-memory machine types) and higher-tier disk configurations (e.g. higher IOPS and throughput).

Candidate machine types include n4-highmem, n4d-highmem, n2-highmem, and n2d-highmem (Rank 0), and c4-highmem and c4d-highmem (Rank 1) (for requested core counts 2, 4, 8, 16, 32). n4, n4d, c4, and c4d use Hyperdisk Balanced with provisioned storage, whereas n2 and n2d use SSD persistent disks.

Warning: The specific machine families and types selected for this hardware class are subject to change. Users should not take a dependency on the underlying machine type selection.

ClusterStatus

The status of a cluster and its instances.

JSON representation
{
  "state": enum (State),
  "detail": string,
  "stateStartTime": string,
  "substate": enum (Substate)
}
Fields
state

enum (State)

Output only. The cluster's state.

detail

string

Optional. Details of cluster's state.

stateStartTime

string (Timestamp format)

Output only. Time when this state was entered (see JSON representation of Timestamp).

Uses RFC 3339, where generated output will always be Z-normalized and use 0, 3, 6 or 9 fractional digits. Offsets other than "Z" are also accepted. Examples: "2014-10-02T15:01:23Z", "2014-10-02T15:01:23.045123456Z" or "2014-10-02T15:01:23+05:30".

substate

enum (Substate)

Output only. Additional state information that includes status reported by the agent.

State

The cluster state.

Enums
UNKNOWN The cluster state is unknown.
CREATING The cluster is being created and set up. It is not ready for use.
RUNNING

The cluster is currently running and healthy. It is ready for use.

Note: The cluster state changes from "creating" to "running" status after the master node(s), first two primary worker nodes (and the last primary worker node if primary workers > 2) are running.

ERROR The cluster encountered an error. It is not ready for use.
ERROR_DUE_TO_UPDATE The cluster has encountered an error while being updated. Jobs can be submitted to the cluster, but the cluster cannot be updated.
DELETING The cluster is being deleted. It cannot be used.
UPDATING The cluster is being updated. It continues to accept and process jobs.
STOPPING The cluster is being stopped. It cannot be used.
STOPPED The cluster is currently stopped. It is not ready for use.
STARTING The cluster is being started. It is not ready for use.
SCHEDULED Cluster creation is currently waiting for resources to be available. Once all resources are available, it will transition to CREATING and then RUNNING.

Substate

The cluster substate.

Enums
UNSPECIFIED The cluster substate is unknown.
UNHEALTHY

The cluster is known to be in an unhealthy state (for example, critical daemons are not running or HDFS capacity is exhausted).

Applies to RUNNING state.

STALE_STATUS

The agent-reported status is out of date (may occur if the service loses communication with the Agent).

Applies to RUNNING state.

ClusterMetrics

Contains cluster daemon metrics, such as HDFS and YARN stats.

Beta Feature: This report is available for testing purposes only. It may be changed before final release.

JSON representation
{
  "hdfsMetrics": {
    string: string,
    ...
  },
  "yarnMetrics": {
    string: string,
    ...
  }
}
Fields
hdfsMetrics

map (key: string, value: string (int64 format))

The HDFS metrics.

An object containing a list of "key": value pairs. Example: { "name": "wrench", "mass": "1.3kg", "count": "3" }.

yarnMetrics

map (key: string, value: string (int64 format))

YARN metrics.

An object containing a list of "key": value pairs. Example: { "name": "wrench", "mass": "1.3kg", "count": "3" }.

Methods

create

Creates a cluster in a project.

delete

Deletes a cluster in a project.

diagnose

Gets cluster diagnostic information.

get

Gets the resource representation for a cluster in a project.

list

Lists all regions/{region}/clusters in a project alphabetically.

patch

Updates a cluster in a project.

start

Starts a cluster in a project.

stop

Stops a cluster in a project.

getIamPolicy

The method google.iam.v1.IAMPolicy.GetIamPolicy is not available in Cloud de Confiance by S3NS.

setIamPolicy

The method google.iam.v1.IAMPolicy.SetIamPolicy is not available in Cloud de Confiance by S3NS.

testIamPermissions

The method google.iam.v1.IAMPolicy.TestIamPermissions is not available in Cloud de Confiance by S3NS.