Google Cloud Dataproc v1 API - Class ClusterConfig (5.29.0) 

public sealed class ClusterConfig : IMessage<ClusterConfig>, IEquatable<ClusterConfig>, IDeepCloneable<ClusterConfig>, IBufferMessage, IMessage

Reference documentation and code samples for the Google Cloud Dataproc v1 API class ClusterConfig.

The cluster config.

Inheritance

object > ClusterConfig

Namespace

Google.Cloud.Dataproc.V1

Assembly

Google.Cloud.Dataproc.V1.dll

Constructors

ClusterConfig()

public ClusterConfig()

ClusterConfig(ClusterConfig)

public ClusterConfig(ClusterConfig other)
Parameter
Name Description
other ClusterConfig

Properties

AutoscalingConfig

public AutoscalingConfig AutoscalingConfig { get; set; }

Optional. Autoscaling config for the policy associated with the cluster. Cluster does not autoscale if this field is unset.

Property Value
Type Description
AutoscalingConfig

AuxiliaryNodeGroups

public RepeatedField<AuxiliaryNodeGroup> AuxiliaryNodeGroups { get; }

Optional. The node group settings.

Property Value
Type Description
RepeatedFieldAuxiliaryNodeGroup

ClusterTier

public ClusterConfig.Types.ClusterTier ClusterTier { get; set; }

Optional. The cluster tier.

Property Value
Type Description
ClusterConfigTypesClusterTier

ClusterType

public ClusterConfig.Types.ClusterType ClusterType { get; set; }

Optional. The type of the cluster.

Property Value
Type Description
ClusterConfigTypesClusterType

ConfigBucket

public string ConfigBucket { get; set; }

Optional. A Cloud Storage bucket used to stage job dependencies, config files, and job driver console output. If you do not specify a staging bucket, Cloud Dataproc will determine a Cloud Storage location (US, ASIA, or EU) for your cluster's staging bucket according to the Compute Engine zone where your cluster is deployed, and then create and manage this project-level, per-location bucket (see Dataproc staging and temp buckets). This field requires a Cloud Storage bucket name, not a gs://... URI to a Cloud Storage bucket.

Property Value
Type Description
string

DataprocMetricConfig

public DataprocMetricConfig DataprocMetricConfig { get; set; }

Optional. The config for Dataproc metrics.

Property Value
Type Description
DataprocMetricConfig

EncryptionConfig

public EncryptionConfig EncryptionConfig { get; set; }

Optional. Encryption settings for the cluster.

Property Value
Type Description
EncryptionConfig

EndpointConfig

public EndpointConfig EndpointConfig { get; set; }

Optional. Port/endpoint configuration for this cluster

Property Value
Type Description
EndpointConfig

Engine

public ClusterConfig.Types.Engine Engine { get; set; }

Optional. The cluster engine.

Property Value
Type Description
ClusterConfigTypesEngine

GceClusterConfig

public GceClusterConfig GceClusterConfig { get; set; }

Optional. The shared Compute Engine config settings for all instances in a cluster.

Property Value
Type Description
GceClusterConfig

InitializationActions

public RepeatedField<NodeInitializationAction> InitializationActions { get; }

Optional. Commands to execute on each node after config is completed. By default, executables are run on master and all worker nodes. You can test a node's role metadata to run an executable on a master or worker node, as shown below using curl (you can also use wget):

ROLE=$(curl -H Metadata-Flavor:Google
http://metadata/computeMetadata/v1/instance/attributes/dataproc-role)
if [[ "${ROLE}" == 'Master' ]]; then
  ... master specific actions ...
else
  ... worker specific actions ...
fi
Property Value
Type Description
RepeatedFieldNodeInitializationAction

LifecycleConfig

public LifecycleConfig LifecycleConfig { get; set; }

Optional. Lifecycle setting for the cluster.

Property Value
Type Description
LifecycleConfig

MasterConfig

public InstanceGroupConfig MasterConfig { get; set; }

Optional. The Compute Engine config settings for the cluster's master instance.

Property Value
Type Description
InstanceGroupConfig

MasterFlexibleConfig

public FlexibleInstanceGroupConfig MasterFlexibleConfig { get; set; }

Optional. The High Obtainability (HO) config settings for the cluster's master instances.

In High Obtainability (HO) mode, flexible configs are the single source of truth for desired cluster configuration across node groups:

  • For single-node clusters: master_flexible_config must be specified, and no worker configs should be set.
  • For multi-node (standard) clusters: both master_flexible_config and [worker_flexible_config][google.cloud.dataproc.v1.ClusterConfig.worker_flexible_config] must be specified. Mixing standard and flexible configs across node groups is not allowed.

Only one of [master_config][google.cloud.dataproc.v1.ClusterConfig.master_config] or master_flexible_config can be set.

Property Value
Type Description
FlexibleInstanceGroupConfig

MetastoreConfig

public MetastoreConfig MetastoreConfig { get; set; }

Optional. Metastore configuration.

Property Value
Type Description
MetastoreConfig

SecondaryWorkerConfig

public InstanceGroupConfig SecondaryWorkerConfig { get; set; }

Optional. The Compute Engine config settings for a cluster's secondary worker instances

Property Value
Type Description
InstanceGroupConfig

SecondaryWorkerFlexibleConfig

public FlexibleInstanceGroupConfig SecondaryWorkerFlexibleConfig { get; set; }

Optional. The High Obtainability (HO) config settings for a cluster's secondary worker instances.

In High Obtainability (HO) mode, flexible configs are the single source of truth for desired cluster configuration. num_instances represents the desired capacity of the secondary worker group and is mutable via cluster updates and autoscaling.

Only one of [secondary_worker_config][google.cloud.dataproc.v1.ClusterConfig.secondary_worker_config] or secondary_worker_flexible_config can be set.

In High Obtainability (HO) mode, secondary workers are optional. If secondary workers are used in an HO cluster, they must be configured using secondary_worker_flexible_config (not secondary_worker_config).

Property Value
Type Description
FlexibleInstanceGroupConfig

SecurityConfig

public SecurityConfig SecurityConfig { get; set; }

Optional. Security settings for the cluster.

Property Value
Type Description
SecurityConfig

SoftwareConfig

public SoftwareConfig SoftwareConfig { get; set; }

Optional. The config settings for cluster software.

Property Value
Type Description
SoftwareConfig

TempBucket

public string TempBucket { get; set; }

Optional. A Cloud Storage bucket used to store ephemeral cluster and jobs data, such as Spark and MapReduce history files. If you do not specify a temp bucket, Dataproc will determine a Cloud Storage location (US, ASIA, or EU) for your cluster's temp bucket according to the Compute Engine zone where your cluster is deployed, and then create and manage this project-level, per-location bucket. The default bucket has a TTL of 90 days, but you can use any TTL (or none) if you specify a bucket (see Dataproc staging and temp buckets). This field requires a Cloud Storage bucket name, not a gs://... URI to a Cloud Storage bucket.

Property Value
Type Description
string

WorkerConfig

public InstanceGroupConfig WorkerConfig { get; set; }

Optional. The Compute Engine config settings for the cluster's worker instances.

Property Value
Type Description
InstanceGroupConfig

WorkerFlexibleConfig

public FlexibleInstanceGroupConfig WorkerFlexibleConfig { get; set; }

Optional. The High Obtainability (HO) config settings for the cluster's worker instances.

In High Obtainability (HO) mode, flexible configs are the single source of truth for desired cluster configuration. num_instances represents the desired capacity of the worker group and is mutable via cluster updates and autoscaling.

Only one of [worker_config][google.cloud.dataproc.v1.ClusterConfig.worker_config] or worker_flexible_config can be set.

In High Obtainability (HO) mode, for multi-node (standard) clusters, both [master_flexible_config][google.cloud.dataproc.v1.ClusterConfig.master_flexible_config] and worker_flexible_config must be specified. Mixing standard and flexible configs across node groups is not allowed. Single-node clusters do not specify worker configs.

Property Value
Type Description
FlexibleInstanceGroupConfig