REST Resource: projects.regions.jobs

Resource: Job

A Dataproc job resource.

JSON representation 
{
  "reference": {
    object (JobReference)
  },
  "placement": {
    object (JobPlacement)
  },
  "status": {
    object (JobStatus)
  },
  "statusHistory": [
    {
      object (JobStatus)
    }
  ],
  "yarnApplications": [
    {
      object (YarnApplication)
    }
  ],
  "driverOutputResourceUri": string,
  "driverControlFilesUri": string,
  "labels": {
    string: string,
    ...
  },
  "scheduling": {
    object (JobScheduling)
  },
  "jobUuid": string,
  "done": boolean,
  "driverSchedulingConfig": {
    object (DriverSchedulingConfig)
  },

  // Union field type_job can be only one of the following:
  "hadoopJob": {
    object (HadoopJob)
  },
  "sparkJob": {
    object (SparkJob)
  },
  "pysparkJob": {
    object (PySparkJob)
  },
  "hiveJob": {
    object (HiveJob)
  },
  "pigJob": {
    object (PigJob)
  },
  "sparkRJob": {
    object (SparkRJob)
  },
  "sparkSqlJob": {
    object (SparkSqlJob)
  }
  // End of list of possible types for union field type_job.
}
Fields
reference

object (JobReference)

Optional. The fully qualified reference to the job, which can be used to obtain the equivalent REST path of the job resource. If this property is not specified when a job is created, the server generates a

jobId

.
placement

object (JobPlacement)

Required. Job information, including how, when, and where to run the job.
status

object (JobStatus)

Output only. The job status. Additional application-specific status information might be contained in the

type_job

and

yarnApplications

fields.
statusHistory[]

object (JobStatus)

Output only. The previous job status.
yarnApplications[]

object (YarnApplication)

Output only. The collection of YARN applications spun up by this job.

Beta Feature: This report is available for testing purposes only. It might be changed before final release.
driverOutputResourceUri

string

Output only. A URI pointing to the location of the stdout of the job's driver program.
driverControlFilesUri

string

Output only. If present, the location of miscellaneous control files which can be used as part of job setup and handling. If not present, control files might be placed in the same location as driver_output_uri.
labels

map (key: string, value: string)

Optional. The labels to associate with this job. Label keys must contain 1 to 63 characters, and must conform to RFC 1035. Label values can be empty, but, if present, must contain 1 to 63 characters, and must conform to RFC 1035. No more than 32 labels can be associated with a job.

An object containing a list of "key": value pairs. Example: { "name": "wrench", "mass": "1.3kg", "count": "3" }.
scheduling

object (JobScheduling)

Optional. Job scheduling configuration.
jobUuid

string

Output only. A UUID that uniquely identifies a job within the project over time. This is in contrast to a user-settable reference.job_id that might be reused over time.
done

boolean

Output only. Indicates whether the job is completed. If the value is false, the job is still in progress. If true, the job is completed, and status.state field will indicate if it was successful, failed, or cancelled.
driverSchedulingConfig

object (DriverSchedulingConfig)

Optional. Driver scheduling configuration.
Union field type_job. Required. The application/framework-specific portion of the job. type_job can be only one of the following:
hadoopJob

object (HadoopJob)

Optional. Job is a Hadoop job.
sparkJob

object (SparkJob)

Optional. Job is a Spark job.
pysparkJob

object (PySparkJob)

Optional. Job is a PySpark job.
hiveJob

object (HiveJob)

Optional. Job is a Hive job.
pigJob

object (PigJob)

Optional. Job is a Pig job.
sparkRJob

object (SparkRJob)

Optional. Job is a SparkR job.
sparkSqlJob

object (SparkSqlJob)

Optional. Job is a SparkSql job.

JobReference

Encapsulates the full scoping used to reference a job.

JSON representation 
{
  "projectId": string,
  "jobId": string
}
Fields
projectId

string

Optional. The ID of the Google Cloud Platform project that the job belongs to. If specified, must match the request project ID.
jobId

string

Optional. The job ID, which must be unique within the project.

The ID must contain only letters (a-z, A-Z), numbers (0-9), underscores (_), or hyphens (-). The maximum length is 100 characters.

If not specified by the caller, the job ID will be provided by the server.

JobPlacement

Dataproc job config.

JSON representation 
{
  "clusterName": string,
  "clusterUuid": string,
  "clusterLabels": {
    string: string,
    ...
  }
}
Fields
clusterName

string

Required. The name of the cluster where the job will be submitted.
clusterUuid

string

Output only. A cluster UUID generated by the Dataproc service when the job is submitted.
clusterLabels

map (key: string, value: string)

Optional. Cluster labels to identify a cluster where the job will be submitted.

An object containing a list of "key": value pairs. Example: { "name": "wrench", "mass": "1.3kg", "count": "3" }.

HadoopJob

A Dataproc job for running Apache Hadoop MapReduce jobs on Apache Hadoop YARN.

JSON representation 
{
  "args": [
    string
  ],
  "jarFileUris": [
    string
  ],
  "fileUris": [
    string
  ],
  "archiveUris": [
    string
  ],
  "properties": {
    string: string,
    ...
  },
  "loggingConfig": {
    object (LoggingConfig)
  },

  // Union field driver can be only one of the following:
  "mainJarFileUri": string,
  "mainClass": string
  // End of list of possible types for union field driver.
}
Fields
args[]

string

Optional. The arguments to pass to the driver. Do not include arguments, such as -libjars or -Dfoo=bar, that can be set as job properties, since a collision might occur that causes an incorrect job submission.
jarFileUris[]

string

Optional. Jar file URIs to add to the CLASSPATHs of the Hadoop driver and tasks.
fileUris[]

string

Optional. HCFS (Hadoop Compatible Filesystem) URIs of files to be copied to the working directory of Hadoop drivers and distributed tasks. Useful for naively parallel tasks.
archiveUris[]

string

Optional. HCFS URIs of archives to be extracted in the working directory of Hadoop drivers and tasks. Supported file types: .jar, .tar, .tar.gz, .tgz, or .zip.
properties

map (key: string, value: string)

Optional. A mapping of property names to values, used to configure Hadoop. Properties that conflict with values set by the Dataproc API might be overwritten. Can include properties set in /etc/hadoop/conf/*-site and classes in user code.

An object containing a list of "key": value pairs. Example: { "name": "wrench", "mass": "1.3kg", "count": "3" }.
loggingConfig

object (LoggingConfig)

Optional. The runtime log config for job execution.
Union field driver. Required. Indicates the location of the driver's main class. Specify either the jar file that contains the main class or the main class name. To specify both, add the jar file to jar_file_uris, and then specify the main class name in this property. driver can be only one of the following:
mainJarFileUri

string

The HCFS URI of the jar file containing the main class. Examples: 'gs://foo-bucket/analytics-binaries/extract-useful-metrics-mr.jar' 'hdfs:/tmp/test-samples/custom-wordcount.jar' 'file:///home/usr/lib/hadoop-mapreduce/hadoop-mapreduce-examples.jar'
mainClass

string

The name of the driver's main class. The jar file containing the class must be in the default CLASSPATH or specified in jarFileUris.

LoggingConfig

The runtime logging config of the job.

JSON representation 
{
  "driverLogLevels": {
    string: enum (Level),
    ...
  }
}
Fields
driverLogLevels

map (key: string, value: enum (Level))

The per-package log levels for the driver. This can include "root" package name to configure rootLogger. Examples: - 'com.google = FATAL' - 'root = INFO' - 'org.apache = DEBUG'

An object containing a list of "key": value pairs. Example: { "name": "wrench", "mass": "1.3kg", "count": "3" }.

Level

The Log4j level for job execution. When running an Apache Hive job, Cloud Dataproc configures the Hive client to an equivalent verbosity level.

Enums
LEVEL_UNSPECIFIED Level is unspecified. Use default level for log4j.
ALL Use ALL level for log4j.
TRACE Use TRACE level for log4j.
DEBUG Use DEBUG level for log4j.
INFO Use INFO level for log4j.
WARN Use WARN level for log4j.
ERROR Use ERROR level for log4j.
FATAL Use FATAL level for log4j.
OFF Turn off log4j.

SparkJob

A Dataproc job for running Apache Spark applications on YARN.

JSON representation 
{
  "args": [
    string
  ],
  "jarFileUris": [
    string
  ],
  "fileUris": [
    string
  ],
  "archiveUris": [
    string
  ],
  "properties": {
    string: string,
    ...
  },
  "loggingConfig": {
    object (LoggingConfig)
  },

  // Union field driver can be only one of the following:
  "mainJarFileUri": string,
  "mainClass": string
  // End of list of possible types for union field driver.
}
Fields
args[]

string

Optional. The arguments to pass to the driver. Do not include arguments, such as --conf, that can be set as job properties, since a collision may occur that causes an incorrect job submission.
jarFileUris[]

string

Optional. HCFS URIs of jar files to add to the CLASSPATHs of the Spark driver and tasks.
fileUris[]

string

Optional. HCFS URIs of files to be placed in the working directory of each executor. Useful for naively parallel tasks.
archiveUris[]

string

Optional. HCFS URIs of archives to be extracted into the working directory of each executor. Supported file types: .jar, .tar, .tar.gz, .tgz, and .zip.
properties

map (key: string, value: string)

Optional. A mapping of property names to values, used to configure Spark. Properties that conflict with values set by the Dataproc API might be overwritten. Can include properties set in /etc/spark/conf/spark-defaults.conf and classes in user code.

An object containing a list of "key": value pairs. Example: { "name": "wrench", "mass": "1.3kg", "count": "3" }.
loggingConfig

object (LoggingConfig)

Optional. The runtime log config for job execution.
Union field driver. Required. The specification of the main method to call to drive the job. Specify either the jar file that contains the main class or the main class name. To pass both a main jar and a main class in that jar, add the jar to jarFileUris, and then specify the main class name in mainClass. driver can be only one of the following:
mainJarFileUri

string

The HCFS URI of the jar file that contains the main class.
mainClass

string

The name of the driver's main class. The jar file that contains the class must be in the default CLASSPATH or specified in SparkJob.jar_file_uris.

PySparkJob

A Dataproc job for running Apache PySpark applications on YARN.

JSON representation 
{
  "mainPythonFileUri": string,
  "args": [
    string
  ],
  "pythonFileUris": [
    string
  ],
  "jarFileUris": [
    string
  ],
  "fileUris": [
    string
  ],
  "archiveUris": [
    string
  ],
  "properties": {
    string: string,
    ...
  },
  "loggingConfig": {
    object (LoggingConfig)
  }
}
Fields
mainPythonFileUri

string

Required. The HCFS URI of the main Python file to use as the driver. Must be a .py file.
args[]

string

Optional. The arguments to pass to the driver. Do not include arguments, such as --conf, that can be set as job properties, since a collision may occur that causes an incorrect job submission.
pythonFileUris[]

string

Optional. HCFS file URIs of Python files to pass to the PySpark framework. Supported file types: .py, .egg, and .zip.
jarFileUris[]

string

Optional. HCFS URIs of jar files to add to the CLASSPATHs of the Python driver and tasks.
fileUris[]

string

Optional. HCFS URIs of files to be placed in the working directory of each executor. Useful for naively parallel tasks.
archiveUris[]

string

Optional. HCFS URIs of archives to be extracted into the working directory of each executor. Supported file types: .jar, .tar, .tar.gz, .tgz, and .zip.

Note: Spark applications must be deployed in cluster mode for correct environment propagation.
properties

map (key: string, value: string)

Optional. A mapping of property names to values, used to configure PySpark. Properties that conflict with values set by the Dataproc API might be overwritten. Can include properties set in /etc/spark/conf/spark-defaults.conf and classes in user code.

An object containing a list of "key": value pairs. Example: { "name": "wrench", "mass": "1.3kg", "count": "3" }.
loggingConfig

object (LoggingConfig)

Optional. The runtime log config for job execution.

HiveJob

A Dataproc job for running Apache Hive queries on YARN.

JSON representation 
{
  "continueOnFailure": boolean,
  "scriptVariables": {
    string: string,
    ...
  },
  "properties": {
    string: string,
    ...
  },
  "jarFileUris": [
    string
  ],

  // Union field queries can be only one of the following:
  "queryFileUri": string,
  "queryList": {
    object (QueryList)
  }
  // End of list of possible types for union field queries.
}
Fields
continueOnFailure

boolean

Optional. Whether to continue executing queries if a query fails. The default value is false. Setting to true can be useful when executing independent parallel queries.
scriptVariables

map (key: string, value: string)

Optional. Mapping of query variable names to values (equivalent to the Hive command: SET name="value";).

An object containing a list of "key": value pairs. Example: { "name": "wrench", "mass": "1.3kg", "count": "3" }.
properties

map (key: string, value: string)

Optional. A mapping of property names and values, used to configure Hive. Properties that conflict with values set by the Dataproc API might be overwritten. Can include properties set in /etc/hadoop/conf/*-site.xml, /etc/hive/conf/hive-site.xml, and classes in user code.

An object containing a list of "key": value pairs. Example: { "name": "wrench", "mass": "1.3kg", "count": "3" }.
jarFileUris[]

string

Optional. HCFS URIs of jar files to add to the CLASSPATH of the Hive server and Hadoop MapReduce (MR) tasks. Can contain Hive SerDes and UDFs.
Union field queries. Required. The sequence of Hive queries to execute, specified as either an HCFS file URI or a list of queries. queries can be only one of the following:
queryFileUri

string

The HCFS URI of the script that contains Hive queries.
queryList

object (QueryList)

A list of queries.

QueryList

A list of queries to run on a cluster.

JSON representation 
{
  "queries": [
    string
  ]
}
Fields
queries[]

string

Required. The queries to execute. You do not need to end a query expression with a semicolon. Multiple queries can be specified in one string by separating each with a semicolon. Here is an example of a Dataproc API snippet that uses a QueryList to specify a HiveJob:

"hiveJob": {
  "queryList": {
    "queries": [
      "query1",
      "query2",
      "query3;query4",
    ]
  }
}

PigJob

A Dataproc job for running Apache Pig queries on YARN.

JSON representation 
{
  "continueOnFailure": boolean,
  "scriptVariables": {
    string: string,
    ...
  },
  "properties": {
    string: string,
    ...
  },
  "jarFileUris": [
    string
  ],
  "loggingConfig": {
    object (LoggingConfig)
  },

  // Union field queries can be only one of the following:
  "queryFileUri": string,
  "queryList": {
    object (QueryList)
  }
  // End of list of possible types for union field queries.
}
Fields
continueOnFailure

boolean

Optional. Whether to continue executing queries if a query fails. The default value is false. Setting to true can be useful when executing independent parallel queries.
scriptVariables

map (key: string, value: string)

Optional. Mapping of query variable names to values (equivalent to the Pig command: name=[value]).

An object containing a list of "key": value pairs. Example: { "name": "wrench", "mass": "1.3kg", "count": "3" }.
properties

map (key: string, value: string)

Optional. A mapping of property names to values, used to configure Pig. Properties that conflict with values set by the Dataproc API might be overwritten. Can include properties set in /etc/hadoop/conf/*-site.xml, /etc/pig/conf/pig.properties, and classes in user code.

An object containing a list of "key": value pairs. Example: { "name": "wrench", "mass": "1.3kg", "count": "3" }.
jarFileUris[]

string

Optional. HCFS URIs of jar files to add to the CLASSPATH of the Pig Client and Hadoop MapReduce (MR) tasks. Can contain Pig UDFs.
loggingConfig

object (LoggingConfig)

Optional. The runtime log config for job execution.
Union field queries. Required. The sequence of Pig queries to execute, specified as an HCFS file URI or a list of queries. queries can be only one of the following:
queryFileUri

string

The HCFS URI of the script that contains the Pig queries.
queryList

object (QueryList)

A list of queries.

SparkRJob

A Dataproc job for running Apache SparkR applications on YARN.

JSON representation 
{
  "mainRFileUri": string,
  "args": [
    string
  ],
  "fileUris": [
    string
  ],
  "archiveUris": [
    string
  ],
  "properties": {
    string: string,
    ...
  },
  "loggingConfig": {
    object (LoggingConfig)
  }
}
Fields
mainRFileUri

string

Required. The HCFS URI of the main R file to use as the driver. Must be a .R file.
args[]

string

Optional. The arguments to pass to the driver. Do not include arguments, such as --conf, that can be set as job properties, since a collision may occur that causes an incorrect job submission.
fileUris[]

string

Optional. HCFS URIs of files to be placed in the working directory of each executor. Useful for naively parallel tasks.
archiveUris[]

string

Optional. HCFS URIs of archives to be extracted into the working directory of each executor. Supported file types: .jar, .tar, .tar.gz, .tgz, and .zip.
properties

map (key: string, value: string)

Optional. A mapping of property names to values, used to configure SparkR. Properties that conflict with values set by the Dataproc API might be overwritten. Can include properties set in /etc/spark/conf/spark-defaults.conf and classes in user code.

An object containing a list of "key": value pairs. Example: { "name": "wrench", "mass": "1.3kg", "count": "3" }.
loggingConfig

object (LoggingConfig)

Optional. The runtime log config for job execution.

SparkSqlJob

A Dataproc job for running Apache Spark SQL queries.

JSON representation 
{
  "scriptVariables": {
    string: string,
    ...
  },
  "properties": {
    string: string,
    ...
  },
  "jarFileUris": [
    string
  ],
  "loggingConfig": {
    object (LoggingConfig)
  },

  // Union field queries can be only one of the following:
  "queryFileUri": string,
  "queryList": {
    object (QueryList)
  }
  // End of list of possible types for union field queries.
}
Fields
scriptVariables

map (key: string, value: string)

Optional. Mapping of query variable names to values (equivalent to the Spark SQL command: SET name="value";).

An object containing a list of "key": value pairs. Example: { "name": "wrench", "mass": "1.3kg", "count": "3" }.
properties

map (key: string, value: string)

Optional. A mapping of property names to values, used to configure Spark SQL's SparkConf. Properties that conflict with values set by the Dataproc API might be overwritten.

An object containing a list of "key": value pairs. Example: { "name": "wrench", "mass": "1.3kg", "count": "3" }.
jarFileUris[]

string

Optional. HCFS URIs of jar files to be added to the Spark CLASSPATH.
loggingConfig

object (LoggingConfig)

Optional. The runtime log config for job execution.
Union field queries. Required. The sequence of Spark SQL queries to execute, specified as either an HCFS file URI or as a list of queries. queries can be only one of the following:
queryFileUri

string

The HCFS URI of the script that contains SQL queries.
queryList

object (QueryList)

A list of queries.

JobStatus

Dataproc job status.

JSON representation 
{
  "state": enum (State),
  "details": string,
  "stateStartTime": string,
  "substate": enum (Substate)
}
Fields
state

enum (State)

Output only. A state message specifying the overall job state.
details

string

Optional. Output only. Job state details, such as an error description if the state is ERROR.
stateStartTime

string (Timestamp format)

Output only. The time when this state was entered.

Uses RFC 3339, where generated output will always be Z-normalized and use 0, 3, 6 or 9 fractional digits. Offsets other than "Z" are also accepted. Examples: "2014-10-02T15:01:23Z", "2014-10-02T15:01:23.045123456Z" or "2014-10-02T15:01:23+05:30".
substate

enum (Substate)

Output only. Additional state information, which includes status reported by the agent.

State

The job state.

Enums
STATE_UNSPECIFIED The job state is unknown.
PENDING The job is pending; it has been submitted, but is not yet running.
SETUP_DONE Job has been received by the service and completed initial setup; it will soon be submitted to the cluster.
RUNNING The job is running on the cluster.
CANCEL_PENDING A jobs.cancel request has been received, but is pending.
CANCEL_STARTED Transient in-flight resources have been canceled, and the request to cancel the running job has been issued to the cluster.
CANCELLED The job cancellation was successful.
DONE The job has completed successfully.
ERROR The job has completed, but encountered an error.
ATTEMPT_FAILURE

Job attempt has failed. The detail field contains failure details for this attempt.

Applies to restartable jobs only.

Substate

The job substate.

Enums
UNSPECIFIED The job substate is unknown.
SUBMITTED

The Job is submitted to the agent.

Applies to RUNNING state.
QUEUED

The Job has been received and is awaiting execution (it might be waiting for a condition to be met). See the "details" field for the reason for the delay.

Applies to RUNNING state.
STALE_STATUS

The agent-reported status is out of date, which can be caused by a loss of communication between the agent and Dataproc. If the agent does not send a timely update, the job will fail.

Applies to RUNNING state.

YarnApplication

A YARN application created by a job. Application information is a subset of

org.apache.hadoop.yarn.proto.YarnProtos.ApplicationReportProto

.

Beta Feature: This report is available for testing purposes only. It may be changed before final release.

JSON representation 
{
  "name": string,
  "state": enum (State),
  "progress": number,
  "trackingUrl": string,
  "vcoreSeconds": string,
  "memoryMbSeconds": string
}
Fields
name

string

Required. The application name.
state

enum (State)

Required. The application state.
progress

number

Required. The numerical progress of the application, from 1 to 100.
trackingUrl

string

Optional. The HTTP URL of the ApplicationMaster, HistoryServer, or TimelineServer that provides application-specific information. The URL uses the internal hostname, and requires a proxy server for resolution and, possibly, access.
vcoreSeconds

string (int64 format)

Optional. The cumulative CPU time consumed by the application for a job, measured in vcore-seconds.
memoryMbSeconds

string (int64 format)

Optional. The cumulative memory usage of the application for a job, measured in mb-seconds.

State

The application state, corresponding to

YarnProtos.YarnApplicationStateProto

.

Enums
STATE_UNSPECIFIED Status is unspecified.
NEW Status is NEW.
NEW_SAVING Status is NEW_SAVING.
SUBMITTED Status is SUBMITTED.
ACCEPTED Status is ACCEPTED.
RUNNING Status is RUNNING.
FINISHED Status is FINISHED.
FAILED Status is FAILED.
KILLED Status is KILLED.

JobScheduling

Job scheduling options.

JSON representation 
{
  "maxFailuresPerHour": integer,
  "maxFailuresTotal": integer
}
Fields
maxFailuresPerHour

integer

Optional. Maximum number of times per hour a driver can be restarted as a result of driver exiting with non-zero code before job is reported failed.

A job might be reported as thrashing if the driver exits with a non-zero code four times within a 10-minute window.

Maximum value is 10.

Note: This restartable job option is not supported in Dataproc workflow templates.
maxFailuresTotal

integer

Optional. Maximum total number of times a driver can be restarted as a result of the driver exiting with a non-zero code. After the maximum number is reached, the job will be reported as failed.

Maximum value is 240.

Note: Currently, this restartable job option is not supported in Dataproc workflow templates.

DriverSchedulingConfig

Driver scheduling configuration.

JSON representation 
{
  "memoryMb": integer,
  "vcores": integer
}
Fields
memoryMb

integer

Required. The amount of memory in MB the driver is requesting.
vcores

integer

Required. The number of vCPUs the driver is requesting.

Methods

cancel

 Starts a job cancellation request.

delete

 Deletes the job from the project.

get

 Gets the resource representation for a job in a project.

list

 Lists regions/{region}/jobs in a project.

submit

 Submits a job to a cluster.

submitAsOperation

 Submits job to a cluster.

getIamPolicy

 The method google.iam.v1.IAMPolicy.GetIamPolicy is not available in Cloud de Confiance by S3NS.

patch

 The method google.cloud.dataproc.v1.JobController.UpdateJob is not available in Cloud de Confiance by S3NS.

setIamPolicy

 The method google.iam.v1.IAMPolicy.SetIamPolicy is not available in Cloud de Confiance by S3NS.

testIamPermissions

 The method google.iam.v1.IAMPolicy.TestIamPermissions is not available in Cloud de Confiance by S3NS.