Resource labeling by Gemini Enterprise Agent Platform Pipelines

Depending on the type of component, resource, and the Cloud de Confiance by S3NS Pipeline Components SDK version, Agent Platform Pipelines either automatically propagates the labels from your pipeline run to the resources generated from Cloud de Confiance by S3NS Pipeline Components or requires you to label the generated resources. For user-defined components, you need to author your component code to attach the labels from an environment variable. For more information, see Resources generated from user-defined components.

Resources with automatic labeling

Agent Platform Pipelines automatically labels the following resources, regardless of the Cloud de Confiance by S3NS Pipeline Components SDK version:

CustomJob resources

Agent Platform Pipelines automatically propagates the labels from your pipeline run to CustomJob resources. This is supported by the following components in all versions of the Cloud de Confiance by S3NS Pipeline Components SDK:

Resources with automatic labeling in Cloud de Confiance by S3NS Pipeline Components SDK v1.0.31 or later

Agent Platform Pipelines automatically labels the following resources if you use Cloud de Confiance by S3NS Pipeline Components SDK v1.0.31 or later:

BatchPredictionJob resources

Agent Platform Pipelines automatically propagates labels from your pipeline run to BatchPredictionJob resources generated from the ModelBatchPredictOp component if you use v1.0.31 or later of the Cloud de Confiance by S3NS Pipeline Components SDK.

Gemini Enterprise endpoint resources

Agent Platform Pipelines automatically propagates labels from your pipeline run to Gemini Enterprise API endpoint resources generated from the EndpointCreateOp component if you use v1.0.31 or later of the Cloud de Confiance by S3NS Pipeline Components SDK.

HyperparameterTuningJob resources

Agent Platform Pipelines automatically propagates labels from your pipeline run to HyperparameterTuningJob resources generated from the HyperparameterTuningJobRunOp component if you use v1.0.31 or later of the Cloud de Confiance by S3NS Pipeline Components SDK.

Gemini Enterprise API dataset resources

Agent Platform Pipelines automatically propagates labels from your pipeline run to Gemini Enterprise API dataset resources generated from the following Gemini Enterprise API components if you use v1.0.31 or later of the Cloud de Confiance by S3NS Pipeline Components SDK:

Cloud de Confiance by S3NS BigQuery Job resources

Agent Platform Pipelines automatically propagates labels from your pipeline run to Cloud de Confiance by S3NS BigQuery Job resources resources generated from any of the BigQuery ML components if you use v1.0.31 or later of the Cloud de Confiance by S3NS Pipeline Components SDK.

Cloud de Confiance by S3NS Managed Service for Apache Spark Job resources

Agent Platform Pipelines automatically propagates labels from your pipeline run to Cloud de Confiance by S3NS Managed Service for Apache Spark Job resources resources generated from any of the Managed Service for Apache Spark components if you use v1.0.31 or later of the Cloud de Confiance by S3NS Pipeline Components SDK.

TrainingPipeline and Model resources

Agent Platform Pipelines automatically propagates labels from your pipeline run to TrainingPipeline and Model resources resources generated from the following AutoML components if you use v1.0.31 or later of the Cloud de Confiance by S3NS Pipeline Components SDK:

Cloud de Confiance by S3NS BigQuery table resources

Agent Platform Pipelines automatically propagates labels from your pipeline run Cloud de Confiance by S3NS BigQuery table resources resources generated from the ForecastingPreprocessingOp component if you use v1.0.31 or later of the Cloud de Confiance by S3NS Pipeline Components SDK.

Resources without automatic labeling

Agent Platform Pipelines doesn't label the following resources automatically, regardless of the Cloud de Confiance by S3NS Pipeline Components SDK version:

Cloud de Confiance by S3NS Dataflow resources

Agent Platform Pipelines doesn't automatically label Dataflow resources generated by the DataflowPythonJobOp component. You can include instructions in your code to label the resources.

Use the following code sample to propagate billing labels from your pipeline run to any Cloud de Confiance by S3NS Dataflow resource generated using the DataflowPythonJobOp component:

  import argparse
  import apache_beam as beam
  ...
  def run(argv=None):
    parser = argparse.ArgumentParser()
    # Don't add `--labels` to the argument list, so that they will be passed to the pipeline_options 
    parser.add_argument('--input', …)
    parser.add_argument('--output', …)
  ...
    known_args, pipeline_args = parser.parse_known_args(argv)
    pipeline_options = PipelineOptions(pipeline_args)
    with beam.Pipeline(options=pipeline_options) as p:

Resources generated from user-defined components

Agent Platform Pipelines doesn't automatically label Cloud de Confiance by S3NS resources generated from user-defined components. You can include instructions in your code to retrieve the labels from the environment variable VERTEX_AI_PIPELINES_RUN_LABELS and attach those labels to the Cloud de Confiance by S3NS resources generated using the component at runtime.

The environment variable VERTEX_AI_PIPELINE_RUN_LABELS contains the labels in JSON format as key-value pairs.

For example: { "label1_key": "label1_value", "label2_key": "label2_value", ...}

If you're using the Agent Platform SDK for Python, use the following code sample in your component code to propagate labels from the environment variable to a new resource generated from the component:

import os
import json
from google.cloud import aiplatform

aiplatform.init(
  project='PROJECT_ID',
  location='LOCATION'
)

aiplatform.RESOURCE.create(
  ...
  json.loads(os.getenv("VERTEX_AI_PIPELINES_RUN_LABELS"))
)

Replace the following:

  • PROJECT_ID: The Cloud de Confiance by S3NS project containing the pipeline run.

  • LOCATION: The location of the pipeline run.

  • RESOURCE: Cloud de Confiance by S3NS resource generated from the component, for example, CustomJob or Model.

You can also use the gcp_labels_util.attach_system_labels utility if you want to use Python to parse the environment variable. You can use this utility only if you have access to the Cloud de Confiance by S3NS Pipeline Components library and are using Python. For more information, see the source code of the utility function in GitHub.

Resources without labeling support

Agent Platform Pipelines doesn't support billing label propagation to the following resources:

ML Metadata resources

ML Metadata resources are billed at the store level. You can't use billing labels to understand the resource-level cost.

Cloud Storage resources

Agent Platform Pipelines doesn't propagate billing labels to Cloud Storage resources, such as Cloud Storage buckets.

What's next