Prepare for Pod snapshots

Google Kubernetes Engine (GKE) Pod snapshots let you save and restore running Pod states to improve workload startup latency. Before you can capture snapshots of your workloads, you must prepare your cluster and storage environment.

Use this document to learn how to enable and configure Pod snapshots on your cluster and set up Cloud Storage to store the snapshots. For more information about how Pod snapshots work, see About GKE Pod snapshots.

Platform admins and operators and Application developers can use this document to configure clusters, create storage buckets, and set up access permissions for Pod snapshots. For more information about the common roles and example tasks referenced in Cloud de Confiance by S3NS content, see Common GKE user roles and tasks.

Before you begin

Before you start, make sure that you have performed the following tasks:

• Enable the Google Kubernetes Engine API.
Enable Google Kubernetes Engine API
• To use the Google Cloud CLI for this task, install and then initialize the gcloud CLI. If you previously installed the gcloud CLI, get the latest version by running the gcloud components update command. Earlier gcloud CLI versions might not support running the commands in this document.

Enable Pod snapshots

You can enable Pod snapshots on a new or existing cluster. Pod snapshots require GKE Sandbox and Workload Identity Federation for GKE. Autopilot clusters enable both features by default. For Standard clusters, you must enable Workload Identity Federation for GKE on the cluster and configure a node pool to run in GKE Sandbox.

To enable Pod snapshots on a cluster, follow the steps for the GKE mode of operation that you want to use:

gcloud container node-pools create NODE_POOL_NAME \
    --cluster=CLUSTER_NAME \
    --node-version=NODE_VERSION \
    --machine-type=MACHINE_TYPE \
    --location=CONTROL_PLANE_LOCATION \
    --image-type=cos_containerd \
    --sandbox type=gvisor

Store snapshots

GKE stores Pod snapshots in a Cloud Storage bucket. Each snapshot contains the application memory and, optionally, the GPU state of the Pod. To authenticate to Cloud Storage, Pod snapshots use the Pod's ServiceAccount object, which requires Workload Identity Federation for GKE.

Pod snapshots require the following bucket configurations:

• Hierarchical namespaces: you must enable hierarchical namespaces to support higher read and write queries per second. Hierarchical namespaces also require that you enable uniform bucket-level access.
• Soft delete: because Pod snapshots use parallel composite uploads, disable data protection features like soft delete. If you leave these features enabled, then soft deletions of the temporary objects can increase your storage bill significantly.
• Location: the Cloud Storage bucket location must be in the same location as the GKE cluster because transferring snapshots across different regions might affect performance.

To create a bucket with the required configurations, run the following command:

gcloud storage buckets create gs://BUCKET_NAME \
    --uniform-bucket-level-access \
    --enable-hierarchical-namespace \
    --soft-delete-duration=0d \
    --location=LOCATION

Replace the following:

• BUCKET_NAME: the name of your bucket.
• LOCATION: the location of your bucket.

For a complete list of options for bucket creation, see Flags in the documentation for the buckets create command.

Grant workloads access to the Cloud Storage bucket

By default, GKE doesn't have access to Cloud Storage. To read and write Pod snapshots, you must grant an IAM role that contains the corresponding permissions to the Kubernetes ServiceAccount object that your workload Pods use, or enable multi-tenancy using short-lived tokens.

Grant roles to specific Kubernetes ServiceAccount objects

To grant roles to specific Kubernetes ServiceAccount objects, follow these steps:

1. To communicate with your cluster by using kubectl commands, get credentials for your cluster:

   gcloud container clusters get-credentials CLUSTER_NAME \
       --location=CONTROL_PLANE_LOCATION
   

   Replace the following:

   • CLUSTER_NAME: the name of your cluster.
   • CONTROL_PLANE_LOCATION: the location of the control plane of your cluster.

2. For each Pod, follow these steps:

   1. Create a ServiceAccount object:

      kubectl create serviceaccount SERVICEACCOUNT_NAME \
          --namespace=NAMESPACE
      

      Replace the following:

      • SERVICEACCOUNT_NAME: the name of your ServiceAccount object.
      • NAMESPACE: the namespace for your Pods.

   2. To let the ServiceAccount object read bucket metadata, grant it the Storage Bucket Viewer role (roles/storage.bucketViewer):

      gcloud storage buckets add-iam-policy-binding gs://BUCKET_NAME \
          --member="principal://iam.googleapis.com/projects/PROJECT_NUMBER/locations/global/workloadIdentityPools/PROJECT_ID.s3ns.svc.id.goog/subject/ns/NAMESPACE/sa/SERVICEACCOUNT_NAME" \
          --role="roles/storage.bucketViewer"
      

      Replace the following:

      • BUCKET_NAME: the name of your bucket.
      • PROJECT_NUMBER: your project number.
      • PROJECT_ID: your project ID.
      • NAMESPACE: the namespace for your Pods.
      • SERVICEACCOUNT_NAME: the name of your ServiceAccount object.

   3. To let the ServiceAccount object read and write snapshot objects in the bucket, grant it the Storage Object User role (roles/storage.objectUser):

      gcloud storage buckets add-iam-policy-binding gs://BUCKET_NAME \
          --member="principal://iam.googleapis.com/projects/PROJECT_NUMBER/locations/global/workloadIdentityPools/PROJECT_ID.s3ns.svc.id.goog/subject/ns/NAMESPACE/sa/SERVICEACCOUNT_NAME" \
          --role="roles/storage.objectUser"
      

Enable multi-tenancy using short-lived tokens

As an alternative to granting permissions to individual ServiceAccount objects, you can enable multi-tenancy with short-lived, scope-restricted tokens. This approach helps to avoid the propagation delay associated with manual IAM bindings. Instead of granting permissions to each ServiceAccount object, you perform a one-time grant of the Storage Admin role (roles/storage.admin) on the snapshot storage bucket to the GKE node service account. The node service account then creates short-lived tokens on demand for specific paths.

Enabling tokens with Pod snapshots requires GKE version 1.35.3-gke.1737000 or later.

To enable multi-tenancy, follow these steps:

1. To grant the node service account permission to use the bucket, run the following command:

   gcloud storage buckets add-iam-policy-binding gs://BUCKET_NAME \
       --member="serviceAccount:service-PROJECT_NUMBER@gcp-sa-gkenode.s3ns-system.iam.gserviceaccount.com" \
       --role="roles/storage.admin"
   

   Replace the following:

   • BUCKET_NAME: the name of your bucket.
   • PROJECT_NUMBER: your project number.

2. When you configure storage for snapshots, set the value of the tokenSource field to federatedP4SA.

Grant the Pod snapshot controller access to the Cloud Storage bucket

To let the Pod snapshot controller delete snapshots inside the Cloud Storage bucket, you must grant the Storage Object User role (roles/storage.objectUser) to the GKE service agent on the project. To grant the role, run the following command:

gcloud projects add-iam-policy-binding PROJECT_ID \
    --member="serviceAccount:service-PROJECT_NUMBER@container-engine-robot.s3ns-system.iam.gserviceaccount.com" \
    --role="roles/storage.objectUser" \
    --condition="expression=resource.name.startsWith(\"projects/_/buckets/BUCKET_NAME\"),title=restrict_to_bucket,description=Restricts access to one bucket only"

Replace the following:

• PROJECT_ID: your project ID.
• PROJECT_NUMBER: your project number.
• BUCKET_NAME: the name of your bucket.

Optional: Create managed folders for the Cloud Storage bucket

To isolate permissions for snapshots from mutually untrusted Pods, which is useful in multi-tenant use cases, create managed folders. To set up managed folders, follow these steps:

1. Create a custom IAM role that contains only the necessary permissions for Pod snapshots:

   gcloud iam roles create podSnapshotGcsReadWriter \
       --project=PROJECT_ID \
       --permissions="storage.objects.get,storage.objects.create,storage.objects.delete,storage.folders.create"
   

   Replace PROJECT_ID with your project ID.

2. Grant the Storage Bucket Viewer role (roles/storage.bucketViewer) to all ServiceAccount objects in the target namespace. This role lets the ServiceAccount objects read bucket metadata, but doesn't grant them permissions to read or write the objects in the bucket.

   gcloud storage buckets add-iam-policy-binding gs://BUCKET_NAME \
       --member="principalSet://iam.googleapis.com/projects/PROJECT_NUMBER/locations/global/workloadIdentityPools/PROJECT_ID.s3ns.svc.id.goog/namespace/NAMESPACE" \
       --role="roles/storage.bucketViewer"
   

   Replace the following:

   • BUCKET_NAME: the name of your bucket.
   • PROJECT_NUMBER: your project number.
   • PROJECT_ID: your project ID.
   • NAMESPACE: the namespace for your Pods.

3. For each ServiceAccount object that needs to store Pod snapshots, follow these steps:

   1. Create a managed folder for the ServiceAccount object:

      gcloud storage managed-folders create gs://BUCKET_NAME/FOLDER_PATH/
      

      Replace the following:

      • BUCKET_NAME: the name of your bucket.
      • FOLDER_PATH: the path for the managed folder (for example, my-app-snapshots).

   2. Grant the ServiceAccount object the custom podSnapshotGcsReadWriter role on the managed folder:

      gcloud storage managed-folders add-iam-policy-binding gs://BUCKET_NAME/FOLDER_PATH/ \
          --member="principal://iam.googleapis.com/projects/PROJECT_NUMBER/locations/global/workloadIdentityPools/PROJECT_ID.s3ns.svc.id.goog/subject/ns/NAMESPACE/sa/SERVICEACCOUNT_NAME" \
          --role="projects/PROJECT_ID/roles/podSnapshotGcsReadWriter"
      

      Replace the following:

      • BUCKET_NAME: the name of your bucket.
      • FOLDER_PATH: the path for the managed folder (for example, my-app-snapshots).
      • PROJECT_NUMBER: your project number.
      • PROJECT_ID: your project ID.
      • NAMESPACE: the namespace for your Pods.
      • SERVICEACCOUNT_NAME: the name of your ServiceAccount object.

Configure storage for snapshots

To specify where to store Pod snapshots, create a PodSnapshotStorageConfig custom resource:

1. Save the following manifest as example-pod-snapshot-storage-config.yaml. This example configures GKE to store Pod snapshots in the FOLDER_PATH/ path inside the Cloud Storage bucket BUCKET_NAME:

   apiVersion: podsnapshot.gke.io/v1
   kind: PodSnapshotStorageConfig
   metadata:
     name: example-pod-snapshot-storage-config
   spec:
     snapshotStorageConfig:
       gcs:
         bucket: "BUCKET_NAME"
         path: "FOLDER_PATH"
         tokenSource: "TOKEN_SOURCE"
   

   Replace the following:

   • BUCKET_NAME: the name of your Cloud Storage bucket.
   • FOLDER_PATH: the path for the Cloud Storage managed folder.
   • TOKEN_SOURCE: the identity provider for access. Use podKSA (default) or federatedP4SA for multi-tenancy.

2. Apply the manifest:

   kubectl apply -f example-pod-snapshot-storage-config.yaml
   

What's next

Learn how to trigger a Pod snapshot.