Use Secret Manager add-on with Google Kubernetes Engine

The integration between Secret Manager and Google Kubernetes Engine (GKE) lets you store sensitive data such as passwords and certificates used by GKE clusters as secrets in Secret Manager.

This page explains how you can use the Secret Manager add-on to access the secrets stored in Secret Manager as volumes mounted in Kubernetes Pods.

This process involves the following steps:

1. Enable the Secret Manager add-on on a new or existing GKE cluster.
2. Configure applications to authenticate to the Secret Manager API.
3. Define which secrets to mount onto Kubernetes Pods using a SecretProviderClass YAML file. The Secret Manager add-on supports both global and regional secrets.
4. Create a volume where the secrets will be mounted. After the volume is attached, applications in the container can access the data in the container file system.

The Secret Manager add-on is derived from the open source Kubernetes Secrets Store CSI Driver and the Google Secret Manager provider. If you're using the open source Secrets Store CSI Driver to access secrets, you can migrate to the Secret Manager add-on. For information, see Migrate from the existing Secrets Store CSI Driver.

Benefits

The Secret Manager add-on provides the following benefits:

• You can use a fully managed and supported solution to access Secret Manager secrets from within GKE without any operational overhead.
• You don't have to write custom code to access secrets stored in Secret Manager.
• You can store and manage all your secrets centrally in Secret Manager and selectively access secrets from GKE pods using the Secret Manager add-on. By doing this, you can use features offered by Secret Manager such as CMEK encryption, fine-grained access control, managed rotation, lifecycle management, and audit logs, along with using Kubernetes features such as passing secrets to containers in the form of mounted volumes.
• The Secret Manager add-on is supported on both Standard clusters and Autopilot clusters.
• The Secret Manager add-on supports nodes that use Container-Optimized OS or Ubuntu node images.

Limitations

The Secret Manager add-on has the following limitations:

• The Secret Manager add-on doesn't support the Sync as Kubernetes Secret feature, which is available in the open source Secrets Store CSI Driver. If you want to synchronize secrets stored in Secret Manager to Kubernetes Secrets, use the integrated secret synchronization feature of Secret Manager. For information, see Synchronize secrets to Kubernetes Secrets.

• The Secret Manager add-on doesn't support Windows Server nodes.

Before you begin

• Enable the Secret Manager and Google Kubernetes Engine APIs.

  Roles required to enable APIs

  To enable APIs, you need the serviceusage.services.enable permission. If you created the project, then you likely already have this permission through the Owner role (roles/owner). Otherwise, you can get this permission through the Service Usage Admin role (roles/serviceusage.serviceUsageAdmin). Learn how to grant roles.

  Enable the APIs

• If you want to use the Google Cloud CLI for this task, install and then initialize the gcloud CLI. If you previously installed the gcloud CLI, get the latest version by running the gcloud components update command.

  You can't manually set up the Secret Manager add-on using the Google Cloud SDK or the Cloud de Confiance console.

• Ensure that your cluster runs GKE version 1.27.14-gke.1042001 or later with a Linux node image.

• If you use a GKE Standard cluster, ensure that your cluster has Workload Identity Federation for GKE enabled. Workload Identity Federation for GKE is enabled by default on an Autopilot cluster. Kubernetes Pods use Workload Identity Federation for GKE to authenticate to the Secret Manager API.

Enable the Secret Manager add-on

You can enable the Secret Manager add-on on both Standard clusters as well as Autopilot clusters.

Enable the Secret Manager add-on on a new GKE cluster

To enable the Secret Manager add-on on cluster creation, do the following:

gcloud container clusters create CLUSTER_NAME \
    --enable-secret-manager \
    --location=LOCATION \
    --cluster-version=VERSION \
    --workload-pool=PROJECT_ID.s3ns.svc.id.goog

gcloud container clusters create CLUSTER_NAME `
    --enable-secret-manager `
    --location=LOCATION `
    --cluster-version=VERSION `
    --workload-pool=PROJECT_ID.s3ns.svc.id.goog

gcloud container clusters create CLUSTER_NAME ^
    --enable-secret-manager ^
    --location=LOCATION ^
    --cluster-version=VERSION ^
    --workload-pool=PROJECT_ID.s3ns.svc.id.goog

gcloud container clusters create-auto CLUSTER_NAME \
    --enable-secret-manager \
    --cluster-version=VERSION \
    --location=LOCATION

gcloud container clusters create-auto CLUSTER_NAME `
    --enable-secret-manager `
    --cluster-version=VERSION `
    --location=LOCATION

gcloud container clusters create-auto CLUSTER_NAME ^
    --enable-secret-manager ^
    --cluster-version=VERSION ^
    --location=LOCATION

After you have enabled the Secret Manager add-on, you can use the Secrets Store CSI Driver in Kubernetes volumes using the driver and provisioner name: secrets-store-gke.csi.k8s.io.

Enable the Secret Manager add-on on an existing GKE cluster

To enable the Secret Manager add-on on an existing cluster, do the following:

gcloud container clusters update CLUSTER_NAME \
    --enable-secret-manager \
    --location=LOCATION \

gcloud container clusters update CLUSTER_NAME `
    --enable-secret-manager `
    --location=LOCATION `

gcloud container clusters update CLUSTER_NAME ^
    --enable-secret-manager ^
    --location=LOCATION ^

Verify the Secret Manager add-on installation

To verify that the Secret Manager add-on is installed on the Kubernetes cluster, run the following command:

  gcloud container clusters describe CLUSTER_NAME --location LOCATION | grep secretManagerConfig -A 4

Replace the following:

• CLUSTER_NAME: the name of the cluster
• LOCATION: the location of your cluster, such as us-central1

Configure automatic rotation of secrets

You can configure the Secret Manager add-on to automatically rotate secrets so that secrets updated in Secret Manager after initial pod deployment are automatically and periodically pushed to the pod. Auto-rotation of mounted secrets lets applications automatically receive updated secrets without requiring a restart or manual intervention. This feature ensures that applications are always using the most up-to-date secrets.

Note the following about configuring auto-rotation of secrets:

• Auto-rotation of secrets is an optional configuration.
• You can configure this feature at the time of creating a new cluster or updating an existing cluster.
• You can configure the auto-rotation frequency by specifying the rotation interval and the rotation interval unit.
• Auto-rotation of secrets is supported in GKE version 1.32.2-gke.1059000 or later.

To configure auto-rotation of secrets, you must enable the enable-secret-manager-rotation feature and configure the rotation interval by setting secret-manager-rotation-interval.

Configure automatic rotation of secrets on a new GKE cluster

To configure auto-rotation of secrets on cluster creation, do the following:

gcloud container clusters create CLUSTER_NAME \
    --cluster-version=VERSION \
    --location=LOCATION \
    --enable-secret-manager \
    --enable-secret-manager-rotation \
    --secret-manager-rotation-interval=ROTATION_INTERVAL
    

gcloud container clusters create CLUSTER_NAME `
    --cluster-version=VERSION `
    --location=LOCATION `
    --enable-secret-manager `
    --enable-secret-manager-rotation `
    --secret-manager-rotation-interval=ROTATION_INTERVAL
    

gcloud container clusters create CLUSTER_NAME ^
    --cluster-version=VERSION ^
    --location=LOCATION ^
    --enable-secret-manager ^
    --enable-secret-manager-rotation ^
    --secret-manager-rotation-interval=ROTATION_INTERVAL
    

gcloud container clusters create CLUSTER_NAME \
    --location=LOCATION \
    --cluster-version=VERSION \
    --workload-pool=PROJECT_ID.s3ns.svc.id.goog \
    --enable-secret-manager \
    --enable-secret-manager-rotation \
    --secret-manager-rotation-interval=ROTATION_INTERVAL
    

gcloud container clusters create CLUSTER_NAME `
    --location=LOCATION `
    --cluster-version=VERSION `
    --workload-pool=PROJECT_ID.s3ns.svc.id.goog `
    --enable-secret-manager `
    --enable-secret-manager-rotation `
    --secret-manager-rotation-interval=ROTATION_INTERVAL
    

gcloud container clusters create CLUSTER_NAME ^
    --location=LOCATION ^
    --cluster-version=VERSION ^
    --workload-pool=PROJECT_ID.s3ns.svc.id.goog ^
    --enable-secret-manager ^
    --enable-secret-manager-rotation ^
    --secret-manager-rotation-interval=ROTATION_INTERVAL
    

Configure automatic rotation of secrets on an existing GKE cluster

To configure auto-rotation of secrets on an existing GKE cluster, do the following:

gcloud container clusters update CLUSTER_NAME \
    --enable-secret-manager \
    --location=LOCATION \
    --enable-secret-manager-rotation \
    --secret-manager-rotation-interval=ROTATION_INTERVAL

gcloud container clusters update CLUSTER_NAME `
    --enable-secret-manager `
    --location=LOCATION `
    --enable-secret-manager-rotation `
    --secret-manager-rotation-interval=ROTATION_INTERVAL

gcloud container clusters update CLUSTER_NAME ^
    --enable-secret-manager ^
    --location=LOCATION ^
    --enable-secret-manager-rotation ^
    --secret-manager-rotation-interval=ROTATION_INTERVAL

Configure applications to authenticate to the Secret Manager API

The Google Secret Manager provider uses the workload identity of the Pod that a secret is mounted onto when authenticating to the Secret Manager API. To allow your applications to authenticate to the Secret Manager API using Workload Identity Federation for GKE, follow these steps:

• Create a Kubernetes ServiceAccount or use an existing Kubernetes ServiceAccount in the same namespace as the Pod onto which you want to mount the secret.

• Create an Identity and Access Management (IAM) allow policy for the secret in Secret Manager.

Pods that use the configured Kubernetes ServiceAccount automatically authenticate as the IAM principal identifier that corresponds to the Kubernetes ServiceAccount when accessing the Secret Manager API.

Create a Kubernetes ServiceAccount

1. Save the following manifest as service-account.yaml:

   apiVersion: v1
   kind: ServiceAccount
   metadata:
     name: KSA_NAME
     namespace: NAMESPACE
   

   Replace the following:

   • KSA_NAME: the name of your new Kubernetes ServiceAccount
   • NAMESPACE: the name of the Kubernetes namespace for the ServiceAccount

2. Apply the manifest:

   kubectl apply -f service-account.yaml
   

3. Create an IAM allow policy that references the new Kubernetes ServiceAccount and grant it permission to access the secret:

   gcloud secrets add-iam-policy-binding SECRET_NAME \
       --role=roles/secretmanager.secretAccessor \
       --member=principal://iam.googleapis.com/projects/PROJECT_NUMBER/locations/global/workloadIdentityPools/PROJECT_ID.s3ns.svc.id.goog/subject/ns/NAMESPACE/sa/KSA_NAME
   

   Replace the following:

   • SECRET_NAME: the name of the secret in Secret Manager
   • PROJECT_NUMBER: your numerical Cloud de Confiance project number
   • PROJECT_ID: the project ID of the Cloud de Confiance by S3NS project that contains your GKE cluster
   • NAMESPACE: the name of the Kubernetes namespace for the ServiceAccount
   • KSA_NAME: the name of your existing Kubernetes ServiceAccount

Use an existing Kubernetes ServiceAccount

Create an IAM allow policy that references the existing Kubernetes ServiceAccount and grant it permission to access the secret:

gcloud secrets add-iam-policy-binding SECRET_NAME \
    --role=roles/secretmanager.secretAccessor \
    --member=principal://iam.googleapis.com/projects/PROJECT_NUMBER/locations/global/workloadIdentityPools/PROJECT_ID.s3ns.svc.id.goog/subject/ns/NAMESPACE/sa/KSA_NAME

Replace the following:

• SECRET_NAME: the name of the secret in Secret Manager
• PROJECT_NUMBER: your numerical Cloud de Confiance project number
• PROJECT_ID: the project ID of the Cloud de Confiance by S3NS project that contains your GKE cluster
• NAMESPACE: the name of the Kubernetes namespace for the ServiceAccount
• KSA_NAME: the name of your existing Kubernetes ServiceAccount

Define which secrets to mount

To specify which secrets to mount as files in the Kubernetes Pod, create a SecretProviderClass YAML manifest and list the secrets to mount and the filename to mount them as. Follow these steps:

1. Save the following manifest as app-secrets.yaml:

   apiVersion: secrets-store.csi.x-k8s.io/v1
   kind: SecretProviderClass
   metadata:
     name: SECRET_PROVIDER_CLASS_NAME
   spec:
     provider: gke
     parameters:
       secrets: |
         - resourceName: "projects/PROJECT_ID/secrets/SECRET_NAME/versions/SECRET_VERSION"
           path: "FILENAME.txt"
   

   Replace the following:

   • SECRET_PROVIDER_CLASS_NAME: the name for your SecretProviderClass object.
   • PROJECT_ID: your project ID.
   • SECRET_NAME: the secret name.
   • SECRET_VERSION: the secret version. The secret version must be in the same region as the cluster.
   • FILENAME.txt: the filename where the secret value will be mounted. You can create multiple files using the resourceName and path variables.

   For a regional secret, resourceName is the complete path to the secret resource that includes the location of the regional secret. For example, "projects/PROJECT_ID/locations/LOCATION/secrets/SECRET_NAME/versions/SECRET_VERSION"

2. Apply the manifest:

   kubectl apply -f app-secrets.yaml -n NAMESPACE
   

   Replace NAMESPACE with the name of the Kubernetes namespace for the ServiceAccount.

3. Verify that the SecretProviderClass object is created:

   kubectl get SecretProviderClasses -n NAMESPACE
   

Configure a volume where the secrets will be mounted

1. Save the following configuration as my-pod.yaml:

   apiVersion: v1
   kind: Pod
   metadata:
     name: POD_NAME
     namespace: NAMESPACE
   spec:
     serviceAccountName: KSA_NAME
     containers:
     - image: IMAGE_NAME
       imagePullPolicy: IfNotPresent
       name: POD_NAME
       resources:
         requests:
           cpu: 100m
       stdin: true
       stdinOnce: true
       terminationMessagePath: /dev/termination-log
       terminationMessagePolicy: File
       tty: true
       volumeMounts:
         - mountPath: "/var/secrets"
           name: mysecret
     volumes:
     - name: mysecret
       csi:
         driver: secrets-store-gke.csi.k8s.io
         readOnly: true
         volumeAttributes:
           secretProviderClass: SECRET_PROVIDER_CLASS_NAME
   

   Replace the following:

   • POD_NAME: the name of the Kubernetes Pod where the secret is mounted
   • NAMESPACE: the name of the Kubernetes namespace for the ServiceAccount
   • KSA_NAME: the Kubernetes ServiceAccount that you set up in the step Configure applications to authenticate to the Secret Manager API
   • IMAGE_NAME: name of the container image
   • SECRET_PROVIDER_CLASS_NAME: the name for your SecretProviderClass object

2. In Standard clusters only, add the following to the template.spec field to place the Pods on node pools that use Workload Identity Federation for GKE.

   Skip this step in Autopilot clusters, which reject this nodeSelector because every node uses Workload Identity Federation for GKE.

   spec:
     nodeSelector:
       iam.gke.io/gke-metadata-server-enabled: "true"
   

3. Apply the configuration to your cluster.

   kubectl apply -f my-pod.yaml
   

This step mounts a volume mysecret at /var/secrets using the CSI driver (secrets-store-gke.csi.k8s.io). This volume references the SecretProviderClass object which acts as the provider.

Migrate from the existing open source Kubernetes Secrets Store CSI Driver

If you use the open source Kubernetes Secrets Store CSI Driver and the Google Secret Manager provider, you can migrate your workloads to the managed Secret Manager add-on.

The managed add-on and the open source driver can run concurrently on the same cluster. This capability lets you migrate individual applications and namespaces incrementally without cluster downtime.

To migrate to the Secret Manager add-on, follow these steps:

1. Enable the Secret Manager add-on on your cluster
2. Create SecretProviderClass manifests
3. Update workload manifests
4. Verify the migration
5. Clean up old open source resources

Step 1: Enable the Secret Manager add-on on your cluster

Enable the Secret Manager add-on on your existing GKE cluster:

gcloud container clusters update CLUSTER_NAME \
    --location=LOCATION \
    --enable-secret-manager \
    [--enable-secret-manager-rotation \
     --secret-manager-rotation-interval=ROTATION_INTERVAL]

Replace the following:

• CLUSTER_NAME: the name of your cluster
• LOCATION: the location of your cluster, such as us-central1
• (Optional) ROTATION_INTERVAL: the frequency at which mounted secrets are automatically updated, such as 120s or 3600s. The minimum allowed value is 60s, and the default is 120s.

Step 2: Create SecretProviderClass manifests

Create a SecretProviderClass manifest with provider: gke for each application. We recommend appending -gke to the resource name so that the old and new SecretProviderClass resources can coexist during rolling updates.

1. Save the following manifest as secret-provider-class-gke.yaml:

   apiVersion: secrets-store.csi.x-k8s.io/v1
   kind: SecretProviderClass
   metadata:
     name: SECRET_PROVIDER_CLASS_NAME-gke # Append -gke to coexist with existing SPC
     namespace: NAMESPACE
   spec:
     provider: gke # Changed from 'gcp'
     parameters:
       secrets: |
         - resourceName: "projects/PROJECT_ID/secrets/SECRET_NAME/versions/SECRET_VERSION"
           path: "FILENAME.txt"
   

   Replace the following:

   • SECRET_PROVIDER_CLASS_NAME-gke: the name for your new SecretProviderClass resource
   • NAMESPACE: the Kubernetes namespace where your workload runs
   • PROJECT_ID: your Cloud de Confiance by S3NS project ID
   • SECRET_NAME: the name of the secret in Secret Manager
   • SECRET_VERSION: the secret version, such as latest or 1
   • FILENAME.txt: the filename where the secret value is mounted

   For regional secrets, specify the full resource path including the location: "projects/PROJECT_ID/locations/LOCATION/secrets/SECRET_NAME/versions/SECRET_VERSION"

2. Apply the new SecretProviderClass manifest:

   kubectl apply -f secret-provider-class-gke.yaml
   

Step 3: Update workload manifests

Update your Pod or Deployment manifests to use the managed driver (secrets-store-gke.csi.k8s.io) and reference the new SecretProviderClass name.

1. Update the volumes section of your workload manifest (such as deployment.yaml):

   spec:
     containers:
     - name: CONTAINER_NAME
       image: IMAGE_NAME
       volumeMounts:
       - name: secrets-volume
         mountPath: "/var/secrets"
         readOnly: true
     volumes:
     - name: secrets-volume
       csi:
         driver: secrets-store-gke.csi.k8s.io # Changed from 'secrets-store.csi.k8s.io'
         readOnly: true
         volumeAttributes:
           secretProviderClass: SECRET_PROVIDER_CLASS_NAME-gke # Reference new SPC name
   

   Replace the following:

   • CONTAINER_NAME: the name of the container in your workload
   • IMAGE_NAME: the name of your container image
   • SECRET_PROVIDER_CLASS_NAME-gke: the name for your new SecretProviderClass resource

2. Apply the updated workload manifest to trigger a rolling deployment:

   kubectl apply -f deployment.yaml
   

Step 4: Verify the migration

Verify that your workloads are successfully using the managed add-on:

1. Check that your pods have rolled out and reached the Running state:

   kubectl get pods -n NAMESPACE
   

   Replace NAMESPACE with the Kubernetes namespace where your workload runs.

2. Verify that secrets are properly mounted and accessible within the container:

   kubectl exec -it POD_NAME -n NAMESPACE -- cat /var/secrets/FILENAME.txt
   

   Replace the following:

   • POD_NAME: the name of your Kubernetes Pod
   • NAMESPACE: the Kubernetes namespace where your workload runs
   • FILENAME.txt: the mounted secret filename configured in your SecretProviderClass

3. (Optional) If you enabled automatic rotation, verify secret updates:

   1. Add a new version of the secret in Secret Manager.
   2. Wait for the configured rotation interval, such as 120 seconds.

   3. Inspect the mounted secret file again inside the pod to verify that the content updated without restarting the pod:

      kubectl exec -it POD_NAME -n NAMESPACE -- cat /var/secrets/FILENAME.txt
      

Step 5: Clean up old open source resources

1. Delete the old SecretProviderClass resources:

   kubectl delete secretproviderclass OLD_SECRET_PROVIDER_CLASS_NAME -n NAMESPACE
   

   Replace the following:

   • OLD_SECRET_PROVIDER_CLASS_NAME: the name of your legacy SecretProviderClass resource, such as my-app-secrets
   • NAMESPACE: the Kubernetes namespace where your workload runs

2. If you deployed the open source Google Secret Manager provider plugin using a manifest, delete the DaemonSet:

   kubectl delete -f https://raw.githubusercontent.com/GoogleCloudPlatform/secrets-store-csi-driver-provider-gcp/main/deploy/provider-gcp-plugin.yaml
   

3. If you installed the open source Secrets Store CSI Driver using Helm, uninstall the Helm release:

   helm uninstall csi-secrets-store -n kube-system
   

Disable the Secret Manager add-on

To disable the Secret Manager add-on on an existing Standard cluster or on an Autopilot cluster, run the following command:

gcloud container clusters update CLUSTER_NAME \
    --no-enable-secret-manager \
    --region=REGION \

gcloud container clusters update CLUSTER_NAME `
    --no-enable-secret-manager `
    --region=REGION `

gcloud container clusters update CLUSTER_NAME ^
    --no-enable-secret-manager ^
    --region=REGION ^

What's next

• Read the Secret Manager GKE integration release notes.