This document explains how to install the open source Agent Sandbox controller on a Google Kubernetes Engine (GKE) cluster to run isolated agent workloads in dedicated Kubernetes Pods. You also learn how to create a Sandbox resource that runs untrusted code in the gVisor runtime. For most agent workloads, use Agent Substrate on GKE instead, because it can run more agents on the same nodes. Use Agent Sandbox on GKE when each agent needs a dedicated Pod, GPUs or TPUs, a full Kubernetes PodSpec, or a GKE Autopilot cluster. To compare Agent Sandbox on GKE with Agent Substrate on GKE and choose an architecture for your AI agent workloads, see Plan an architecture for AI agent workloads on GKE.
For an overview of how Agent Sandbox on GKE isolates untrusted AI-generated code, see About Agent Sandbox on GKE.
Costs
Agent Sandbox on GKE is open source software that you can use at no extra charge. GKE pricing applies to the resources that you create, including the compute resources that the Agent Sandbox controller uses.
To avoid unnecessary charges, ensure that you disable GKE or delete the project after you have completed this document.
Before you begin
-
In the Cloud de Confiance console, on the project selector page, select or create a Cloud de Confiance project.
Roles required to select or create a project
- Select a project: Selecting a project doesn't require a specific IAM role—you can select any project that you've been granted a role on.
-
Create a project: To create a project, you need the Project Creator role
(
roles/resourcemanager.projectCreator), which contains theresourcemanager.projects.createpermission. Learn how to grant roles.
-
Verify that billing is enabled for your Cloud de Confiance project.
Enable the Google Kubernetes Engine API, if it is not already enabled.
Roles required to enable APIs
To enable APIs, you need the
serviceusage.services.enablepermission. If you created the project, then you likely already have this permission through the Owner role (roles/owner). Otherwise, you can get this permission through the Service Usage Admin role (roles/serviceusage.serviceUsageAdmin). Learn how to grant roles.-
In the Cloud de Confiance console, activate Cloud Shell.
Define environment variables
To simplify the commands that you run in this document, you can set environment variables in Cloud Shell. In Cloud Shell, define the following useful environment variables by running the following commands:
export CLUSTER_NAME="agent-sandbox-cluster"
export LOCATION="us-central1"
export AGENT_SANDBOX_VERSION="v1.0.6"
export NODE_POOL_NAME="agent-sandbox-pool"
export MACHINE_TYPE="e2-standard-2"
Here's an explanation of these environment variables:
CLUSTER_NAME: the name of your GKE cluster—for example,agent-sandbox-cluster.LOCATION: the Cloud de Confiance by S3NS region or zone where your GKE cluster is created. If you create an Autopilot cluster, set this to the region (for example,us-central1). Autopilot clusters are always regional. If you create a Standard cluster, set this to the zone (for example,us-central1-a). To reduce costs, this document creates a zonal cluster, which uses fewer nodes than a regional cluster.AGENT_SANDBOX_VERSION: the version of the open source Agent Sandbox controller to install. This document uses version 1.0.6. To see other versions, go to the Agent Sandbox releases page.NODE_POOL_NAME: the name of the node pool that will run your agent Pods—for example,agent-sandbox-pool. This variable is only required if you are creating a GKE Standard cluster.MACHINE_TYPE: the machine type of the nodes in your cluster's default node pool and in the gVisor node pool—for example,e2-standard-2. For details about different machine series and choosing between different options, see the Machine families resource and comparison guide. This variable is only required if you are creating a GKE Standard cluster.
Create a GKE cluster
Agent Sandbox on GKE isolates agent workloads by using GKE Sandbox with gVisor or microVMs. If you don't specify a GKE Sandbox runtime (gVisor or microVMs), your Pods run without kernel isolation, so don't use them to run untrusted code. This document uses gVisor. Autopilot clusters enable gVisor automatically. In Standard clusters, you create a node pool that has gVisor enabled. To use microVMs instead, see Enable the microVM sandbox type. The recommended security policy allows only gVisor, so it rejects Sandbox resources that use microVMs.
Autopilot
To create an Autopilot cluster, make sure that the LOCATION
environment variable is set to a region, such as us-central1, and then
run the following command:
gcloud container clusters create-auto ${CLUSTER_NAME} \
--location=${LOCATION}
Standard
To create a Standard cluster, you create the cluster and then add a node pool with gVisor enabled. To reduce costs, this document creates a zonal cluster with a single node per pool.
Set the
LOCATIONenvironment variable to a zone, such asus-central1-a:export LOCATION="us-central1-a"Create the cluster:
gcloud container clusters create ${CLUSTER_NAME} \ --location=${LOCATION} \ --machine-type=${MACHINE_TYPE} \ --num-nodes=1Create a separate node pool with gVisor enabled:
gcloud container node-pools create ${NODE_POOL_NAME} \ --cluster=${CLUSTER_NAME} \ --machine-type=${MACHINE_TYPE} \ --location=${LOCATION} \ --num-nodes=1 \ --image-type=cos_containerd \ --sandbox=type=gvisorTo use microVMs instead of gVisor, create the node pool as described in Enable the microVM sandbox type. The rest of this document uses gVisor.
To use an existing Standard cluster instead, make sure that the cluster
includes a node pool that has gVisor enabled. If the cluster doesn't have such a
node pool, create one by running the preceding
gcloud container node-pools create command.
To enable the kubectl command-line tool to connect to the cluster, get
the cluster credentials:
gcloud container clusters get-credentials ${CLUSTER_NAME} \
--location=${LOCATION}
Install Agent Sandbox
To install Agent Sandbox, apply a release manifest from the
Agent Sandbox GitHub repository. The
sandbox-with-extensions.yaml manifest installs the
Agent Sandbox controller and the CustomResourceDefinitions for
the Sandbox resource and for the SandboxTemplate, SandboxClaim, and
SandboxWarmPool extension resources.
Apply the release manifest:
kubectl apply -f https://github.com/kubernetes-sigs/agent-sandbox/releases/download/${AGENT_SANDBOX_VERSION}/sandbox-with-extensions.yamlThe manifest creates the
agent-sandbox-systemnamespace and deploys the controller in that namespace.Wait until the controller is ready:
kubectl rollout status deployment/agent-sandbox-controller \ --namespace=agent-sandbox-system \ --timeout=120s
You can also install Agent Sandbox by using Helm. For instructions, see the Agent Sandbox Helm chart README.
Verify the installation
Verify that the Agent Sandbox CustomResourceDefinitions are installed:
kubectl get crd \ sandboxes.agents.x-k8s.io \ sandboxtemplates.extensions.agents.x-k8s.io \ sandboxclaims.extensions.agents.x-k8s.io \ sandboxwarmpools.extensions.agents.x-k8s.ioThe output lists the four CustomResourceDefinitions.
Verify that the controller is running:
kubectl get deployment agent-sandbox-controller \ --namespace=agent-sandbox-systemWhen the controller is running, the
READYcolumn shows1/1.
Apply the recommended security policy
By default, Agent Sandbox on GKE doesn't restrict the
configuration of the Pods that run your agents. To help enforce isolation
settings for your agent Pods, apply the recommended secure-sandbox-policy
ValidatingAdmissionPolicy
from the Agent Sandbox GitHub repository. This policy rejects any
Sandbox resource that doesn't meet the Agent Sandbox on
GKE deployment requirements.
Apply the policy:
kubectl apply -f https://raw.githubusercontent.com/kubernetes-sigs/agent-sandbox/${AGENT_SANDBOX_VERSION}/examples/policy/vap/secure-sandbox-policy.yamlApply the policy binding. The binding enforces the policy in all namespaces except system namespaces, such as
kube-system:kubectl apply -f https://raw.githubusercontent.com/kubernetes-sigs/agent-sandbox/${AGENT_SANDBOX_VERSION}/examples/policy/vap/secure-sandbox-binding.yaml
The policy validates Sandbox resources. When you create a SandboxWarmPool resource or a SandboxClaim resource, the controller creates Sandbox resources from a SandboxTemplate resource. If the template doesn't meet the requirements, the policy rejects the Sandbox resource, and the error appears in the controller logs. To view the logs, run the following command:
kubectl logs deployment/agent-sandbox-controller \
--namespace=agent-sandbox-system
You own this policy, and you can modify it to fit your workloads. For more information, see Sandbox security policies.
Agent Sandbox on GKE deployment requirements
If you apply the recommended security policy, your workload manifests, such as a Sandbox or SandboxTemplate resource, must include specific security and configuration settings. If a Sandbox resource doesn't meet these requirements, the admission controller rejects it. The container requirements apply to all containers in the Pod, including init containers and ephemeral containers.
Required configuration
Your deployment manifest must include the following settings:
runtimeClassName: gvisor: runs the Pod in the gVisor runtime of GKE Sandbox. The policy rejects Sandbox resources that use any other runtime, including microVMs.automountServiceAccountToken: false: prevents the Pod from automatically mounting the default service account token.securityContext.runAsNonRoot: true: ensures the container does not run as the root user.securityContext.capabilities.drop: ["ALL"]: drops all Linux capabilities from the container.resources.limits: you must specify CPU and memory limits to prevent potential denial-of-service (DoS) scenarios.nodeSelector: must targetsandbox.gke.io/runtime: gvisor.tolerations: must include a toleration for thesandbox.gke.io/runtime=gvisor:NoScheduletaint.
Prohibited configuration
Your deployment manifest must not include any of the following:
hostNetwork: true,hostPID: true, orhostIPC: true.privileged: truein container security contexts.procMount: Unmaskedin container security contexts.HostPathvolumes.- Added capabilities (
capabilities.add). hostPortsettings.- Custom sysctls.
- Projected volumes for service account tokens or certificates.
Deploy the Sandbox Router (optional)
The Sandbox Router is an HTTP proxy that routes requests from clients, such as the Agent Sandbox Python client, to the Pods that run your agents. If you plan to use the Python client, for example as described in Isolate AI code execution with Agent Sandbox on GKE, deploy the router.
You deploy the router in the agent-sandbox-system namespace. By default, the
network policy that Agent Sandbox on GKE creates for
each SandboxTemplate resource allows incoming traffic to agent Pods only from
the router Pods in this namespace.
Create the router's service account and the RBAC permissions that let the router look up agent Pods:
kubectl apply \ -f https://raw.githubusercontent.com/kubernetes-sigs/agent-sandbox/${AGENT_SANDBOX_VERSION}/sandbox-router/deploy/serviceaccount.yaml \ -f https://raw.githubusercontent.com/kubernetes-sigs/agent-sandbox/${AGENT_SANDBOX_VERSION}/sandbox-router/deploy/rbac.yamlDeploy the router. The following command sets the router image to the same version as the controller:
curl -sSL https://raw.githubusercontent.com/kubernetes-sigs/agent-sandbox/${AGENT_SANDBOX_VERSION}/sandbox-router/deploy/deployment.yaml \ | sed "s|sandbox-router-go:latest|sandbox-router-go:${AGENT_SANDBOX_VERSION}|" \ | kubectl apply -f -Create the router's Service:
kubectl apply -f https://raw.githubusercontent.com/kubernetes-sigs/agent-sandbox/${AGENT_SANDBOX_VERSION}/sandbox-router/deploy/service.yamlWait until the router is ready:
kubectl rollout status deployment/sandbox-router \ --namespace=agent-sandbox-system \ --timeout=120s
This router configuration is for development and testing. For production settings, such as TLS, replica counts, and network policies for the router, see the Sandbox Router deployment README.
Export controller metrics (optional)
The Agent Sandbox controller exports Prometheus metrics at the
/metrics path on the metrics port (port 8080). To collect these metrics
by using Google Cloud Managed Service for Prometheus and to query them in Cloud Monitoring,
apply a PodMonitoring custom resource.
GKE enables managed collection by default on Autopilot clusters that run GKE version 1.25 or later and on Standard clusters that run GKE version 1.27 or later. If your cluster doesn't have managed collection enabled, see Enable managed collection manually.
Because Prometheus counters and histograms don't record activity that occurred before collection started, apply the PodMonitoring resource after you install the controller so that Cloud Monitoring captures early cluster and Sandbox resource events.
In Cloud Shell, create a file named
agent-sandbox-monitoring.yamlwith the following content:apiVersion: monitoring.googleapis.com/v1 kind: PodMonitoring metadata: name: agent-sandbox-controller namespace: agent-sandbox-system spec: selector: matchLabels: app: agent-sandbox-controller endpoints: - port: metrics path: /metrics interval: 30sApply the manifest:
kubectl apply -f agent-sandbox-monitoring.yaml
After you apply the PodMonitoring resource, Google Cloud Managed Service for Prometheus scrapes the controller metrics every 30 seconds. You can query these metrics with PromQL in Cloud Monitoring by using Metrics Explorer.
In Cloud Monitoring, Google Cloud Managed Service for Prometheus reserves the
namespace label for the namespace of the scraped controller Pod
(agent-sandbox-system). For metrics that define their own namespace label,
Google Cloud Managed Service for Prometheus renames the metric label to
exported_namespace. To filter or group metrics by the Sandbox resource's
namespace, use the exported_namespace label. For more information, see
Reserved labels.
For example, you can query the following metrics:
agent_sandbox_build_info: build information about the controller, such as the Git commit and version.agent_sandboxes: the point-in-time number of Sandbox resources in the cluster. This metric has labels for readiness, expiration, launch type, owner, creator, template, and the Sandbox resource's namespace (exported_namespace). For example, the following PromQL query groups Sandbox resources by namespace and readiness:sum by (exported_namespace, ready_condition) (agent_sandboxes)agent_sandbox_claim_controller_startup_latency_ms: a histogram that measures the latency in milliseconds from when the controller first observes a SandboxClaim resource until the claim reaches theReadystatus. Because this metric is a Prometheus histogram, querying the base metric name returns no data directly. You must query its_bucket,_sum, or_countseries. For example, the following PromQL query calculates the 95th percentile startup latency over a five-minute rate window:histogram_quantile( 0.95, sum by (le) ( rate(agent_sandbox_claim_controller_startup_latency_ms_bucket[5m]) ) )
For more information about exported metrics and the optional components that they depend on, see the Controller Metrics Reference.
Deploy a sandboxed environment
To deploy a sandboxed environment, define a SandboxTemplate resource and keep pre-warmed Sandbox resources ready by using a SandboxWarmPool resource. You can then request a Sandbox resource from this warm pool by using a SandboxClaim resource. Alternatively, you can create a Sandbox resource directly, but this approach doesn't support warm pools.
SandboxTemplate, SandboxWarmPool, SandboxClaim, and Sandbox are Kubernetes custom resources.
Recommended: Create SandboxTemplate and SandboxWarmPool resources
The SandboxTemplate resource acts as a reusable blueprint. The SandboxWarmPool resource maintains a specified number of pre-warmed Sandbox resources with running Pods that are ready to be claimed. This warm pool reduces startup latency.
To deploy a sandboxed environment by creating SandboxTemplate and SandboxWarmPool resources, complete the following steps:
In Cloud Shell, create a file named
sandbox-template.yamlwith the following content:apiVersion: extensions.agents.x-k8s.io/v1beta1 kind: SandboxTemplate metadata: name: python-runtime-template namespace: default spec: podTemplate: metadata: labels: sandbox-type: python-runtime spec: runtimeClassName: gvisor # Required automountServiceAccountToken: false # Required securityContext: runAsNonRoot: true # Required nodeSelector: sandbox.gke.io/runtime: gvisor # Required tolerations: - key: "sandbox.gke.io/runtime" value: "gvisor" effect: "NoSchedule" # Required containers: - name: runtime image: registry.k8s.io/agent-sandbox/python-runtime-sandbox:v0.1.0 ports: - containerPort: 8888 resources: requests: cpu: "250m" memory: "512Mi" limits: cpu: "500m" memory: "1Gi" # Required securityContext: capabilities: drop: ["ALL"] # Required restartPolicy: OnFailureApply the SandboxTemplate manifest:
kubectl apply -f sandbox-template.yamlCreate a file named
sandbox-warmpool.yamlwith the following content:apiVersion: extensions.agents.x-k8s.io/v1beta1 kind: SandboxWarmPool metadata: name: python-runtime-warmpool namespace: default labels: app: python-runtime-warmpool spec: replicas: 2 sandboxTemplateRef: # This must match the name of the SandboxTemplate. name: python-runtime-templateApply the SandboxWarmPool manifest:
kubectl apply -f sandbox-warmpool.yamlWait until the warm pool's Sandbox resources and their Pods are ready:
kubectl wait sandboxwarmpool python-runtime-warmpool \ --for=jsonpath='{.status.readyReplicas}'=2 \ --timeout=600sIf you create a SandboxClaim resource before the pool's Sandbox resources and their Pods are ready, the controller creates a new Sandbox resource for the claim instead of assigning one from the pool. In an Autopilot cluster, the Pods can take several minutes to become ready because GKE might need to create nodes for them.
Create a SandboxClaim resource
The SandboxClaim resource requests a Sandbox resource from the warm pool. Because you created a warm pool, the controller assigns a ready Sandbox resource (and its running Pod) from the pool to the claim instead of creating a new one.
To request a Sandbox resource from the warm pool by creating a SandboxClaim resource, complete the following steps:
Create a file named
sandbox-claim.yamlwith the following content:apiVersion: extensions.agents.x-k8s.io/v1beta1 kind: SandboxClaim metadata: name: sandbox-claim namespace: default spec: warmPoolRef: # This must match the name of the SandboxWarmPool. name: python-runtime-warmpoolApply the SandboxClaim manifest:
kubectl apply -f sandbox-claim.yamlVerify that the Sandbox resource, claim, and warm pool are ready:
kubectl get sandboxwarmpool,sandboxclaim,sandbox,pod
Alternative: Create a Sandbox resource directly
If you don't need the fast startup times provided by warm pools, you can deploy a Sandbox resource directly without using templates.
To deploy a sandboxed environment by creating a Sandbox resource directly, complete the following steps:
Create a file named
sandbox.yamlwith the following content:apiVersion: agents.x-k8s.io/v1beta1 kind: Sandbox metadata: name: sandbox-example-2 spec: podTemplate: metadata: labels: sandbox: sandbox-example spec: runtimeClassName: gvisor restartPolicy: OnFailure automountServiceAccountToken: false # Required securityContext: runAsNonRoot: true # Required runAsUser: 1000 # Required if image defaults to root (for example, busybox) nodeSelector: sandbox.gke.io/runtime: gvisor tolerations: - key: "sandbox.gke.io/runtime" value: "gvisor" effect: "NoSchedule" # Required containers: - name: my-container image: busybox command: ["/bin/sh", "-c"] args: ["sleep 3600000; echo 'Container finished successfully'; exit 0"] securityContext: capabilities: drop: ["ALL"] # Required allowPrivilegeEscalation: false resources: limits: cpu: "100m" memory: "128Mi" # RequiredApply the Sandbox manifest:
kubectl apply -f sandbox.yamlVerify that the Sandbox resource is running:
kubectl get sandbox
Upgrade Agent Sandbox
To upgrade Agent Sandbox, apply the release manifest for the new version:
export AGENT_SANDBOX_VERSION="NEW_VERSION"
kubectl apply -f https://github.com/kubernetes-sigs/agent-sandbox/releases/download/${AGENT_SANDBOX_VERSION}/sandbox-with-extensions.yaml
Replace NEW_VERSION with the version that you want to install, such as a
version listed on the
Agent Sandbox releases page.
Before you upgrade, review the release notes for the new version. If your
cluster runs a version of Agent Sandbox earlier than 1.0.0, you
must migrate your resources to the v1beta1 API before you upgrade, because
version 1.0.0 removed the v1alpha1 API. For instructions, see the
API migration guide.
If you deployed the Sandbox Router or applied the recommended security policy, repeat those steps for the new version.
Uninstall Agent Sandbox
Check for Agent Sandbox resources that are still in use:
kubectl get sandboxes,sandboxclaims,sandboxwarmpools,sandboxtemplates \ --all-namespacesIf you applied the recommended security policy, delete the policy binding and the policy:
kubectl delete validatingadmissionpolicybinding secure-sandbox-binding kubectl delete validatingadmissionpolicy secure-sandbox-policyIf you deployed the Sandbox Router, delete its cluster-wide RBAC resources:
kubectl delete -f https://raw.githubusercontent.com/kubernetes-sigs/agent-sandbox/${AGENT_SANDBOX_VERSION}/sandbox-router/deploy/rbac.yamlThe router's other resources are in the
agent-sandbox-systemnamespace, which the next step deletes.To delete the Agent Sandbox controller, its namespace, and its CustomResourceDefinitions, delete the release manifest for the version that you installed:
kubectl delete -f https://github.com/kubernetes-sigs/agent-sandbox/releases/download/${AGENT_SANDBOX_VERSION}/sandbox-with-extensions.yaml
Clean up resources
To avoid incurring charges to your Cloud de Confiance by S3NS account, delete the GKE cluster that you created.
gcloud container clusters delete ${CLUSTER_NAME} \
--location=${LOCATION} \
--quiet
If you created an Autopilot cluster, the location is the region (for
example, us-central1). If you created a Standard cluster, the
location is the zone (for example, us-central1-a).
What's next
- Learn how to save and restore Agent Sandbox on GKE environments with Pod snapshots.
- Learn how GKE Sandbox isolates workloads by using gVisor or microVMs.
- Learn more about GKE security.
- Explore the Agent Sandbox open-source project on GitHub.
- Learn how to use open source Kata Containers with Agent Sandbox on GKE. Kata Containers isn't a Cloud de Confiance product. If you install and use this software, then you're responsible for management and troubleshooting. Google's support and SLAs don't apply to Kata Containers.