Configure message encryption

This document discusses how to configure customer-managed encryption keys (CMEK) for Pub/Sub.

Pub/Sub encrypts messages with Google Cloud-powered encryption keys by default. No additional setup is required to use Google Cloud-powered encryption keys.

About CMEK

CMEKs are encryption keys that you own and are managed and stored in Cloud Key Management Service (Cloud KMS). If you need more control over the encryption keys used to protect Pub/Sub data, you can use CMEKs. Some organizations also mandate the use of CMEKs.

CMEKs give you full control over your encryption keys, letting you manage their lifecycle, rotation, and access policies. When you configure Pub/Sub with a CMEK, the service automatically encrypts all data using the specified key. Cloud KMS usage for CMEK might incur additional costs depending on your usage patterns.

Every message is encrypted at the following states and layers:

  • At rest

    • Hardware layer
    • Infrastructure layer
    • Application layer

  • In transit

At the application layer, Pub/Sub individually encrypts incoming messages as soon as the message is received. This implementation adds the following features:

CMEK for Pub/Sub

Pub/Sub uses the envelope encryption pattern with CMEK. In this approach, the messages are not encrypted by Cloud KMS. Instead Cloud KMS is used to encrypt Data Encryption Keys (DEKs) created by Pub/Sub for each topic. These DEKs are stored only in encrypted, or wrapped, form by Pub/Sub. Before storing a DEK, the service sends the DEK to Cloud KMS to be encrypted with the key encryption key (KEK) specified on the topic. A new DEK is generated for each topic approximately every six hours.

Before Pub/Sub publishes messages to a subscription, it encrypts them using the newest DEK that was generated for the topic. Pub/Sub decrypts the messages shortly before they are delivered to subscribers.

Before you begin

You can configure CMEK for Pub/Sub using the Trusted Cloud console or the Google Cloud CLI.

Complete the following tasks:

  • Enable the Cloud KMS API.

  • Create a key ring and a key in Cloud KMS. Keys and key rings cannot be deleted.

For instructions on how to accomplish these tasks, see the Cloud KMS quickstart guide.

Required roles and permissions to configure CMEK

Pub/Sub uses a Trusted Cloud service agent to access Cloud KMS. The service agent is maintained internally by Pub/Sub for each project, and is not visible on the Service Accounts page in the Trusted Cloud console by default.

The Pub/Sub service agent has the form service-${PROJECT_NUMBER}@gcp-sa-pubsub.s3ns-system.iam.gserviceaccount.com.

Pub/Sub requires specific permissions to encrypt and decrypt data using CMEK.

Complete the following steps to set up the required access:

  • Grant the Pub/Sub service agent the Cloud KMS Crypto Key Encrypter/Decrypter (roles/cloudkms.cryptoKeyEncrypterDecrypter) role.

    gcloud kms keys add-iam-policy-binding CLOUD_KMS_KEY_NAME \
    --member=serviceAccount:service-PROJECT_NUMBER@gcp-sa-pubsub.s3ns-system.iam.gserviceaccount.com \
    --role=roles/cloudkms.cryptoKeyEncrypterDecrypter

    Replace the following:

    • CLOUD_KMS_KEY_NAME: The name of the Cloud KMS key.

      The key is of the format projects/PROJECT_ID/locations/LOCATION/keyRings/KEY_RING/cryptoKeys/CRYPTO_KEY.

      An example is projects/test-project/locations/us-central1/keyRings/test-keyring/cryptoKeys/test-key.

    • PROJECT_NUMBER: The project number of the Pub/Sub project.

For more information about granting IAM roles, see Granting roles on a resource.

Configure a topic with CMEK

You can configure CMEK for a topic using the Trusted Cloud console or gcloud CLI.

Console

To create a topic with CMEK, follow these steps:

  1. In the Trusted Cloud console, go to the Pub/Sub Topics page.

    Go to Topics

  2. Click Create topic.

  3. In the Topic ID field, enter an ID for your topic.

    For more information about naming topics, see the naming guidelines.

  4. For Encryption, click Cloud KMS key.

  5. Select the key type. If you don't see the Select a customer-managed key dropdown, ensure that you have enabled the Cloud KMS API for the project.

  6. Click Create topic.

gcloud

To create a topic with CMEK, run the gcloud pubsub topics create command:

    gcloud pubsub topics create TOPIC_ID --topic-encryption-key=ENCRYPTION_KEY

Replace the following:

Update CMEK for a topic

You have the flexibility to change the CMEK linked to a Pub/Sub topic. You can use gcloud CLI to update the CMEK. However, this change doesn't apply retroactively.

Messages published to the topic before the key changes remain encrypted with the original key. If a topic was created without a CMEK, you can add one later. Existing messages continue to be protected with the default Google Cloud-powered encryption keys. Changing a topic's CMEK does not re-encrypt previously published messages. These messages continue to be protected with the key that they were originally encrypted with.

Pub/Sub has a caching mechanism for keys that lasts approximately 5 minutes. It may take up to this duration for Pub/Sub to recognize and start using the new key version.

Audit logs

Cloud KMS produces audit logs when keys are enabled, disabled, or used by Pub/Sub to encrypt and decrypt messages. This is useful in debugging issues with publish or delivery availability.

Cloud KMS keys are attached to audit logs for Pub/Sub topic resources. Pub/Sub does not include any other Cloud KMS-related information.

Monitoring and troubleshooting

Issues with key access can have these effects:

  • Delays in message delivery

  • Publish errors

Monitor publish and pull request errors using the following metrics, grouped by response_class and response_code:

  • topic/send_request_count
  • subscription/pull_request_count
  • subscription/streaming_pull_response_count

StreamingPull response has a 100% error rate. This is an indication that the stream has ended, not that requests are failing. To monitor StreamingPull, look for the FAILED_PRECONDITION response code.

Publishing and message delivery can fail with FAILED_PRECONDITION errors for multiple reasons.

For push subscriptions, there is no way to directly detect CMEK-specific delivery issues. Instead:

  • Monitor the size and age of the backlog of a push subscription using subscription/num_unacked_messages.

  • Monitor subscription/oldest_unacked_message_age for unusual spikes.

  • Use publish errors and CMEK audit logs to spot issues.

Disabling and re-enabling keys

There are two ways to prevent Pub/Sub from decrypting your message data:

  • Recommended: Disable the Cloud KMS key you've associated with the topic using Pub/Sub. This approach affects only the Pub/Sub topics and subscriptions that are associated with that specific key.

  • Revoke the Pub/Sub CryptoKey Encrypter/Decrypter role from the Pub/Sub service account (service-$PROJECT_NUMBER@gcp-sa-pubsub.s3ns-system.iam.gserviceaccount.com) using IAM. This approach affects all of the project's Pub/Sub topics and the subscriptions that contain messages encrypted using CMEK.

Although neither operation confirms instantaneous access revocation, IAM changes generally propagate faster. To learn more, see Cloud KMS resource consistency and Access change propagation.

When Pub/Sub cannot access a Cloud KMS key, message publishing and delivery with StreamingPull or pull fails with FAILED_PRECONDITION errors. Message delivery to push endpoints will stop. To resume delivery and publishing, restore access to the Cloud KMS key.

Once the Cloud KMS key is accessible to Pub/Sub, publishing is available within 12 hours and message delivery resumes within 2 hours.

Although intermittent outages of less than a minute for Cloud KMS are unlikely to significantly interrupt publishing and delivery, extended Cloud KMS unavailability has the same effect as key revocation.