Troubleshoot CMEK and default setting errors

This document describes how to find and mitigate common CMEK configuration errors, and it describes how to identify errors that occur when setting the default resource location.

Troubleshoot setting the default resource location

You attempt to update the default storage location for an organization or folder, but the command fails with an error similar to the following:

ERROR: (gcloud.logging.settings.update) INVALID_ARGUMENT: The KMS key location must match the storage location. Received KMS key location: us-central1, storage location: us-west1
- '@type': type.googleapis.com/google.rpc.DebugInfo
  detail: '[ORIGINAL ERROR] generic::invalid_argument: The KMS key location must match
    the storage location. Received KMS key location: us-central1, storage location:
    us-west1 [google.rpc.error_details_ext] { message: "The KMS key location must
    match the storage location. Received KMS key location: us-central1, storage location:
    us-west1" }'

To resolve this error, set the default storage location for the organization or folder to match the location of the Cloud Key Management Service key for the organization or folder.

Troubleshoot CMEK

As you configure CMEK, the Trusted Cloud project containing the Cloud KMS key is notified of related issues. For example, updates fail when the KMS_KEY_NAME is invalid, when the associated service account doesn't have the required Cloud Key Management Service CryptoKey Encrypter/Decrypter role, or when access to the key is disabled.

After you configure CMEK, at least on of the following occurs:

  • You received a notification from Cloud Logging about CMEK access issues.

  • You notice that CMEK isn't enabled for the _Default and _Required log buckets when you create new Trusted Cloud projects in your organization or in a folder.

  • You receive errors when you read from log buckets with CMEK enabled, or if you try to create or update log buckets.

The notification provides information about the failure and it contains that you can take to mitigate the issue:

Error Recommendation
Cryptographic key permission denied

The Logging service account associated with your Trusted Cloud project doesn't have sufficient IAM permissions to operate on the specified Cloud KMS key. Follow the instructions in the error or see the following documents:
Cryptographic key is disabled The specified Cloud KMS key was disabled. Follow the instructions in the error to re-enable the key.
Cryptographic key was destroyed

The specified Cloud KMS key was destroyed. Follow the instructions or see the following documents:

Identify project that contains the Cloud KMS key

To identify the ID of the Trusted Cloud project that contains the encryption key used by a log bucket, folder, or organization, do the following:

PROJECT 

gcloud logging settings describe --project=PROJECT_ID

Before you run the command, replace PROJECT_ID with the project ID that contains the log bucket.

FOLDER 

gcloud logging settings describe --folder=FOLDER_ID

Before you run the command, replace FOLDER_ID with the ID of the folder.

ORGANIZATION 

gcloud logging settings describe --organization=ORGANIZATION_ID

Before you run the command, replace ORGANIZATION_ID with the ID of the organization.

The previous command returns information similar to the following:

kmsServiceAccountId: KMS_SERVICE_ACCT_NAME@gcp-sa-logging.s3ns-system.iam.gserviceaccount.com
loggingServiceAccountId: SERVICE_ACCT_NAME@gcp-sa-logging.s3ns-system.iam.gserviceaccount.com

For organizations and folders, the following field is also returned:

kmsKeyName: projects/KMS_PROJECT_ID/locations/LOCATION/keyRings/KMS_KEY_RING/cryptoKeys/KEY

The value of the kmsKeyName field include thes Trusted Cloud project that stores the key.

Verify key usability

To verify the key's usability, run the following command to list all keys:

gcloud kms keys list \
--location=KMS_KEY_LOCATION \
--keyring=KMS_KEY_RING

This command returns information about each key in a tabular format. The first line of the output is a list of column names:

NAME PURPOSE ...

Verify that the Cloud KMS key is listed in the command's output as ENABLED, and that the purpose of the key is symmetric encryption: the PURPOSE column must contain ENCRYPT_DECRYPT and the PRIMARY_STATE column must contain ENABLED.

If necessary, create a new key.

Verify permissions configuration

Service accounts that are associated with the organization's CMEK settings must have the Cloud KMS CryptoKey Encrypter/Decrypter role for the configured key.

To list the key's IAM policy, run the following command:

gcloud kms keys get-iam-policy KMS_KEY_NAME

If necessary, add the service account that contains the Cloud KMS CryptoKey Encrypter/Decrypter role to the key.