Configure log buckets

This document describes how to create and manage Cloud Logging buckets using the Cloud de Confiance console, the Google Cloud CLI, and the Logging API. It also provides instructions for creating and managing log buckets at the Cloud de Confiance project level. You can't create log buckets at the folder or organization level; however, Cloud Logging automatically creates _Default and _Required log buckets at the folder and organization level for you.

For a conceptual overview of buckets, see Routing and storage overview: Log buckets.

This document doesn't describe how to create a log bucket that uses a customer-managed encryption key (CMEK). If you are interested in that topic, then see Configure CMEK for logs storage.

Before you begin

To get started with buckets, do the following:

Create a bucket

You can create a maximum of 100 buckets per Cloud de Confiance project.

To create a user-defined log bucket for your Cloud de Confiance project, do the following:

Cloud de Confiance console

To create a log bucket in your Cloud de Confiance project, do the following:

  1. In the Cloud de Confiance console, go to the Logs Storage page:

    Go to Logs Storage

    If you use the search bar to find this page, then select the result whose subheading is Logging.

  2. Click Create log bucket.

  3. Enter a Name and Description for your bucket.

  4. Optional: To select the storage region for your logs, click the Select log bucket region menu and select a region. If you don't select a region, then the global region is used, which means that the logs could be physically located in any supported region.

  5. Click Create bucket.

    It might take a moment for these steps to complete.

gcloud

To create a log bucket, run the gcloud logging buckets create command:

gcloud logging buckets create BUCKET_ID --location=LOCATION

For example, if you want to create a bucket with the BUCKET_ID my-bucket in the global region, your command would look like the following:

gcloud logging buckets create my-bucket --location global --description "My first bucket"

REST

To create a bucket, use the projects.locations.buckets.create method. Prepare the arguments to the method as follows:

  1. Set the parent parameter to be the resource in which to create the bucket: projects/PROJECT_ID/locations/LOCATION

    The variable LOCATION refers to the region in which you want your logs to be stored.

    For example, if you want to create a bucket for project my-project in the in the global region, your parent parameter would look like this: projects/my-project/locations/global

  2. Set the bucketId parameter; for example, my-bucket.

  3. Call the synchronous method projects.locations.buckets.create to create the bucket.

After creating a bucket, create a sink to route log entries to your bucket and configure log views to control who can access the logs in your new bucket and which logs are accessible to them.

Manage buckets

This section describes how to manage your log buckets using the Google Cloud CLI or the Cloud de Confiance console.

Update a bucket

To update the properties of your bucket, such as the description, do the following:

Cloud de Confiance console

To update your bucket's properties, do the following:

  1. In the Cloud de Confiance console, go to the Logs Storage page:

    Go to Logs Storage

    If you use the search bar to find this page, then select the result whose subheading is Logging.

  2. For the bucket you want to update, click More.

  3. Select Edit bucket.

  4. Edit your bucket as needed.

  5. Click Update bucket.

gcloud

To update your bucket's properties, run the gcloud logging buckets update command:

gcloud logging buckets update BUCKET_ID --location=LOCATION UPDATED_ATTRIBUTES

For example:

gcloud logging buckets update my-bucket --location=global --description "Updated description"

REST

To update your bucket's properties, use projects.locations.buckets.patch in the Logging API.

Lock a bucket

When you lock a bucket against updates, you also lock the bucket's retention policy. After a retention policy is locked, you can't delete the bucket until every log entry in the bucket has fulfilled the bucket's retention period. If you want to prevent the accidental deletion of a project that contains a locked log bucket, then add a lien to the project. To learn more, see Protecting projects with liens.

To prevent anyone from updating or deleting a log bucket, lock the bucket. To lock the bucket, do the following:

Cloud de Confiance console

The Cloud de Confiance console doesn't support locking a log bucket.

gcloud

To lock your bucket, run the gcloud logging buckets update command with the --locked flag:

gcloud logging buckets update BUCKET_ID --location=LOCATION --locked

For example:

gcloud logging buckets update my-bucket --location=global --locked

REST

To lock your bucket's attributes, use projects.locations.buckets.patch in the Logging API. Set the locked parameter to true.

List buckets

To list the log buckets associated with a Cloud de Confiance project, and to see details such as retention settings, do the following:

Cloud de Confiance console

In the Cloud de Confiance console, go to the Logs Storage page:

Go to Logs Storage

If you use the search bar to find this page, then select the result whose subheading is Logging.

A table named Log buckets lists the buckets associated with the current Cloud de Confiance project.

The table lists the following attributes for each log bucket:

  • Name: The name of the log bucket.
  • Description: The description of the bucket.
  • Retention period: The number of days that the bucket's data will be stored by Cloud Logging.
  • Region: The geographic location in which the bucket's data is stored.
  • Status: Whether the bucket is locked or unlocked.

If a bucket is pending deletion by Cloud Logging, then its table entry is annotated with a warning symbol.

gcloud

Run the gcloud logging buckets list command:

gcloud logging buckets list

You see the following attributes for the log buckets:

  • LOCATION: The region in which the bucket's data is stored.
  • BUCKET_ID: The name of the log bucket.
  • RETENTION_DAYS: The number of days that the bucket's data will be stored by Cloud Logging.
  • LIFECYCLE_STATE: Indicates whether the bucket is pending deletion by Cloud Logging.
  • LOCKED: Whether the bucket is locked or unlocked.
  • CREATE_TIME: A timestamp that indicates when the bucket was created.
  • UPDATE_TIME: A timestamp that indicates when the bucket was last modified.

You can also view the attributes for just one bucket. For example, to view the details for the _Default log bucket in the global region, run the gcloud logging buckets describe command:

gcloud logging buckets describe _Default --location=global

REST

To list the log buckets associated with a Cloud de Confiance project, use projects.locations.buckets.list in the Logging API.

View a bucket's details

To view the details of a single log bucket, do the following:

Cloud de Confiance console

In the Cloud de Confiance console, go to the Logs Storage page:

Go to Logs Storage

If you use the search bar to find this page, then select the result whose subheading is Logging.

On the log bucket, click More and then select View bucket details.

The dialog lists the following attributes for the log bucket:

  • Name: The name of the log bucket.
  • Description: The description of the log bucket.
  • Retention period: The number of days that the bucket's data will be stored by Cloud Logging.
  • Region: The geographic location in which the bucket's data is stored.

gcloud

Run the gcloud logging buckets describe command.

For example, the following command reports the details of the _Default bucket:

gcloud logging buckets describe _Default --location=global

You see the following attributes for the log bucket:

  • createTime: A timestamp that indicates when the bucket was created.
  • description: The description of the log bucket.
  • lifecycleState: Indicates whether the bucket is pending deletion by Cloud Logging.
  • name: The name of the log bucket.
  • retentionDays: The number of days that the bucket's data will be stored by Cloud Logging.
  • updateTime: A timestamp that indicates when the bucket was last modified.

REST

To view the details of a single log bucket, use projects.locations.buckets.get in the Logging API.

Delete a bucket

You can delete log buckets that satisfy one of the following:

  • The log bucket is unlocked.
  • The log bucket is locked and all log entries in the log bucket have fulfilled the bucket's retention period.

You can't delete a log bucket that is locked against updates when that log bucket stores log entries that haven't fulfilled the bucket's retention period.

After you issue the delete command, the log bucket transitions to the DELETE_REQUESTED state, and it stays in that state for 7 days. During this time period, Logging continues to route logs to the log bucket. You can stop routing logs to the log bucket by deleting or modifying the log sinks that route log entries to the bucket.

You can't create a new log bucket that uses the same name as a log bucket that is in the DELETE_REQUESTED state.

To delete a log bucket, do the following:

Cloud de Confiance console

To delete a log bucket, do the following:

  1. In the Cloud de Confiance console, go to the Logs Storage page:

    Go to Logs Storage

    If you use the search bar to find this page, then select the result whose subheading is Logging.

  2. Locate the bucket that you want to delete, and click More.

  3. Select Delete bucket.

  4. On the confirmation panel, click Delete.

  5. On the Logs Storage page, your bucket has an indicator that it's pending deletion. The bucket, including all the logs in it, is deleted after 7 days.

gcloud

To delete a log bucket, run the gcloud logging buckets delete command:

gcloud logging buckets delete BUCKET_ID --location=LOCATION

You can't delete a log bucket when that bucket has a linked BigQuery dataset:

REST

To delete a bucket, use projects.locations.buckets.delete in the Logging API.

Restore a deleted bucket

You can restore, or undelete, a log bucket that's in the pending deletion state. To restore a log bucket, do the following:

Cloud de Confiance console

To restore a log bucket that is pending deletion, do the following:

  1. In the Cloud de Confiance console, go to the Logs Storage page:

    Go to Logs Storage

    If you use the search bar to find this page, then select the result whose subheading is Logging.

  2. For the bucket you want to restore, click More, and then select Restore deleted bucket.

  3. On the confirmation panel, click Restore.

  4. On the Logs Storage page, the pending-deletion indicator is removed from your log bucket.

gcloud

To restore a log bucket that is pending deletion, run the gcloud logging buckets undelete command:

gcloud logging buckets undelete BUCKET_ID --location=LOCATION

REST

To restore a bucket that is pending deletion, use projects.locations.buckets.undelete in the Logging API.

Write to a bucket

You don't directly write logs to a log bucket. Rather, you write logs to Cloud de Confiance resource: a Cloud de Confiance project, folder, or organization. The sinks in the parent resource then route the logs to destinations, including log buckets. A sink routes logs to a log bucket destination when the logs match the sink's filter and the sink has permission to route the logs to the log bucket.

Read from a bucket

Each log bucket has a set of log views. To read logs from a log bucket, you need access to a log view on the log bucket. Log views let you grant a user access to only a subset of the logs stored in a log bucket. For information about how to configure log views, and how to grant access to specific log views, see Configure log views on a log bucket.

To read logs from a log bucket, do the following:

Cloud de Confiance console

  1. In the Cloud de Confiance console, go to the Logs Explorer page:

    Go to Logs Explorer

    If you use the search bar to find this page, then select the result whose subheading is Logging.

  2. To customize which logs are displayed in the Logs Explorer, click Refine scope, and then select an option. For example, you can view logs stored in a project or by log view.

  3. Click Apply. The Query results pane reloads with logs that match the option you selected.

For more information, see Logs Explorer overview: Refine scope.

gcloud

To read logs from a log bucket, use the gcloud logging read command and add a LOG_FILTER to select data:

gcloud logging read LOG_FILTER --bucket=BUCKET_ID --location=LOCATION --view=LOG_VIEW_ID

REST

To read logs from a log bucket, use the entries.list method. Set resourceNames to specify the appropriate bucket and log view, and set filter to select data.

For detailed information about the filtering syntax, see Logging query language.

Troubleshoot common issues

If you encounter problems when using log buckets, refer to the following troubleshooting steps and answers to common questions.

Why can't I delete this bucket?

If you're trying to delete a bucket, do the following:

  • Verify that you have the correct permissions to delete the bucket. For the list of the permissions that you need, see Access control with IAM.

  • Determine whether the bucket is locked by listing the bucket's attributes. If the bucket is locked, check the bucket's retention period. You can't delete a locked bucket until all of the logs in the bucket have fulfilled the bucket's retention period.

Which service accounts are routing logs to my bucket?

To determine if any service accounts have IAM permissions to route logs to your bucket, do the following:

  1. In the Cloud de Confiance console, go to the IAM page:

    Go to IAM

    If you use the search bar to find this page, then select the result whose subheading is IAM & Admin.

  2. From the Permissions tab, view by Roles. You see a table with all the IAM roles and principals associated with your Cloud de Confiance project.

  3. In the table's Filter text box, enter Logs Bucket Writer.

    You see any principals with the Logs Bucket Writer role. If a principal is a service account, its ID contains the string s3ns-system.iam.gserviceaccount.com.

  4. Optional: If you want to remove a service account from being able to route logs to your Cloud de Confiance project, select the check box for the service account and click Remove.

Why do I see logs for a Cloud de Confiance project even though I excluded them from my _Default sink?

You might be viewing logs in a log bucket in a centralized Cloud de Confiance project, which aggregates logs from across your organization.

If you're using the Logs Explorer to access these logs and see logs that you excluded from the _Default sink, then your view might be set to the Cloud de Confiance project level.

To fix this issue, select Log view in the Refine scope menu and then select the log view associated with the _Default bucket in your Cloud de Confiance project. You shouldn't see the excluded logs anymore.

What's next

For information on the log bucket API methods, refer to the LogBucket reference documentation.

If you manage an organization or a folder, then you can specify the location of the _Default and _Required log buckets of child resources. You can also configure whether log buckets use CMEK and the behavior of the _Default log sink. For more information, see Configure default settings for organizations and folders.

For information on addressing common use cases with log buckets, see the following topics: