Troubleshoot routing and storing logs

This document explains common routing and storage issues and how to use the Trusted Cloud console to view and troubleshoot configuration mistakes or unexpected results.

For general information about viewing log data, see View logs in sink destinations.

Troubleshoot log routing

This section describes how to troubleshoot common issues when routing your log entries.

Destination contains unwanted log entries

You are viewing the log entries routed to a destination and determine that the destination contains unwanted log entries.

To resolve this condition, update the exclusion filters for your sinks that route log entries to the destination. Exclusion filters let you exclude selected log entries from being routed to a destination.

For example, assume that you create an aggregated sink to route log entries in an organization to a destination. To exclude the log entries from a specific project from being routed to the destination, add the following exclusion filter to the sink:

logName:projects/PROJECT_ID

You can also exclude log entries from multiple projects by using the logical-OR operator to join logName clauses.

Destination is missing log entries

Perhaps the most common sink-related issue is that log entries seem to be missing from the destination of a sink.

In some cases, an error isn't generated but you might notice that log entries are unavailable when you try to access them in your destination. If you suspect that your sink isn't properly routing log entries, then check your sink's system log-based metrics:

  • exports/byte_count: Number of bytes in log entries that were routed.
  • exports/log_entry_count: Number of log entries that were routed.
  • exports/error_count: Number of log entries that failed to be routed.

The metrics have labels that record the counts by sink name and destination name and let you know whether your sink is routing log entries successfully or or failing.

If your sink metrics indicate that your sink isn't performing as you expected, here are some possible reasons and what to do about them:

Latency

  • No matching log entries have been received since you created or updated your sink; only new log entries are routed.

    Try waiting an hour and check your destination again.

  • Matching log entries are late-arriving.

    There can be a delay before you can view your log entries in the destination. Try waiting a few hours and check your destination again.

Viewing scope/filter is incorrect

  • The scope you're using to view log entries stored in a log bucket is incorrect.

    Scope your search to one or more log views as follows:

    • If you're using the Logs Explorer, then use the Refine scope button.

    • If you're using the gcloud CLI, then use the gcloud logging read command and add a --view=AllLogs flag.

  • The time range you're using to select and view data in your sink destination is too narrow.

    Try broadening the time range that you're using when selecting data in your sink destination.

Error in sink filter

  • The sink's filter is incorrect and not capturing the log entries you expected to see in your destination.

    Edit your sink's filter by using the Log Router in the Trusted Cloud console. To verify you entered the correct filter, select Preview logs in the Edit sink panel. This opens the Logs Explorer in a new tab with the filter pre-populated. For instructions about viewing and managing your sinks, see Manage sinks.

View errors

For each of the supported sink destinations, Logging provides error messages for improperly configured sinks.

There are several ways to view these sink-related errors; these methods are described in the following sections:

  • View the error logs generated for the sink.
  • Receive sink error notifications by email. The sender of this email is logging-noreply@google.com.

Error logs

The recommended method for inspecting your sink-related errors in detail is to view the error log entries generated by the sink. For details about viewing log entries, see View logs by using the Logs Explorer.

You can use the following query in the query-editor pane in the Logs Explorer to review your sink's error logs. The same query works in the Logging API and the gcloud CLI.

Before you copy the query, replace the variable SINK_NAME with the name of the sink you're trying to troubleshoot. You can find your sink's name on the Log Router page in the Trusted Cloud console.

logName:"logging.googleapis.com%2Fsink_error"
resource.type="logging_sink"
resource.labels.name="SINK_NAME"

For example, if your sink's name is my-sink-123, then the log entry might look similar to the following:

{
   errorGroups: [
   0: {
   id: "COXu96aNws6BiQE"
   }]
   insertId: "170up6jan"
   labels: {
      activity_type_name: "LoggingSinkConfigErrorV2"
      destination: "pubsub.googleapis.com/projects/my-project/topics/my-topic"
      error_code: "topic_not_found"
      error_detail: ""
      sink_id: "my-sink-123"
   }
   logName: "projects/my-project/logs/logging.googleapis.com%2Fsink_error"
   receiveTimestamp: "2024-07-11T14:41:42.578823830Z"
   resource: {
   labels: {
      destination: "pubsub.googleapis.com/projects/my-project/topics/my-topic"
      name: "my-sink-123"
      project_id: "my-project"
   }
   type: "logging_sink"
   }
   severity: "ERROR"
   textPayload: "Cloud Logging sink configuration error in my-project, sink my-sink-123: topic_not_found ()"
   timestamp: "2024-07-11T14:41:41.296157014Z"
}

The LogEntry field labels and its nested key-value information helps you target the source of your sink's error; it contains the affected resource, affected sink, and error code. The labels.error_code field contains a shorthand description of the error, letting you know which component of your sink needs reconfiguring.

To resolve this failure, edit your sink. For example, you might edit your sink by using the Log Router page:

Go to Log Router

Email notifications

Essential Contacts sends sink configuration error email notifications to contacts assigned to the Technical notification category for a Trusted Cloud project or its parent resource. If the resource does not have a configured contact for Technical notifications, then users listed as IAM Project Owner roles/owner for the resource receive the email notification.

The email message contains the following information:

  • Resource ID: The name of the Trusted Cloud project or other Trusted Cloud resource where the sink was configured.
  • Sink name: The name of the sink that contains the configuration error.
  • Sink destination: The full path of the sink's routing destination; for example, pubsub.googleapis.com/projects/PROJECT_ID/topics/TOPIC_ID
  • Error code: Shorthand description of the error category; for example, topic_not_found.
  • Error detail: Detailed information about the error, including recommendations for troubleshooting the underlying error.

The sender of this email is logging-noreply@google.com.

To view and manage your sinks, use the Log Router page:

Go to Log Router

Any sink configuration errors that apply to the resource appear in the list as a Cloud Logging sink configuration error. Each error contains a link to one of the log entries generated by the faulty sink. To examine the underlying errors in detail, see the section Error logs.

Types of sink errors

The following sections describe broad categories of sink-related errors and how you can troubleshoot them.

Incorrect destination

If you set up a sink but then see a configuration error that the destination couldn't be found when Logging attempted to route log entries, here are some possible reasons:

  • Your sink's configuration contains a misspelling or other formatting error in the specified sink destination.

    You need to update the sink's configuration to properly specify the existing destination.

  • The specified destination might have been deleted.

    You can either change the sink's configuration to use a different, existing destination or recreate the destination with the same name.

To resolve these types of failure, edit your sink. For example, you might edit your sink by using the Log Router page:

Go to Log Router

Your sink begins routing log entries when the destination is found and new log entries that match your filter are received by Logging.

Managing sinks issues

If you disabled a sink to stop storing log entries in a log bucket but still see log entries being routed, then wait a few minutes for changes to the sink to apply.

Permissions issues

When a sink tries to route a log entry but lacks the appropriate IAM permissions for the sink's destination, the sink reports an error, which you can view, and skips the log entry.

When you create a sink, the sink's service account must be granted the appropriate destination permissions. If you create the sink in the Trusted Cloud console in the same Trusted Cloud project, then the Trusted Cloud console typically assigns these permissions automatically. However, if you create the sink in a different Trusted Cloud project, or by using gcloud CLI or the Logging API, then you must configure the permissions manually.

If you're seeing permission-related errors for your sink, then add the necessary permissions or update your sink to use a different destination. For instructions on how to update these permissions, see Destination permissions.

There is a slight delay between creating the sink and using the sink's new service account to authorize writing to the destination. Your sink begins routing log entries when any permissions are corrected and new log entries that match your filter are received by Logging.

Organizational policy issues

If you're trying to route a log entry but encounter an organization policy that constrains Logging from writing to the sink's destination, then the sink can't route to the selected destination and reports an error.

If you're seeing errors related to organization policies, then you can do the following:

  • Update the organization policy for the destination to remove the constraints blocking the sink from routing log entries; this presupposes that you have the appropriate permissions to update the organization policy.

    You might examine whether a Resource Location Restriction (constraints/gcp.resourceLocations) exists. This constraint determines the locations where data can be stored. Also, some services support constraints that might affect a log sink. For example, there are several restrictions that might apply when a Pub/Sub destination is selected. For a list of possible constraints, see Organization policy constraints.

    For instructions, see Creating and editing policies.

  • If you can't update the organization policy, then update your sink in the Log Router page to use a compliant destination.

    Go to Log Router

Your sink begins routing log entries when the organization policy no longer blocks the sink from writing to the destination and new log entries that match your filter are received by Logging.

Encryption key issues

If you're using encryption keys, whether managed with Cloud Key Management Service or by you, to encrypt the data in the sink's destination, then you might see related errors. Here are some possible issues and ways to fix them:

  • The Cloud KMS key can't be found.

    • The Trusted Cloud project that contains the Cloud KMS key configured to encrypt the data isn't found.

    • Use a valid Cloud KMS key from an existing Trusted Cloud project.

  • The location of the Cloud KMS key doesn't match the location of the destination.

    • If the Trusted Cloud project that contains the Cloud KMS key is located in a region that differs from the region of the destination, then encryption fails and the sink can't route data to that destination.

    • Use a Cloud KMS key contained by a Trusted Cloud project whose region matches the sink's destination.

  • Encryption key access is denied to the sink's service account.

    • Even if the sink was successfully created with the correct service account permissions, this error message displays if the sink destination uses an encryption key that doesn't give the service account sufficient permissions to encrypt or decrypt the data.

    • Grant the Cloud KMS CryptoKey Encrypter/Decrypter role for the service account specified in the sink's writerIdentity field for the key used in the destination. Also verify that the Cloud KMS API is enabled.

Quota issues

When sinks write log entries, destination-specific quotas apply to the Trusted Cloud projects in which the sinks were created. If the quotas are exhausted, then the sink stops routing log entries to the destination.

For example, your sink might be routing too many log entries too quickly.

To fix the quota exhaustion issues, decrease the amount of log data being routed by updating your sink's filter to match fewer log entries. You might use the sample function in your filter to select a fraction of the total number of log entries.

When quota is available, your sink routes log entries to the sink's destination.

For details on the limits that might apply when you route log entries, review the appropriate destination's quota information:

In addition to the general sink error types, here are the most common destination-specific error types and how you can fix them.

Errors routing to Cloud Logging buckets

You might encounter a situation where you can see log entries in the Logs Explorer that you excluded with your sink. You can still see these log entries if any of following conditions are true:

  • You're running your query in the Trusted Cloud project that generated the log entries.

    To fix this, verify you're running your query in the correct Trusted Cloud project.

  • The excluded log entries were sent to multiple log buckets; you're seeing a copy of the same log you meant to exclude.

    To fix this, check your sinks in the Log Router page to verify you aren't including the log entries in other sinks' filters.

  • You have access to views in the log bucket where the log entries were sent. In this case, you can see those log entries by default.

    To avoid seeing these log entries in the Logs Explorer, you can refine the scope of your search to your source Trusted Cloud project or bucket.

Troubleshoot storing logs

Why can't I delete this bucket?

If you're trying to delete a bucket, do the following:

  • Ensure that you have the correct permissions to delete the bucket. For the list of the permissions that you need, see Access control with IAM.

  • Determine whether the bucket is locked by listing the bucket's attributes. If the bucket is locked, check the bucket's retention period. You can't delete a locked bucket until all of the logs in the bucket have fulfilled the bucket's retention period.

Which service accounts are routing logs to my bucket?

To determine if any service accounts have IAM permissions to route logs to your bucket, do the following:

  1. In the Trusted Cloud console, go to the IAM page:

    Go to IAM

    If you use the search bar to find this page, then select the result whose subheading is IAM & Admin.

  2. From the Permissions tab, view by Roles. You see a table with all the IAM roles and principals associated with your Trusted Cloud project.

  3. In the table's Filter text box, enter Logs Bucket Writer.

    You see any principals with the Logs Bucket Writer role. If a principal is a service account, its ID contains the string s3ns-system.iam.gserviceaccount.com.

  4. Optional: If you want to remove a service account from being able to route logs to your Trusted Cloud project, select the check box for the service account and click Remove.

Why do I see logs for a Trusted Cloud project even though I excluded them from my _Default sink?

You might be viewing logs in a log bucket in a centralized Trusted Cloud project, which aggregates logs from across your organization.

If you're using the Logs Explorer to access these logs and see logs that you excluded from the _Default sink, then your view might be set to the Trusted Cloud project level.

To fix this issue, select Log view in the Refine scope menu and then select the log view associated with the _Default bucket in your Trusted Cloud project. You shouldn't see the excluded logs anymore.