Create and manage log scopes

This document describes how you can create and manage log scopes, which you can use to help you efficiently find the log entries that you want to view or analyze. If you only want to view and analyze the log entries that originate in a project, folder, or organization, then this document isn't for you. However, if you rely on log sinks to route logs to other projects or to user-defined log buckets, or if you use log views, then the information in this document might help you efficiently find specific log entries.

This document doesn't describe how to view your logs. For information about that topic, see View logs by using the Logs Explorer.

About log scopes

Log scopes are persistent, project-level resources that list a set of resources. These resources can be projects, folders, organizations, and log views. For example, you could define a log scope that lists the projects that contain resources used for production, or one that lists the log views that include log entries for a specific resource type.

When you create a Trusted Cloud project, folder, or organization resource, Logging creates a log scope named _Default. This scope includes the project, folder, or organization that was created. When a searched resource is a Trusted Cloud project, folder, or organization, the results include the log entries that originate in the resource and then are stored in a log bucket. The log bucket can be in any project. When a project is searched, the results also include log entries that are routed to the project by a sink in another project and then stored in a log bucket.

You can create log scopes. You can also edit and delete the log scopes that you create. However, you can't edit or delete the log scope named _Default.

You use a log scope to control which resources the Logs Explorer page searches for log data. When you open that page and select a log scope, the page searches the resources listed in that scope and then refreshes the display.

For projects, the default log scope determines the set of resources that the Logs Explorer page searches when it opens. However, your Identity and Access Management (IAM) roles on the searched resources and the time-range setting determine which log entries are fetched from storage. When projects are created, the log scope named _Default is designated as the default log scope.

How log scopes differ from centralized log storage

Both centralized log storage and log scopes provide a way for you to view log data that originates in different projects.

When you centralize your log storage, you configure the sinks in an organization or folder to route log entries to a single storage location. Centralized storage provides a single place to query for log data, which simplifies your queries when you are searching for trends or investigating issues. From a security perspective, you also have one storage location, which simplifies the tasks of your security analysts.

When a query is issued to the resources listed in a log scope, the individual query results are combined. A log scope facilitates read-time aggregation of log data which might be stored in different locations. However, a log scope can also be used to provide read access to a one or more log views on a centralized log bucket.

When the Logs Explorer page opens, it issues queries to the resources listed in the default log scope. Therefore, configure the default scope so that the page shows you the data that you usually want to view. For example, you might set the default log scope to list a log view, which when queried, returns the log data for an App Hub application.

Best practices

Because log scopes provide a way for you to define and save a configuration for future use, we recommend that you create log scopes for complex search configurations.

For example, suppose that you are troubleshooting an issue and want to view the log entries for all virtual machine (VM) instances owned by your team. To accomplish this task, you might do the following:

  1. You determine that the log entries that you want to view are stored in multiple log buckets and in multiple projects. For most log buckets, a log view exists that includes the log entries that you want to analyze. Where a log view doesn't exist, you can create one.

  2. You decide to create a log scope because you expect to have a similar troubleshooting task in the future.

  3. You open the Logs Explorer page in the Trusted Cloud console and then use the Refine scope menu to select your new log scope.

  4. You review the log entries and find the information you need to resolve the issue you were investigating.

  5. After you resolve the issue, you share the failure cause with your colleagues. You also share that you expect to see similar failures in the future, so you created a log scope that will let you, or whomever is investigating the failure, quickly find relevant log entries.

App Hub applications and log scopes

Your App Hub applications might write log data to multiple projects. Your log data might be stored in the project in which it originates, or an organization administrator might have configured centralized storage. To view your application's log data, create a log scope, configure it to list the projects or log views that store your application's log data, and then configure it as the default log scope. When you complete those steps, the Logs Explorer page automatically displays the data written by your application, even when that data is stored in different projects or in a centralized log bucket.

Create the custom log scope in the project from which you will view your log data. This project is either your App Hub host project or the management project of your app-enabled folder. For example, if the folder's display name is My Folder, then the management project's display name is My Folder-mp.

Limitations

  • You can't delete or modify the log scope named _Default.
  • Only Trusted Cloud projects support a default log scope.
  • You can't add folders or organizations to a user-defined log scope.
  • Log scopes are created in the global location.

Before you begin

  • To get the permissions that you need to create and view log scopes, ask your administrator to grant you the Logs Configuration Writer (roles/logging.configWriter) IAM role on your project, folder, or organization. For more information about granting roles, see Manage access to projects, folders, and organizations.

    This predefined role contains the permissions required to create and view log scopes. To see the exact permissions that are required, expand the Required permissions section:

    Required permissions

    The following permissions are required to create and view log scopes:

    • To create and manage log scopes: logging.logScopes.{create, delete, get, list, update}

    You might also be able to get these permissions with custom roles or other predefined roles.

List log scopes

gcloud

To list the log scopes in a project, use the gcloud logging scopes list command:

 gcloud logging scopes list --project=PROJECT_ID

Before running the command, update the following fields:

  • PROJECT_ID: The identifier of the project. For App Hub configurations, select the App Hub host project or the app-enabled folder's management project.

To get the details of a log scope in a project, use the gcloud logging scopes describe command:

 gcloud logging scopes describe LOG_SCOPE_ID --project=PROJECT_ID

Before running the command, update the following fields:

  • PROJECT_ID: The identifier of the project.
  • LOG_SCOPE_ID: The ID of the log scope. For example, my-scope.

Terraform

You can use Terraform to create and modify a log scope. However, you can't use Terraform to list log scopes.

REST

The Cloud Logging API contains commands that list the log scopes in a resource, or that report the details of a specific log scope. For a complete list of commands, see the API reference documentation.

For Trusted Cloud projects, use the following commands:

In the previous commands, the parent field has the following syntax:

projects/PROJECT_ID/locations/LOCATION_ID

The fields in the previous expression have the following meaning:

  • PROJECT_ID: The identifier of the project. For App Hub configurations, select the App Hub host project or the app-enabled folder's management project.
  • LOCATION_ID must be set to global.

Create a log scope

You can create 100 log scopes per project. A log scope can include a total of 100 log views and projects; however, it can only include 5 projects. You can't add folders or organizations to a log scope.

gcloud

To create a log scope in a project, use the gcloud logging scopes create command:

 gcloud logging scopes create LOG_SCOPE_ID --project=PROJECT_ID \
   --description=DESCRIPTION \
   --resource-names=RESOURCE_NAMES

Before running the command, update the following fields:

  • PROJECT_ID: The identifier of the project. For App Hub configurations, select the App Hub host project or the app-enabled folder's management project.
  • LOG_SCOPE_ID: The ID of the log scope. For example, my-scope.

  • DESCRIPTION: Optional. The description of the log scope. Format the description as a string.

  • RESOURCE_NAMES: A comma-separated list of the fully-qualified names of projects or log views. For example, to include my-project in the log scope, specify projects/my-project.

Terraform

To learn how to apply or remove a Terraform configuration, see Basic Terraform commands. For more information, see the Terraform provider reference documentation.

To create a log scope in a project, folder, or organization by using Terraform, use the Terraform resource google_logging_log_scope.

In the command, set the following fields:

  • parent: The fully-qualified name of your project, folder, or organization. For example, you might set this field to "projects/PROJECT_ID", where PROJECT_ID is the ID of your Trusted Cloud project. For App Hub configurations, select the App Hub host project or the app-enabled folder's management project.
  • locations: Set to "global".

  • name: Set to the fully-qualified name of the log scope. For projects, the format of this field is:

    "projects/PROJECT_ID/locations/global/logScopes/LOG_SCOPE_ID"

    In the previous expression, LOG_SCOPE_ID is the name of a log scope, such as "production".

  • resource_names: A array of projects and log views, where each project and log view is specified by using their fully-qualified name.

  • description: A brief description. For example, "Scope for production resources".

REST

The Cloud Logging API also supports creating log scopes in a folder or organization. For more information, see the API reference documentation.

For Trusted Cloud projects, use the following command:

In the previous commands, the parent field has the following syntax:

projects/PROJECT_ID/locations/LOCATION_ID

The fields in the previous expression have the following meaning:

  • PROJECT_ID: The identifier of the project. For App Hub configurations, select the App Hub host project or the app-enabled folder's management project.
  • LOCATION_ID must be set to global.

Modify or delete a log scope

gcloud

To modify the description of list of resources in a log scope in a project, use the gcloud logging scopes update command:

 gcloud logging scopes update LOG_SCOPE_ID --project=PROJECT_ID \
   --description=DESCRIPTION \
   --resource-names=RESOURCE_NAMES

Before running the command, update the following fields:

  • PROJECT_ID: The identifier of the project. For App Hub configurations, select the App Hub host project or the app-enabled folder's management project.
  • LOG_SCOPE_ID: The ID of the log scope. For example, my-scope.

  • DESCRIPTION: The description of the log scope. Format the description as a string. Omit this field when you don't want to change the description of the log scope.

  • RESOURCE_NAMES: A comma-separated list of the fully-qualified names of projects or log views. Omit this field when you don't want to change the list of resources.

To delete a log scope in a project, use the gcloud logging scopes delete command:

 gcloud logging scopes delete LOG_SCOPE_ID --project=PROJECT_ID

Before running the command, update the following fields:

  • PROJECT_ID: The identifier of the project. For App Hub configurations, select the App Hub host project or the app-enabled folder's management project.
  • LOG_SCOPE_ID: The ID of the log scope. For example, my-scope.

Terraform

To learn how to apply or remove a Terraform configuration, see Basic Terraform commands. For more information, see the Terraform provider reference documentation.

To modify a log scope in a project, folder, or organization by using Terraform, use the Terraform resource google_logging_log_scope.

REST

The Cloud Logging API contains commands that can modify or delete a log scope. For a complete list of commands, see the API reference documentation.

For Trusted Cloud projects, use the following commands:

In the previous commands, the parent field has the following syntax:

projects/PROJECT_ID/locations/LOCATION_ID/logScopes/LOG_SCOPE_ID

The fields in the previous expression have the following meaning:

  • PROJECT_ID: The identifier of the project. For App Hub configurations, select the App Hub host project or the app-enabled folder's management project.
  • LOCATION_ID must be set to global.
  • LOG_SCOPE_ID: The ID of the log scope. For example, my-scope.

What's next