This document describes aggregated sinks, which let you collate and route log entries that originate in resources within a folder or organization to a supported destination. We recommend that you use aggregated sinks to route your log data to a central storage location.
About aggregated sinks
An aggregated sink is similar to a project-level sink in that an aggregated sink contains filters and a destination. However, the Log Router sends to an aggregated sink the following log entries:
- All log entries that originate in a folder or organization.
- All log entries that originate in the child resources of the folder or organization.
For example, if you create a folder-level aggregated sink, then the Log Router sends to that sink all log entries that originate in the folder or in child resources of the folder.
When aggregated sinks exist in the resource hierarchy of a log entry, the Log Router initially sends the log entry to those sinks. Because aggregated sinks can be intercepting or non-intercepting, the Log Router might not send a log entry that is routed by an aggregated sink sent to the project-level sinks.
- Intercepting aggregated sink
An intercepting aggregated sink prevents log entries from being routed to sinks in child resources, except for the
_Requiredsinks in the resources where the log entries originate. An intercepting aggregated sink can be useful in preventing duplicate copies of log entries from being stored in multiple places.For example, suppose that you need to enable Data Access audit logs for auditing purposes. To simplify your analysis, you want to store these logs in a central location. However, for security and cost reasons, you also want to prevent these logs from being stored at the project level. For this scenario, you can create an intercepting aggregated sink.
- Non-intercepting aggregated sink
A non-intercepting aggregated sink doesn't affect how log entries are routed to other sinks. That is, even when a log entry matches the filter of a non-intercepting aggregated sink, that log entry is then forwarded to other sinks in the log entry's resource hierarchy. A non-intercepting aggregated sink lets you maintain visibility of log entries in the resources in which they were generated.
For example, you might create a non-intercepting aggregated sink that routes all log entries generated from the folders contained by an organization to a central log bucket. The log entries are stored in the central log bucket. However, because the sink is non-intercepting, the Log Router also sends log entries to the log sinks in the resource in which they were generated.
Routing examples
This section illustrates how a log entry that originates in a project might flow through the sinks in its resource hierarchy.
Example: No aggregated sinks exist
When no aggregated sinks exist in the resource hierarchy of the log entry, the log entry is sent to the log sinks in the project where the log entry originates. A project-level sink routes the log entry to the sink's destination when the log entry matches the sink's inclusion filter but doesn't match any of the sink's exclusion filters.
Example: A non-intercepting aggregated sink exists
Assume that a non-intercepting aggregated sink exists in the resource hierarchy for a log entry. After the Log Router sends the log entry to the non-intercepting aggregated sink, the following occurs:
The non-intercepting aggregated sink routes the log entry to the sink's destination when the log entry matches the inclusion filter but doesn't match any exclusion filter.
The Log Router sends the log entry to the log sinks in the project where the log entry originated.
A project-level sink routes the log entry to the sink's destination when the log entry matches the sink's inclusion filter but doesn't match any of the sink's exclusion filters.
Example: An intercepting aggregated sink exists
Assume that an intercepting aggregated sink exists in the resource hierarchy for a log entry. After the Log Router sends the log entry to the intercepting aggregated sink, one of the following occurs:
The log entry matches the inclusion filter but doesn't match any exclusion filter:
- The log entry is routed to the destination of the intercepting aggregated sink.
- The log entry is sent to the
_Requiredsink in the project where the log entry originated.
The log entry doesn't match the inclusion filter or it matches at least one exclusion filter:
- The log entry isn't routed by the intercepting aggregated sink.
The Log Router sends the log entry to the log sinks in the project where the log entry originated.
A project-level sink routes the log entry to the sink's destination when the log entry matches the sink's inclusion filter but doesn't match any of the sink's exclusion filters.
Supported destinations for aggregated sinks
This section lists which destinations are supported for aggregated sinks.
Intercepting sinks
The destination of an intercepting aggregated sink must be a Trusted Cloud project.
The log sinks in the destination project reroute the log entries to their destinations. All destinations except projects are supported. For example, the log sinks in the destination project might reroute log entries to a log bucket.
Non-intercepting sinks
The destination of an non-intercepting aggregated sink can be any of the following:
The destination of a sink can be in a different resource than the sink. For example, you can use a log sink to route log entries from one project to a log bucket stored in a different project.
The following destinations are supported:
- Trusted Cloud project
Select this destination when you want the log sinks in the destination project to reroute your log entries, or when you have created an intercepting aggregated sink. The log sinks in the project that is the sink destination can reroute the log entries to any supported destination except a project.
- Log bucket
- Select this destination when you want to store your log data in resources managed by Cloud Logging. Log data stored in log buckets can be viewed and analyzed using services like the Logs Explorer.
- Pub/Sub topic
- Select this destination when you want to export your log data from Trusted Cloud by S3NS and then use a third-party integration. Log entries are formatted into JSON and then routed to a Pub/Sub topic.
Best practices
We recommend that the destination of an aggregated sink be a Trusted Cloud project.
With this destination, the log sinks in the destination Trusted Cloud project
reroute the log entries.
The _Required sink only routes log entries that match its filter and that
originate in the resource where the sink is defined. Therefore, if you want
to store additional copies of log entries that match the filter of the
_Required sink, then you must create a custom log sink or modify the filter
of the _Default log sink.
When you create an intercepting sink, we recommend that you do the following:
Consider whether child resources need independent control of routing their log entries. If a child resource needs independent control of certain log entries, the verify that your intercepting sink doesn't route those log entries.
Add contact information to the description of an intercepting sink. This might be helpful if those who manage the intercepting sink are different from those who manage the projects whose log entries are being intercepted.
Test your sink configuration by first creating a non-intercepting aggregated sink to verify that the correct log entries are being routed.
What's next
To learn how to create an aggregated sink, see Collate and route organization- and folder-level logs to supported destinations
For a tutorial, see Aggregate and store your organization's logs.
For information about managing existing sinks, see Route logs to supported destinations: Manage sinks.
To learn how to view your logs in their destinations, as well as how the logs are formatted and organized, see View logs in sink destinations