Aggregated sinks overview

This document describes aggregated sinks, which let you collate and route log entries that originate in resources within a folder or organization to a supported destination. We recommend that you use aggregated sinks to route your log data to a central storage location.

About aggregated sinks

An aggregated sink is similar to a project-level sink in that an aggregated sink contains filters and a destination. However, the Log Router sends to an aggregated sink the following log entries:

  • All log entries that originate in a folder or organization.
  • All log entries that originate in the child resources of the folder or organization.

For example, if you create a folder-level aggregated sink, then the Log Router sends to that sink all log entries that originate in the folder or in child resources of the folder.

When aggregated sinks exist in the resource hierarchy of a log entry, the Log Router initially sends the log entry to those sinks. Because aggregated sinks can be intercepting or non-intercepting, the Log Router might not send a log entry that is routed by an aggregated sink sent to the project-level sinks.

Intercepting aggregated sink

An intercepting aggregated sink prevents log entries from being routed to sinks in child resources, except for the _Required sinks in the resources where the log entries originate. An intercepting aggregated sink can be useful in preventing duplicate copies of log entries from being stored in multiple places.

For example, suppose that you need to enable Data Access audit logs for auditing purposes. To simplify your analysis, you want to store these logs in a central location. However, for security and cost reasons, you also want to prevent these logs from being stored at the project level. For this scenario, you can create an intercepting aggregated sink.

Non-intercepting aggregated sink

A non-intercepting aggregated sink doesn't affect how log entries are routed to other sinks. That is, even when a log entry matches the filter of a non-intercepting aggregated sink, that log entry is then forwarded to other sinks in the log entry's resource hierarchy. A non-intercepting aggregated sink lets you maintain visibility of log entries in the resources in which they were generated.

For example, you might create a non-intercepting aggregated sink that routes all log entries generated from the folders contained by an organization to a central log bucket. The log entries are stored in the central log bucket. However, because the sink is non-intercepting, the Log Router also sends log entries to the log sinks in the resource in which they were generated.

Routing examples

This section illustrates how a log entry that originates in a project might flow through the sinks in its resource hierarchy.

Example: No aggregated sinks exist

When no aggregated sinks exist in the resource hierarchy of the log entry, the log entry is sent to the log sinks in the project where the log entry originates. A project-level sink routes the log entry to the sink's destination when the log entry matches the sink's inclusion filter but doesn't match any of the sink's exclusion filters.

Example: A non-intercepting aggregated sink exists

Assume that a non-intercepting aggregated sink exists in the resource hierarchy for a log entry. After the Log Router sends the log entry to the non-intercepting aggregated sink, the following occurs:

  1. The non-intercepting aggregated sink routes the log entry to the sink's destination when the log entry matches the inclusion filter but doesn't match any exclusion filter.

  2. The Log Router sends the log entry to the log sinks in the project where the log entry originated.

    A project-level sink routes the log entry to the sink's destination when the log entry matches the sink's inclusion filter but doesn't match any of the sink's exclusion filters.

Example: An intercepting aggregated sink exists

Assume that an intercepting aggregated sink exists in the resource hierarchy for a log entry. After the Log Router sends the log entry to the intercepting aggregated sink, one of the following occurs:

  • The log entry matches the inclusion filter but doesn't match any exclusion filter:

    1. The log entry is routed to the destination of the intercepting aggregated sink.
    2. The log entry is sent to the _Required sink in the project where the log entry originated.

  • The log entry doesn't match the inclusion filter or it matches at least one exclusion filter:

    1. The log entry isn't routed by the intercepting aggregated sink.

    2. The Log Router sends the log entry to the log sinks in the project where the log entry originated.

      A project-level sink routes the log entry to the sink's destination when the log entry matches the sink's inclusion filter but doesn't match any of the sink's exclusion filters.

Supported destinations for aggregated sinks

This section lists which destinations are supported for aggregated sinks.

Intercepting sinks

The destination of an intercepting aggregated sink must be a Google Cloud project.

The log sinks in the destination project reroute the log entries to their destinations. All destinations except projects are supported. For example, the log sinks in the destination project might reroute log entries to a log bucket.

Non-intercepting sinks

The destination of an non-intercepting aggregated sink can be any of the following:

The destination of a sink can be in a different resource than the sink. For example, you can use a log sink to route log entries from one project to a log bucket stored in a different project.

The following destinations are supported:

Google Cloud project

Select this destination when you want the log sinks in the destination project to reroute your log entries, or when you have created an intercepting aggregated sink. The log sinks in the project that is the sink destination can reroute the log entries to any supported destination except a project.

Log bucket

Select this destination when you want to store your log data in resources managed by Cloud Logging. Log data stored in log buckets can be viewed and analyzed using services like the Logs Explorer and Log Analytics.

If you want to join your log data with other business data, then you can store your log data in a log bucket and create a linked BigQuery dataset. A linked dataset is a read-only dataset that can be queried like any other BigQuery dataset.

BigQuery dataset
Select this destination when you want to join your log data with other business data. The dataset you specify must be write-enabled. Don't set the destination of a sink to be a linked BigQuery dataset. Linked datasets are read-only.
Cloud Storage bucket
Select this destination when you want long-term storage of your log data. The Cloud Storage bucket can be in the same project in which log entries originate, or in a different project. Log entries are stored as JSON files.
Pub/Sub topic
Select this destination when you want to export your log data from Google Cloud and then use third-party integrations like Splunk or Datadog. Log entries are formatted into JSON and then routed to a Pub/Sub topic.

Best practices

We recommend that the destination of an aggregated sink be a Google Cloud project. With this destination, the log sinks in the destination Google Cloud project reroute the log entries. The _Required sink only routes log entries that match its filter and that originate in the resource where the sink is defined. Therefore, if you want to store additional copies of log entries that match the filter of the _Required sink, then you must create a custom log sink or modify the filter of the _Default log sink.

When you create an intercepting sink, we recommend that you do the following:

  • Consider whether child resources need independent control of routing their log entries. If a child resource needs independent control of certain log entries, the verify that your intercepting sink doesn't route those log entries.

  • Add contact information to the description of an intercepting sink. This might be helpful if those who manage the intercepting sink are different from those who manage the projects whose log entries are being intercepted.

  • Test your sink configuration by first creating a non-intercepting aggregated sink to verify that the correct log entries are being routed.

Intercepting aggregated sinks and log-based metrics

Log-based metrics are Cloud Monitoring metrics that are derived from the content of log entries. The way a log entry is routed determines which log-based metrics that log entry can count toward. Because an intercepting aggregated sink affects how log entries are routed, creating this type of sink can result in changes to the values of existing log-based metrics.

For more information, see How routing log entries affects log-based metrics.

Aggregated sinks and VPC Service Controls

The following limitations apply when you use aggregated sinks and VPC Service Controls:

  • Aggregated sinks can access data from projects inside a service perimeter. To restrict aggregated sinks from accessing data inside a perimeter, we recommend using IAM to manage Logging permissions.

  • VPC Service Controls doesn't support adding folder or organization resources to service perimeters. Therefore, you can't use VPC Service Controls to protect folder- and organization-level logs, including aggregate logs. To manage Logging permissions at the folder or organizational level, we recommend using IAM.

  • If you route logs by using a folder- or organization-level sink to a resource that a service perimeter protects, then you must add an ingress rule to the service perimeter. The ingress rule must allow access to the resource from the service account that the aggregated sink uses. For more information, refer to the following pages:

  • When you specify an ingress or egress policy for a service perimeter, you can't use ANY_SERVICE_ACCOUNT and ANY_USER_ACCOUNT as an identity type when you use a log sink to route logs to Cloud Storage resources. However, you can use ANY_IDENTITY as the identity type.

What's next