Introduction to BigQuery monitoring

BigQuery monitoring lets you track resource usage, job execution, and operational health across your workloads to identify bottlenecks and maintain reliability.

Monitoring and logging are crucial for running reliable applications in the cloud. BigQuery workloads are no exception, especially if your workload has high volumes or is mission critical.

Monitoring sources vary by sampling frequency and aggregation level. For example, information schema views often provide higher granularity than Cloud Monitoring metrics. As a result, metric charts might temporarily diverge from information schema data until aggregation smooths out differences. When designing a monitoring strategy, assess latency, precision, and accuracy against your requirements.

Metrics

Metrics are numerical values that are collected at regular intervals and made available for analysis. You can use metrics to:

  • Create charts and dashboards.
  • Trigger alerts for conditions or situations that need human intervention.
  • Analyze historical performance.

In the case of BigQuery, the available metrics include the number of jobs that are running, how many bytes were scanned during a query, and the distribution of query times. The metrics for a query only become available after the query succeeds and can take up to seven minutes to be reported. Metrics for queries that fail are not reported. For a complete list of available metrics including their sample rates, visibility, and limitations, see bigquery under Cloud de Confiance metrics.

Use Cloud Monitoring to view BigQuery metrics and create charts and alerts. Each metric has a resource type, either bigquery_dataset, bigquery_project, or global, and a set of labels. You can group or filter each metric by using the labels.

For example, to chart the number of interactive queries in flight, use the following PromQL statement, which filters by priority equal to interactive:

{"bigquery.googleapis.com/query/count", monitored_resource="global", priority="interactive"}

The next example gets the number of load jobs in flight, grouped into 10-minute intervals:

avg_over_time({"bigquery.googleapis.com/job/num_in_flight",
  monitored_resource="bigquery_project",
  job_type="load"
}[10m])

For more information, see Creating charts and alerts for BigQuery.

Logs

Logs are text records that are generated in response to particular events or actions. BigQuery creates log entries for actions such as creating or deleting a table, purchasing slots, or running a load job. For more information about logging in Cloud de Confiance, see Cloud Logging.

A log is an append-only collection of log entries. For example, you could write your own log entries to a log named projects/PROJECT_ID/logs/my-test-log. Many Cloud de Confiance services, including BigQuery, create a type of log called audit logs. These logs record:

  • Administrative activity, such as creating or modifying resources.
  • Data access, such as reading user-provided data from a resource.
  • System events that are generated by Google systems, rather than by user actions.

Audit logs are written in a structured JSON format. The base data type for Cloud de Confiance log entries is the LogEntry structure. This structure contains the name of the log, the resource that generated the log entry, the timestamp (UTC), and other basic information.

The details of the logged event are contained in a subfield called the payload field. For audit logs, the payload field is named protoPayload. The value of this field is an AuditLog structure, indicated by the value of the protoPayload.@type field, which is set to type.googleapis.com/google.cloud.audit.AuditLog.

For operations on datasets, tables, and jobs, BigQuery writes audit logs in two different formats, although both share the AuditLog base type.

In the older format:

  • The resource.type field is bigquery_resource.
  • Details about the operation are written to the protoPayload.serviceData field. The value of this field is an AuditData structure.

In the newer format:

  • The resource.type field is either bigquery_project or bigquery_dataset. The bigquery_project resource has log entries about jobs, while the bigquery_dataset resource has log entries about storage.
  • Details about the operation are written to the protoPayload.metadata field. The value of this field is a BigQueryAuditMetadata structure.

We recommend consuming logs in the newer format. For more information, see Audit logs migration guide.

Here is an abbreviated example of a log entry that shows a failed operation:

{
  "protoPayload": {
    "@type": "type.googleapis.com/google.cloud.audit.AuditLog",
    "status": {
      "code": 5,
      "message": "Not found: Dataset my-project:my-dataset was not found in location US"
    },
    "authenticationInfo": { ... },
    "requestMetadata":  { ... },
    "serviceName": "bigquery.googleapis.com",
    "methodName": "google.cloud.bigquery.v2.JobService.InsertJob",
    "metadata": {
  },
  "resource": {
    "type": "bigquery_project",
    "labels": { .. },
  },
  "severity": "ERROR",
  "logName": "projects/my-project/logs/cloudaudit.googleapis.com%2Fdata_access",
  ...
}

For operations on BigQuery Reservations, the protoPayload is an AuditLog structure, and the protoPayload.request and protoPayload.response fields contain more information. You can find the field definitions in BigQuery Reservation API. For more information, see Monitoring BigQuery Reservations.

BigQuery information schema views

Information schema views are another source of insights in BigQuery, which you can use in conjunction with metrics and logs.

These views contain metadata about jobs, datasets, tables, and other BigQuery entities. For example, you can get real-time metadata about which BigQuery jobs ran over a specified time period, and then group or filter the results by project, user, tables referenced, and other dimensions.

You can use this information to perform more detailed analysis about your BigQuery workloads, and answer questions like:

  • What is the average slot utilization for all queries over the past 7 days for a given project?
  • Which users submitted a batch load job for a given project?
  • What streaming errors occurred in the past 30 minutes, grouped by error code?

In particular, look at jobs metadata, streaming metadata, and reservations metadata to get insights into the performance of your BigQuery workloads.

You can find example information schema queries on GitHub that show an organization's slot and reservation utilization, job execution, and job errors. For example, the following query provides a list of queries that are either pending or running. These queries are ordered by the length of time since they were created in the us region:

SELECT
    creation_time,
    project_id,
    user_email,
    job_id,
    job_type,
    priority,
    state,
    TIMESTAMP_DIFF(CURRENT_TIMESTAMP(), start_time,second) as running_time_sec
 FROM
   `region-us`.INFORMATION_SCHEMA.JOBS_BY_PROJECT
 WHERE
    creation_time BETWEEN TIMESTAMP_SUB(CURRENT_TIMESTAMP(), INTERVAL 1 DAY) AND CURRENT_TIMESTAMP()
    AND state != "DONE"
ORDER BY
    running_time_sec DESC

For more information, see Troubleshoot BigQuery performance with these dashboards.

In addition to writing your own queries, you can use BigQuery monitoring tools in the Cloud de Confiance console:

  • Monitor health, resource utilization, and jobs. View operational health dashboards and BigQuery Admin Resource Charts that track slot usage, job concurrency, and job run time across your organization and reservations.
  • Monitor jobs. Search, filter, and group individual query executions across your organization to inspect query execution graphs, identify top slot consumers, and troubleshoot problematic jobs.

Telemetry map

The following table maps BigQuery features and resources to their operational monitoring telemetry, audited API services and resources, and audit log reference documentation. You can use it to find the telemetry that you need to troubleshoot issues, optimize performance, control costs, audit usage, and maintain compliance.

Unlike many Cloud de Confiance by S3NS services that write operational job logs to Cloud Logging, BigQuery writes operational job telemetry to information schema views such as INFORMATION_SCHEMA.JOBS. These views capture query execution stages, slot usage, bytes billed, and all submitted jobs, including in-flight and failed jobs.

BigQuery uses Logging primarily to emit Cloud Audit Logs, which record administrative and data access activity. When you audit this activity, you can determine who accessed resources or data, where, and when.

Audit logs aren't emitted for every job disruption or failure. For example, failed jobs don't generate TableDataRead or TableDataChange events, and jobCompletedEvent only records successful job completions. For details about scenarios that don't emit audit logs, see Data access in the BigQuery audit logs overview.

In the following table, an asterisk (*) in an INFORMATION_SCHEMA view name indicates scope-specific view variants, such as _BY_PROJECT, _BY_USER, _BY_FOLDER, and _BY_ORGANIZATION.

Feature or resource Operational monitoring Cloud Audit Logs
Queries and jobs

Information schema views:

  • JOBS*: real-time and historical job execution, slot milliseconds, bytes billed, execution stages, and all submitted jobs regardless of completion status
  • JOBS_TIMELINE*: per-timeslice job state and slot usage

Console:

Metrics: Cloud Monitoring metrics, such as query/count, query/execution_times, and job/num_in_flight.

API service: bigquery.googleapis.com.

Audited resources: jobs in JobService (monitored resource: bigquery_project).

Audit logs: Data Access logs using BigQueryAuditMetadata (JobInsertion and JobChange events), recording who submitted or canceled a job, job status, statement type, and referenced tables or views. See BigQuery audit logs overview.
Tables and views

Information schema views:

Metrics: Monitoring metrics, such as storage/stored_bytes and storage/table_count.

API services: bigquery.googleapis.com and bigquerystorage.googleapis.com. Data Access logs for BigQuery Storage API use bigquery.googleapis.com, while Policy Denied logs use bigquerystorage.googleapis.com.

Audited resources: tables, views, and materialized views in TableService and TableDataService; and storage read sessions and write streams in BigQueryRead and BigQueryWrite (monitored resources: bigquery_dataset and bigquery_table).

Audit logs: Data Access logs using BigQueryAuditMetadata (TableDataRead and TableDataChange events for table reads, modifications, referenced fields, source job, and BigQuery Storage API stream reads and writes), plus Admin Activity, Data Access, and System Event logs for TableCreation, TableChange, and TableDeletion events, including table Identity and Access Management (IAM) policy changes and table expirations. See BigQuery audit logs overview.
Datasets

Information schema views:

API service: bigquery.googleapis.com.

Audited resources: datasets, dataset IAM policies, and dataset ACLs in DatasetService and IAMPolicy (monitored resource: bigquery_dataset).

Audit logs: Admin Activity and Data Access logs using BigQueryAuditMetadata (DatasetCreation, DatasetChange, and DatasetDeletion events), including dataset IAM policy changes in bindingDeltas and ACL updates in accessChanges. See BigQuery audit logs overview.
Slots and reservations

Information schema views:

Console:

Metrics: Monitoring metrics, such as slots/allocated and slots/total_available.

API service: bigqueryreservation.googleapis.com for the BigQuery Reservation API.

Audited resources: reservations, capacity commitments, and reservation assignments in ReservationService (monitored resource: bigquery_project).

Audit logs: Admin Activity and Data Access logs for creating, modifying, deleting, and listing reservations, capacity commitments, and project or folder assignments. See BigQuery Reservation API audit logging.
Column-level security and data masking

Information schema views:

API services: datacatalog.googleapis.com for Data Catalog policy tags, bigquerydatapolicy.googleapis.com for the BigQuery Data Policy API, and bigquery.googleapis.com.

Audited resources: taxonomies and policy tags in PolicyTagManager, data policies in DataPolicyService, and policy-tagged table columns (monitored resources: audited_resource and bigquery_dataset).

Audit logs: Admin Activity and Data Access logs for taxonomy and policy tag management and IAM policy changes (Audit policy tags); data policy management and grantee updates (BigQuery Data Policy audit logging); and TableDataRead.fields logs for reads on protected columns (BigQuery audit logs overview).
Routines, including UDFs and stored procedures

Information schema views:

API service: bigquery.googleapis.com.

Audited resources: user-defined functions (UDFs), table functions, and stored procedures in RoutineService (monitored resources: bigquery_dataset and bigquery_project).

Audit logs: Admin Activity and Data Access logs using BigQueryAuditMetadata (RoutineCreation, RoutineChange, and RoutineDeletion events), plus child job events for stored procedure runs. See BigQuery audit logs overview.
External connections

Information schema views:

  • OBJECT_PRIVILEGES: connection privileges
  • JOBS*: federated queries and remote function jobs that use connections

API service: bigqueryconnection.googleapis.com for the BigQuery Connection API.

Audited resources: connections and connection IAM policies in ConnectionService (monitored resource: audited_resource).

Audit logs: Admin Activity and Data Access logs for connection creation, credential updates, deletion, listing, and IAM policy changes. See BigQuery Connection API audit logging.
BigQuery sharing

Information schema views:

Dashboard: sharing usage metrics dashboard.

API service: analyticshub.googleapis.com.

Audited resources: data exchanges, listings, subscriptions, and exchange or listing IAM policies in AnalyticsHubService (monitored resource: audited_resource).

Audit logs: Admin Activity and Data Access logs for data exchange and listing creation, updates, deletion, IAM policy changes, and subscriber operations. See Sharing audit logging.
BigQuery Data Transfer Service

Metrics and logs:

API service: bigquerydatatransfer.googleapis.com for the BigQuery Data Transfer Service API.

Audited resources: transfer configurations, transfer runs, and data sources in DataTransferService (monitored resource: audited_resource).

Audit logs: Admin Activity and Data Access logs for creating, updating, deleting, listing, and starting transfer configurations and runs. See BigQuery Data Transfer Service audit logging.
BigLake catalogs and tables

Information schema views:

  • TABLES: table metadata
  • JOBS*: external object storage query statistics (metadata_cache_statistics and object_storage_stats)

API services: biglake.googleapis.com and bigquery.googleapis.com.

Audited resources: metastore catalogs, databases, tables, and partitions in MetastoreService and HiveMetastoreService, and Apache Iceberg catalogs, namespaces, and tables in IcebergCatalogService (monitored resources: audited_resource and bigquery_dataset).

Audit logs: Admin Activity and Data Access logs for catalog, database, namespace, table, partition, and catalog IAM policy operations. See BigLake audit logging.
BigQuery Migration Service

Status and reports:

  • Batch SQL translation job status in the Cloud de Confiance console
  • Workflow and subtask status in the BigQuery Migration API
  • Translation summary reports in Cloud Storage

API service: bigquerymigration.googleapis.com for the BigQuery Migration API.

Audited resources: migration workflows, migration subtasks, and SQL translation requests in MigrationService and SqlTranslationService (monitored resource: audited_resource).

Audit logs: Data Access logs for creating, starting, listing, viewing, and deleting migration workflows and subtasks, and translating SQL queries. See BigQuery Migration Service audit logging.

What's next