BigQuery release notes

Colab Enterprise notebooks in BigQuery let you do the following in Preview:

• Explain code with Gemini assistance
• Fix and explain errors with Gemini assistance

The Merchant Center best sellers report supports multi-client accounts (MCAs). If you have an MCA, you can use the aggregator_id to query the tables. The BestSellersEntityProductMapping table maps the best-selling entities to the products in the sub-accounts' inventory. This provides a consolidated view of best-selling products, which you can then join with product data for more detailed insights. This feature is generally available (GA).

BigQuery offers optional job creation mode to speed up small queries that you use in your dashboards, data exploration, and other workflows. This mode automatically optimizes eligible queries and uses a cache to improve latency. This feature is generally available (GA).

You can select multiple columns and perform data preparation tasks on them, including dropping columns. For more information, see Prepare data with Gemini. This feature is generally available (GA).

BigQuery resource utilization charts have the following changes:

• The default timeline shown in the event timeline chart has changed from one to six hours.
• Several improvements have been made to the views, including a new reservation slot usage view. This view helps monitor idle, baseline, and autoscaled slot usage.

This feature is in Preview.

Dataplex automatic discovery in BigQuery scans your data in Cloud Storage buckets to extract and catalog metadata, creating BigLake, external, or object tables for analytics and AI for insights, security, and governance. This feature is generally available (GA).

You can now enable fine-grained access control on BigQuery metastore Iceberg tables. This feature is generally available (GA).

You can now create, view, modify, and delete Apache Iceberg resources in BigQuery metastore. This feature is generally available (GA).

You can now connect BigQuery metastore to Apache Flink. This feature is generally available (GA).

You can now define a _CHANGE_SEQUENCE_NUMBER for BigQuery change data capture (CDC) to manage streaming UPSERT ordering for BigQuery. This feature is generally available (GA).

You can include data preparation tasks in BigQuery pipelines that execute your code assets in sequence at a scheduled time. This feature is in Preview.

You can now enable metadata caching for SQL translation, which can significantly reduce latency for subsequent translation requests. This feature is in preview.

Dataform now supports the CMEK organization policy.

You can configure reusable, default Cloud resource connections in a project. Default connections are available in Preview.

BigQuery Data Transfer Service now supports custom reports for Google Ads. You can use Google Ads Query Language (GAQL) queries in your transfer configuration to ingest custom Google Ads reports and fields beyond those available in the standard reports and fields. This feature is now generally available (GA).

Gemini in BigQuery can help you complete Python code with contextually appropriate recommendations that are based on content in the query editor. This feature is now generally available (GA).

You can use the best sellers and price competitiveness migration guides to transition to the newer version of the reports. This feature is in preview.

BigQuery resource utilization charts provide metrics views and more chart configuration options in Preview.

You can now use the @@location system variable to set the location in which to run a query. This feature is in preview.

BigQuery data preparation provides context-aware join operation recommendations from Gemini. Data preparation is available in Preview.

You can now set conditional IAM access on BigQuery datasets with access control lists (ACLs). This feature is generally available (GA).

BigQuery metastore lets you access and manage metadata from a variety of processing engines, including BigQuery and Apache Spark. BigQuery metastore supports BigQuery tables and open formats such as Apache Iceberg. This feature is in preview.

The BigQuery Data Transfer Service can now transfer data from the following data sources:

• MySQL
• PostgreSQL

Transfers from these data sources are supported in Preview.

In the navigation menu, you can now go to the Settings page to set default settings that are applied when you start a session in BigQuery Studio. This feature is in preview.

The BigQuery migration assessment for Oracle now includes a total cost of ownership (TCO) calculator that provides an estimation of compute and storage costs for migrating your Oracle data warehouse to BigQuery. This feature is in preview.

We have rearranged the navigation menu into new categories. This feature is generally available (GA).

In BigQuery ML, you can now forecast multiple time series at once by using the new TIME_SERIES_ID_COL option that is available in ARIMA_PLUS_XREG multivariate time series models. Try this feature with the Forecast multiple time series with a multivariate model tutorial.

This feature is in preview.

You can now use BigQuery Omni Virtual Private Cloud (VPC) allowlists to restrict access to AWS S3 buckets and Azure Blob Storage from specific BigQuery Omni VPCs. This feature is generally available (GA).

The Sovereign Controls for EU control package now supports BigQuery Data Transfer Service. For more information, see Supported products by control package. This feature is generally available (GA).

You can now manage data canvases, data preparations, notebooks, saved queries, and workflows in Dataplex. Metadata of data canvases, data preparations, notebooks, saved queries, and workflows is automatically available in Dataplex, without additional configuration. This feature is generally available (GA).

You can now search for and view the metadata of data canvases, data preparations, notebooks, saved queries, and workflows in the Dataplex console. This feature is in preview.

You can now use the Google Cloud Code extension for VS Code to work with BigQuery datasets and notebooks in your VS Code environment. This feature is in preview.

Regional endpoints, which help you run your workloads in compliance with data residency and data sovereignty requirements, are now generally available (GA). With regional endpoints, your request traffic is routed directly to the region specified in the endpoint. For more information, see BigQuery regional endpoints.

You can now discover, procure, and commercialize your Analytics Hub listings on Google Cloud Marketplace to share data offerings at scale. This feature is in preview.

You can now replicate a dataset from the source region to one or more other regions with cross-region dataset replication. This feature is now generally available (GA).

BigQuery Managed Disaster Recovery provides managed failover and redundant compute capacity for business-critical workloads. It is intended for use in the case of a total region outage and is supported with the BigQuery Enterprise Plus edition only. This feature is now generally available (GA).

You can now create remote models in BigQuery ML based on the gemini-2.0-flash-exp model in Vertex AI. To create remote models, you can use either SQL or BigQuery DataFrames.

You can use the ML.GENERATE_TEXT function with these remote models to perform generative natural language tasks for text stored in BigQuery tables. You can also use the ML.GENERATE_TEXT function with these remote models to perform generative AI tasks, for example audio transcription or document classification, using image, video, audio, PDF, or text content stored in BigQuery object tables.

Try this feature by using either the Generate text by using the ML.GENERATE_TEXT function how-to topic, or the BigFrames Gemini 2.0 Text Generation Simple Example notebook.

This feature is in preview.

You can create a search index on columns containing INT64 or TIMESTAMP data and BigQuery can optimize predicates that use those columns. This feature is generally available (GA).

The following BigQuery ML features are now available:

• Creating remote models based on the Vertex AI gemini-1.5-flash and gemini-1.5-pro models.
• Using the ML.GENERATE_TEXT function with these remote models to perform generative natural language tasks for text stored in BigQuery tables.
• Using the ML.GENERATE_TEXT function with these remote models to perform generative AI tasks, for example audio transcription or document classification, using image, video, audio, PDF, or text content stored in BigQuery object tables.

Try these features with the Generate text by using the ML.GENERATE_TEXT function how-to topic.

These features are now generally available (GA).

The following BigQuery ML features are now available:

• You can perform supervised tuning on a remote model based on a Vertex AI Gemini 1.5 flash or Gemini 1.5 pro model.
• You can evaluate a Vertex AI LLM using the ML.EVALUATE function. Pre-trained PaLM and Gemini models and tuned Gemini models are supported for evaluation.

Try tuning and evaluating an LLM with the Customize an LLM by using supervised fine tuning how-to topic or the Use tuning and evaluation to improve model performance tutorial.

These BigQuery ML features are generally available (GA).

BigQuery now offers the following Gemini-enhanced SQL translation features:

• In interactive translation mode, you can use Gemini-enhanced SQL translations to customize translated GoogleSQL queries. This feature is generally available (GA).

• You can generate AI suggestions for batch translations using the Gemini model. The suggestions are based on a Gemini-based configuration YAML file. This feature is in Preview.

• After running an interactive SQL translation, you can request a Gemini-generated text explanation that includes a summary of the translated SQL query. This feature is in Preview.

Dataplex automatic discovery lets you scan data in Cloud Storage buckets to extract and catalog metadata. Automatic discovery creates BigLake or external tables and object tables you can use for analytics and AI, and catalogs that data in Dataplex Catalog. This feature is available in public preview.

BigQuery provides context-aware transformation recommendations from Gemini for cleansing data for analysis. Data preparation is available in Preview.

You can now view, trigger, and pause Airflow DAGs in BigQuery. This feature is in Preview.

You can now manage notebook schedules on the Orchestration page. Notebook scheduling is in Preview.

Custom organization policies let you allow or deny specific operations on BigQuery Data Transfer Service transfer configurations to meet your organization's compliance and security requirements. This feature is generally available (GA).

You can now use fine-grained DML to optimize the execution of UPDATE, DELETE, and MERGE statements on tables. This feature is in Preview.

Use the BigQuery migration assessment for Oracle to assess the complexity of migrating data from your Oracle data warehouse to BigQuery. This feature is in preview.

BigQuery tables for Apache Iceberg bring the convenience of BigQuery storage optimization to Apache Iceberg tables that reside in your own cloud buckets. BigQuery tables for Apache Iceberg let you use BigQuery without moving data out of buckets that you control. This feature is now in preview.

You can now export and load Parquet files that include GeoParquet metadata. This feature is generally available (GA).

You can now use pipe syntax anywhere you write GoogleSQL. Pipe syntax supports a linear query structure designed to make your queries easier to read, write, and maintain. This feature is in Preview.

You can now create an external dataset in BigQuery that links to an existing database in Spanner. This feature is in preview.

ODBC driver update, release 3.0.7 1016

• [New] Connector authentication on Google Cloud VMs: The connector now supports authentication through Application Default Credentials using the Google internal metadata server, eliminating the need for a keyfile. This feature works only on Google Cloud Compute Engine VMs.
• [Resolved] The output for PrimaryKeys previously denoted the Key Sequence as a 0-indexed value. This has been corrected to a 1-indexed value, indicating the sequential order of the primary key's column within the primary key itself.

You can now enable, disable, and analyze history-based optimizations for queries. This feature is generally available (GA).

You can now use flexible column names with BigQuery tables and views for extracting, loading, streaming, and querying data. This feature is generally available (GA).

You can now use the operational health dashboard to get a single-pane view of key metrics such as slot usage, shuffle usage, errors, and total storage in real time. This feature is generally available (GA).

You can now create a materialized view replica directly from the Google Cloud console. This feature is generally available (GA).

Cloud console updates: You can now use keyboard shortcuts to control tab navigation in the details pane. This feature is generally available (GA).

You can now use Cloud KMS Autokey to automate the creation and use of customer-managed encryption keys (CMEKs), including the Cloud HSM service. This feature is generally available (GA).

BigQuery ML now offers the following expanded embedding support features:

• Using the ML.GENERATE_EMBEDDING function with a remote model based on a Vertex AI multimodalembedding large language model (LLM) to create multimodal embeddings, which embed text, image, and video into the same semantic space.
• Using the ML.GENERATE_EMBEDDING function with a principal component analysis (PCA) model or autoencoder model to create embeddings for structured independent and identically distributed random variables (IID) data.
• Using the ML.GENERATE_EMBEDDING function with a matrix factorization model to create embeddings for user or item data.

Try these capabilities with the following tutorials:

• Generate image embeddings by using the ML.GENERATE_EMBEDDING function
• Generate video embeddings by using the ML.GENERATE_EMBEDDING function
• Generate text embeddings by using the ML.GENERATE_EMBEDDING function
• Generate and search multimodal embeddings

These features are generally available (GA).

BigQuery ML now offers the following AI features:

• You can process documents from BigQuery object tables by doing the following:

  1. Creating a remote model based on the Document AI API, including specifying a document processor to use.
  2. Using the ML.PROCESS_DOCUMENT function with a Document AI-based remote model to process the documents.

  Try this feature with the Process documents with the ML.PROCESS_DOCUMENT function how-to.

• You can transcribe audio files from BigQuery object tables by doing the following:

  1. Creating a remote model based on the Speech-to-Text API, including specifying a speech recognizer to use.
  2. Using the ML.TRANSCRIBE function with a Speech-to-Text-based remote model to transcribe the audio files.

  Try this feature with the Transcribe audio files with the ML.TRANSCRIBE function how-to.

These BigQuery ML feature are generally available (GA).

You can now create workflows to execute code assets in sequence at a scheduled time. This feature is in Preview.

You can perform model monitoring in BigQuery ML. The following model monitoring functions are now generally available (GA):

• ML.DESCRIBE_DATA: compute descriptive statistics for a set of training or serving data.
• ML.VALIDATE_DATA_SKEW: compute the statistics for a set of serving data, and then compare them to the statistics for the data used to train a BigQuery ML model in order to identify anomalous differences between the two data sets.
• ML.VALIDATE_DATA_DRIFT: compute and compare the statistics for two sets of serving data in order to identify anomalous differences between the two data sets.
• ML.TFDV_DESCRIBE: compute fine-grained descriptive statistics for a set of training or serving data. This function provides the same behavior as the TensorFlow tfdv.generate_statistics_from_csv API.
• ML.TFDV_VALIDATE: compute and compare the statistics for training and serving data, or two sets of serving data, in order to identify anomalous differences between the two data sets. This function provides the same behavior as the TensorFlow tfdv.validate_statistics API.

You can now batch migrate classic saved queries to saved queries. This feature is in Preview for projects that have fewer than 2500 classic saved queries.

You can now use a CREATE MODEL statement to create a contribution analysis model in BigQuery ML. You can use a contribution analysis model with the ML.GET_INSIGHTS function to generate insights about changes to key metrics in your multi-dimensional data.

Try this feature with the Get data insights from a contribution analysis model tutorial.

This feature is in preview.

You can store columns in your vector indexes and pre-filter data in your vector searches to improve query efficiency. This feature is Generally Available.

You can now use the partial ordering mode in BigQuery DataFrames to generate more efficient queries. This feature is in Preview.

You can now use Terraform to manage IAM tags on datasets and tables. This feature is generally available (GA).

The BigQuery Data Transfer Service can now transfer campaign reporting and configuration data from Display & Video 360 into BigQuery, including Creative, Partner, and Advertiser tables. This feature is generally available (GA).

You can now use vector search and vector index features in BigQuery.

You can use the VECTOR_SEARCH function to search embeddings in order to identify semantically similar entities.

You can use vector indexes to make VECTOR_SEARCH more efficient, with the trade-off of returning more approximate results.

You can try the vector search and vector index capabilities by using the Search embeddings with vector search tutorial.

The BigQuery vector search and vector index features are generally available (GA).

The BigQuery Data Transfer Service now supports incremental transfers when you migrate your data from your Teradata data warehouses to BigQuery. This feature is generally available (GA).

Delta Lake BigLake tables are now generally available (GA). Delta Lake is an open source, tabular data storage format that supports petabyte scale data tables.

You can now use the GROUP BY clause and the SELECT DISTINCT clause with the ARRAY and STRUCT data types. This feature is in Preview.

You can now query data in AlloyDB using a federated query. This feature is now generally available (GA).

The following Gemini in BigQuery features are now generally available (GA):

• Data insights
• Data canvas
• SQL and Python code assistance features:
  • Use the SQL generation tool
  • Prompt to generate SQL queries
  • Explain a SQL query
  • Generate Python code
• Partitioning and clustering recommendations

To learn how to enable and activate Gemini in BigQuery features, see Set up Gemini in BigQuery.

Phrase support for the SEARCH function is now generally available (GA).

You can now create remote models in BigQuery ML based on the Anthropic Claude model in Vertex AI.

Use the ML.GENERATE_TEXT function with these remote models to perform generative natural language tasks for text stored in BigQuery tables. Try this feature with the Generate text by using the ML.GENERATE_TEXT function how-to topic.

This feature is in preview.

You can now use EXPORT DATA statements to directly export BigQuery data to Bigtable (reverse ETL). This feature is generally available (GA).

Python code completion is now available for all BigQuery projects. This feature is available in preview. To learn how to enable and activate Gemini in BigQuery features, see Set up Gemini in BigQuery.

You can now perform anomaly detection with BigQuery ML multivariate time series (ARIMA_PLUS_XREG) models. This feature lets you detect anomalies in historical time series data or in new data with multiple feature columns. You can try this feature by using the Perform anomaly detection with a multivariate time-series forecasting model tutorial. This feature is generally available (GA).

You can now view your BigQuery insights and recommendations using the Recommendations page in the Google Cloud console. You can also view your BigQuery insights and recommendations using the following INFORMATION_SCHEMA views:

• INSIGHTS
• RECOMMENDATIONS
• RECOMMENDATIONS_BY_ORGANIZATION

These features are now in preview.

You can now get lower latency for small queries with the new short query optimized mode. BigQuery automatically determines which queries may be accelerated while other queries continue to run like before. This feature is now in preview.

You can now use time series and range functions to support time series analysis. This feature is now generally available (GA).

The JSON_KEYS function, which extracts unique JSON keys from a JSON expression, is in Preview.

Some JSON functions that take a JSONPath let you specify a mode that allows flexibility in how the JSONPath matches the JSON data structure. This feature is in Preview.

You can now create a materialized view over Apache Iceberg table that is partition aligned with the base table. The materialized view only supports time-based partition transformation, for example, YEAR, MONTH, DAY, and HOUR. This feature is in preview.

When you translate SQL queries from your source database, you can use configuration YAML files to optimize and improve the performance of your translated SQL. This feature is in preview.

Workload management now provides the following benefits:

• The autoscaler now scales up immediately.
• The autoscaler now scales more precisely.
• The autoscaler scales to the nearest multiple of 50 slots, instead of 100.
• You can now purchase capacity commitments, set baseline slots, and set autoscale max slots in incremental steps of 50 slots.
• If one minute or more has passed since the most recent increase in capacity, you can now reduce capacity without resetting the one minute minimum. This allows for multiple consecutive decreases without a one minute delay between them.

These features are now generally available (GA).

You can now use the output_dimensionality argument of the ML.GENERATE_EMBEDDING function when you use the function with a remote model based on a Vertex AI multimodalembedding model. The output_dimensionality argument lets you specify the number of dimensions to use when generating embeddings. This feature is in Preview.

The RANGE data type is now a supported JSON encoding. This feature is Generally Available (GA).

You can now use the administrative jobs explorer to help you quickly monitor jobs activity across your organization. This feature is generally available (GA).

Vector indexes support the TreeAH index type, which uses Google's ScaNN algorithm. The TreeAH index is optimized for batch queries that process hundreds or more query vectors. This feature is in Preview.

You can now use table explorer to examine table data and create data exploration queries. This feature is in preview.

IAM deny policies now support additional permissions, including bigquery.tables.getData which can deny permission to read tables. Consider special cases when you create deny policies for bigquery.tables.getData and other BigQuery permissions. This feature is in preview.

You can now configure SAP Datasphere connections with network attachments to help secure connections. SAP Datasphere connections are in preview.

Manifest files are now supported for Amazon S3 and Azure Blob Storage. This feature is generally available (GA).

The CHANGES change history function is now in preview. This table-valued function provides a history of table changes over a window of time and captures the following operations:

• CREATE TABLE DDL statement
• INSERT DML statement
• Data appended or changed as part of a MERGE DML statement
• UPDATE DML statement
• DELETE DML statement
• Loading data into BigQuery
• Streaming ingestion
• TRUNCATE TABLE DML statement
• Jobs configured with a writeDisposition of WRITE_TRUNCATE
• Individual table partition deletions

You can use data manipulation language (DML) to modify rows that have been recently written to a BigQuery table by the Storage Write API. This is now generally available (GA).

The BigQuery continuous queries feature is now in preview.

Continuous queries let you build long-lived, continuously processing SQL statements that can analyze, process, and perform machine learning (ML) inference on incoming data in BigQuery in real time. You can configure continuous queries to replicate query results to a Pub/Sub topic, Bigtable instance, or another BigQuery table, a process also known as Reverse ETL.

You can use continuous queries to perform the following tasks, using the accessible language of SQL:

• Transform incoming data and act immediately on insights.
• Use Vertex AI to apply real time ML insights.
• Build automated event-driven data pipelines.
• Replicate real-time events to downstream operational systems like Bigtable.

To try BigQuery continuous queries, see Create continuous queries.

You can now use BigQuery Omni Virtual Private Cloud (VPC) allowlists to restrict access to AWS S3 buckets and Azure Blob Storage from specific BigQuery Omni VPCs. This feature is in preview.

The following BigQuery migration assessment features are now generally available (GA):

• When you run a migration assessment, the migration assessment now automatically creates a BigQuery dataset to store the assessment results. You can also choose to store assessment results in an existing empty dataset or manually create a dataset with a custom name.
• While a migration assessment is running, you can view the assessment report with partial data. You can also view its progress and estimated completion time in the status icon tooltip.
• You can view more information and errors about a migration assessment in the assessment details page.

You can now configure the default storage billing model for new datasets. This feature is generally available (GA).

When you run a migration assessment for Amazon Redshift, Teradata, or Snowflake, the service also creates a dataset containing only highly aggregated assessment results. This aggregated dataset doesn't contain any query logs; therefore, no personally identifiable information (PII) or business-sensitive information is visible. You can share this dataset with users that are not in your project. This feature is in preview.

You can now use EXPORT DATA statements to reverse ETL BigQuery data to Spanner. This feature is in preview.

Cloud console updates: You can now drag a tab in the details pane to open a new column and compare tabs. You can also drag the tab to a new position in the current or an adjacent column. This feature is in preview.

The following Analytics Hub features are now generally available:

• Making exchanges and listings publicly discoverable.
• Highlighting listings in the Featured section of the Analytics Hub catalog.
• Generating unauthenticated URLs for public listings.

Data publishers can now share Pub/Sub topics and manage subscriptions in Analytics Hub. This feature is in preview.