(Cloud Composer 3) Validation error messages now use the correct format for image versions.
New images are available in Cloud Composer 2:
New Airflow builds are available in Cloud Composer 3:
You can now specify an order in which Airflow searches for secrets by overriding the
[secrets]backends_order Airflow configuration option.
Fixed an issue in Cloud Composer REST API that allowed some environment.patch operations to succeed when multiple update masks that aren't related to each other were passed in a request. Now operations with such masks fail with an error.
New Airflow builds are available in Cloud Composer 3:
Cloud Composer versions 2.6.2 and 2.6.3 have reached their end of support period.
New images are available in Cloud Composer 2:
Cloud Composer 3 supports Customer Managed Encryption Keys (CMEK).
]]>Database retention policy is available in Cloud Composer 3. You can use this feature to automatically delete older records from the Airflow database, which helps to maintain the Airflow database's size.
]]>Cloud Composer 3 is now available in Stockholm (europe-north2). The change is gradually rolling out.
]]>Starting April 14, 2025, the Deployment Manager API won't be automatically enabled when you enable Cloud Composer API because this API isn't used by the Cloud Composer service.
]]>(Cloud Composer 3) Fixed a problem with configuring access to SMTP servers that don't support user and password authentication. The [smtp]smtp_user and [smtp]smtp_password Airflow configuration options are no longer preconfigured with placeholder values.
Improved the forbidden APIs check. When a new environment is created, Cloud Composer now checks for forbidden APIs that are required by a specific major version of Cloud Composer.
(Airflow 2.10.2 and 2.9.3) Preinstalled packages were changed:
virtualenv was downgraded from 20.29.1 to 20.28.1Cloud Composer version 2.6.1 has reached its end of support period.
New images are available in Cloud Composer 2:
New Airflow builds are available in Cloud Composer 3:
(Cloud Composer 3) The GOOGLE_CLOUD_PROJECT environment variable is changed to reserved for Cloud Composer 3. This fixes an issue where creating an environment with this variable failed with a non-specific error.
(Cloud Composer 2) The list of permissions reported by dependency_permissions_check_count metric was updated. Now the metric reports more permissions that are required for a functioning environment. Permissions that weren't necessary were replaced or removed.
(Cloud Composer 2) The dependency_permissions_check_count metric now correctly reports the number of missing permissions.
(Available without upgrading) Fixed a problem that caused Terraform to unnecessarily force the replacement of cloud_composer_network_ipv4_cidr_block, web_server_ipv4_cidr_block, and cloud_sql_ipv4_cidr_block fields in some configurations of Cloud Composer 2 and Cloud Composer 1 environments.
(New Cloud Composer environments only) Cloud Composer now enforces SSL connections to Cloud SQL instances.
This change implements a security compliance standard in Cloud Composer and enforces SSL usage. You can adopt other security standards by following recommendations listed in Vulnerability findings. Previously, it was possible to create a non-SSL connection to the Airflow database. After the change, the connection will be refused.
(Airflow 2.10.2 and 2.9.3) Preinstalled packages were changed:
logbook was removed from preinstalled packagesminimal-snowplow-tracker was removed from preinstalled packagesmashumaro was downgraded from 3.15 to 3.14New images are available in Cloud Composer 2:
New Airflow builds are available in Cloud Composer 3:
Cloud Composer versions 2.6.0 and 2.6.1 have reached their end of support period.
]]>Starting March 2025, the default version for new Cloud Composer environments changes from Cloud Composer 2 to Cloud Composer 3.
New environments that are created without a specified version will use the default Airflow build of Cloud Composer 3, composer-3-airflow-2. Currently, the default version is composer-2-airflow-2.
Cloud Composer 2 is no longer available in Mexico (northamerica-south1). It is not possible to select this region when creating a new Cloud Composer 2 environment. Existing Cloud Composer 2 environments and both new and existing Cloud Composer 3 environments are not affected by this change.
]]>All Cloud Composer environment's GKE clusters are set up with maintenance exclusions from January 21, 2025 to February 10, 2025. For more information, see Maintenance exclusions.
]]>(Cloud Composer 3) The issue with upgrading Airflow builds is now resolved. You can now upgrade Airflow builds for Cloud Composer 3 in the asia-south1, asia-northeast2, europe-west1, europe-west3, and europe-north1 regions.
]]>At the beginning of February 2025, we will delete inactive environments that are non-recoverable in Cloud Composer 1, Cloud Composer 2, and Cloud Composer 3. After this change, non-recoverable environments that have listed problems will be deleted automatically.
Cloud Composer 1 and Cloud Composer 2 environments that have both of the following problems present at the same time will be deleted:
Cloud Composer 3 environments that have the following problem will be deleted:
This change doesn't affect buckets of these environments. You can still recover your DAGs and other data from the environment's bucket and then delete the bucket manually. See Delete environments for information about data that is not deleted automatically together with the environment.
]]>Java Runtime in Airflow workers and schedulers is updated from version 17 to version 21.
(Cloud Composer 3) Fixed an issue with KubernetesPodOperator failing to run properly when do_xcom_push was set to True.
Fixed a problem where synchronization between the environment's bucket and Airflow components failed because incorrect objects in the bucket were not ignored.
(Available without upgrading) In Cloud Composer 2 and Cloud Composer 1, a more meaningful error message is produced when the Composer Service Agent account doesn't have necessary permissions to access the network during the environment creation.
Fixed an issue where log messages were incorrectly formatted for non-string content, such as dictionaries and numbers.
(Available without upgrading) In Cloud Composer 3, fixed an issue where deleting an environment with an attached VPC network failed if the project accept list of the network attachment was cleared manually.
Terminating Airflow workers now generate log messages with information about still running tasks.
(Cloud Composer 3) The zone in the Environment.config.node_config.location field is no longer populated in the Cloud Composer API.
(Available without upgrading) Starting from Cloud Composer version 2.11.0 with Airflow 2.10.2 and 2.9.3, the enable_task_context_logger Airflow configuration option is no longer blocked.
(Airflow 2.10.2 and 2.9.3) The following preinstalled provider packages were upgraded to new major versions:
apache-airflow-providers-apache-beam upgraded from 5.9.1 to 6.0.0apache-airflow-providers-dbt-cloud upgraded from 3.11.2 to 4.0.0apache-airflow-providers-hashicorp upgraded from 3.8.0 to 4.0.0apache-airflow-providers-http upgraded from 4.13.3 to 5.0.0apache-airflow-providers-mysql upgraded from 5.7.4 to 6.0.0apache-airflow-providers-sendgrid upgraded from 3.6.0 to 4.0.0apache-airflow-providers-sqlite upgraded from 3.9.1 to 4.0.0apache-airflow-providers-ssh upgraded from 3.14.0 to 4.0.0(Airflow 2.10.2 and 2.9.3) The apache-airflow-providers-cncf-kubernetes package was upgraded to version 10.1.0 in Cloud Composer 2 images and Cloud Composer 3 builds. For more information about changes, see the apache-airflow-providers-cncf-kubernetes changelog from version 10.0.1 to version 10.1.0.
(Airflow 2.10.2 and 2.9.3) The following preinstalled packages were downgraded:
certifi downgraded from 2024.12.14 to 2024.8.30.charset-normalizer downgraded from 3.4.1 to 3.4.0.cryptography downgraded from 44.0.0 to 43.0.3.filelock downgraded from 3.16.1 to 3.15.4.google-auth downgraded from 2.37.0 to 2.35.0.keyring downgraded from 25.6.0 to 25.5.0.urllib3 downgraded from 2.3.0 to 2.2.3.zipp downgraded from 3.21.0 to 3.20.2.New Airflow builds are available in Cloud Composer 3:
New images are available in Cloud Composer 2:
Cloud Composer version 2.5.5 has reached its end of support period.
]]>January 31, 2025 update: The issue is resolved.
(Cloud Composer 3 only) We are currently experiencing an issue with upgrading Airflow builds for Cloud Composer 3 in asia-south1, asia-northeast2, europe-west1, europe-west3, europe-north1 regions.
The upgrades are temporarily disabled as we continue our work to restore the listed functionalities. We will release an additional announcement after the issue is resolved.
]]>The following recently released Cloud Composer 3 Airflow builds and Cloud Composer 2 versions are rolled back and aren't available for creating and upgrading existing environments. We will roll out new builds in the next release.
In April 2025, Cloud Composer 2 environments will always use the environment's service account for performing PyPI packages installations:
Starting April 13, 2025, we are removing the default environment's service account setting. This change enhances security and provides greater control over your Cloud Composer environments.
To address this change:
April 3, 2025 update: This change is postponed.
Starting after April 13, 2025, Cloud Composer 3 will unify its billing with BigQuery. The Cloud Composer 3 standard milli DCU-hours SKU will be replaced with the new BigQuery Engine for Apache Airflow SKU, which will be calculated based on the number of BigQuery slot hours that your Cloud Composer 3 environments consume. Other SKUs will be renamed and moved under the BigQuery hierarchy. The change will be rolled out gradually to all regions supported by Cloud Composer 3. For more information about the change, see Transition to BigQuery slot-hour-based pricing.
]]>(Cloud Composer 3) You can now use custom certificates when installing packages from your private repository. This change is gradually rolled out to all Cloud Composer 3 environments. To obtain this change earlier, upgrade the Airflow build of your environment.
(Cloud Composer 3) New metrics are available for Cloud Composer 3 environments:
(Available without upgrading) Improved the error message generated when the Cloud Composer Service Agent service account is missing permissions on the project or on the environment's service account.
Fixed an issue where the user-defined Cloud DNS configuration for Google API domains would break Cloud Composer 3 environment creation and attachment of VPC networks.
(Cloud Composer 3 only) The /data folder is now synchronized with Airflow triggerers.
(Cloud Composer 2) Cloud Composer 2 environments that use PSC interfaces will no longer try to allocate IP ranges for VPC peerings in the tenant project. Environments that use PSC instead of VPC peerings do not use these ranges. This fixes a problem where these ranges overlapped with ranges used for the PSC subnetwork.
(New Cloud Composer 2 environments only) Cloud Composer 2 environments in versions 2.10.2 and later always use the environment's service account for performing PyPI packages installations. This change applies only to newly created environments, existing environments that are upgraded to 2.10.2 and later versions will not get this change.
The issue with automatic environment upgrades and upgrading Airflow builds in Cloud Composer 3 is resolved and these operations are working. If you think that your environment is still impacted by this issue, please reach out to the Cloud Support team.
The Redis persistent disk is now automatically deleted together with the environment. This persistent disk is used by the Redis queue and stores only technical data.
The importlib-resources package was removed from preinstalled packages.
Removed the warning log message about in-memory storage because it doesn't apply to Cloud Composer.
The worker_autoscale Airflow configuration option is blocked in Cloud Composer 2. Previously, it was blocked only in Cloud Composer 3.
Fixed a problem with the interpolation of pip.conf file. Now the pip.conf file is not interpolated and can contain unescaped % characters.
Cloud Composer version 2.5.4 has reached its end of support period.
New images are available in Cloud Composer 2:
New Airflow builds are available in Cloud Composer 3:
Cloud Composer 2.10.1 is a version with an extended upgrade timeline.
]]>January 10, 2025 update: The issue is resolved.
(Cloud Composer 3 only) We are currently experiencing an issue with automatic environment upgrades and upgrading Airflow builds for Cloud Composer 3 in asia-south1, europe-west1, and asia-northeast2 regions.
The upgrades are temporarily disabled as we continue our work to restore the listed functionalities. We will release an additional announcement after the issue is resolved.
In January 2025, we will delete inactive Cloud Composer 1 environments that are non-recoverable. Environments that have both of the following problems present at the same time will be deleted:
This change doesn't affect buckets of these environments. You can still recover your DAGs and other data from the environment's bucket and then delete the bucket manually. See Delete environments for information about data that is not deleted automatically together with the environment.
]]>Cloud Composer 3 is now available in Mexico (northamerica-south1).
]]>Cloud Composer 3 is now generally available (GA):
When creating new environments in Google Cloud console, it's now required to explicitly select a service account for the environment. We recommend to create a user-managed service account and use it for Cloud Composer environments.
]]>In January 2025, we plan to release Cloud Composer 2 versions that will always use the environment's service account for performing PyPI packages installations:
(Cloud Composer 3) It is now possible to upgrade an environment if the [sentry]sentry_on Airflow configuration option is set to true.
The COMPOSER_AGENT_BUILD_SERVICE_ACCOUNT environment variable is changed to reserved. This change improves the security of Cloud Composer environments.
(Cloud Composer 2) Airflow worker liveness check configuration was changed to be consistent with the configuration used in Cloud Composer 3. In particular, this change increases the timeout, giving the liveness check more time to detect unhealthy Airflow workers. This change is gradually rolled out to all regions supported by Cloud Composer.
Cloud Composer no longer adds any missing IAM permissions to the Cloud Storage bucket when it is used to create an environment with a custom environment's bucket. Make sure that the environment's service account has permissions from the Composer Worker role on the bucket.
Increased allowed timeouts when detecting tasks stuck in the "queued" state during the Airflow worker liveness check. This change makes it less likely that checks will incorrectly fail in specific scenarios. This change is gradually rolled out to all regions supported by Cloud Composer.
Cloud Composer version 2.5.3 has reached its end of support period.
New images are available in Cloud Composer 2:
New Airflow builds are available in Cloud Composer 3:
New Cloud Composer 3 environments can now be created in VPC SC. This feature is gradually rolled out to all regions supported by Cloud Composer.
Improved Airflow worker liveness checks to detect workers with unexpected idle task slots. This feature improves the stability of Airflow by better detection of unhealthy Airflow workers. This feature is gradually rolled out to all regions supported by Cloud Composer.
Fixed the issue in the environment's component responsible for uploading the logs of Airflow components to Cloud Logging. This bug sometimes lead to a situation where Cloud Composer-level log might be missing for an Airflow component. The same log was still available on the Kubernetes-component level.
(Cloud Composer 3) KubernetesPodOperator now works when the do_xcom_push parameter is set to True.
(Cloud Composer 2) If an upgrade operation fails, Cloud Composer 2 now restores the environment with the correct number of triggers.
(Cloud Composer 2 only) It is now possible to upgrade an environment if the [sentry]sentry_on Airflow configuration option is set to true.
Long log entries now have proper task instance annotations.
(New Cloud Composer 3 environments) Increased the maximum number of internet connections that each Airflow worker can support at the same time.
The maximum limit on the database size during upgrades in Cloud Composer 3 is now the same as the limit for snapshots (20 GB).
The aiohttp package was downgraded from 3.11.0 to 3.10.11.
(Airflow 2.10.2 and 2.9.3) The apache-airflow-providers-google package was upgraded to version 10.26.0 in Cloud Composer 2 images and Cloud Composer 3 builds. For more information about changes, see the apache-airflow-providers-google changelog from version 10.25.0 to version 10.26.0.
(Available without upgrading) Fixed an issue where Airflow workers sometimes generated incomplete or unreadable output.
(Airflow 2.10.2 and 2.9.3) The apache-airflow-providers-cncf-kubernetes package was upgraded to version 10.0.1 in Cloud Composer 2 images and Cloud Composer 3 builds. For more information about changes, see the apache-airflow-providers-cncf-kubernetes changelog from version 9.0.1 to version 10.0.1.
The default version of Airflow is changed to 2.10.2.
Airflow 2.7.3 is no longer included in Cloud Composer images and builds.
New images are available in Cloud Composer 2:
Cloud Composer version 2.5.2 has reached its end of support period.
New Airflow builds are available in Cloud Composer 3:
Scheduled snapshots are available in Cloud Composer 3. This feature will be gradually rolled out to all regions supported by Cloud Composer 3.
Cloud Composer 2 is now available in Mexico (northamerica-south1).
]]>All Cloud Composer environment's GKE clusters are set up with maintenance exclusions from December 20, 2024 to January 2, 2025. For more information, see Maintenance exclusions.
]]>In December 2024, Google will remove the following previously deprecated Airflow operators from the apache-airflow-providers-google package.
The new version of this package will be included in one of the future releases of Cloud Composer and the change will be announced in the Release Notes. After this change, it will not be possible to use these operators in your DAGs.
Make sure that you use up-to-date alternatives of the removed operators instead. For more information about removed and deprecated Airflow operators and their up-to-date alternatives, see Deprecated and removed Airflow operators.
Operators that will be removed in December 2024: DataPipelineHook, CreateDataPipelineOperator, RunDataPipelineOperator, AutoMLDatasetLink, AutoMLDatasetListLink, AutoMLModelLink, AutoMLModelTrainLink, AutoMLModelPredictLink.
]]>Airflow 2.10.2 is available in Cloud Composer.
(Airflow 2.7.3) Backported #35887 to fix an issue that occurred during the DST transition. The issue affected DAGs with timezone-aware cron schedule and caused infinite loops in the Airflow scheduler.
Improved the error message generated when a Cloud Composer 3 environment creation fails because of missing permissions.
(Airflow 2.10.2 and 2.9.3) The apache-airflow-providers-cncf-kubernetes package was upgraded to version 9.0.1 in Cloud Composer 2 images and Cloud Composer 3 builds. For more information about changes, see the apache-airflow-providers-cncf-kubernetes changelog from version 9.0.0 to version 9.0.1.
(Airflow 2.10.2 and 2.9.3) The apache-airflow-providers-google package was upgraded to version 10.25.0 in Cloud Composer 2 images and Cloud Composer 3 builds. For more information about changes, see the apache-airflow-providers-google changelog from version 10.24.0 to version 10.25.0.
New images are available in Cloud Composer 2:
New Airflow builds are available in Cloud Composer 3:
Support dates for previous Cloud Composer 3 builds are available. All Cloud Composer 3 builds with Airflow 2.9.3 are supported until November 13, 2025.
]]>(Airflow 2.9.3 and 2.7.3) The docutils package was removed from preinstalled packages.
(Cloud Composer 3) Fixed an issue that affected the speed of PyPI package installation. PyPI packages are now installed slightly faster.
New images are available in Cloud Composer 2:
New Airflow builds are available in Cloud Composer 3:
Cloud Composer 2.9.7 is a version with an extended upgrade timeline.
Cloud Composer version 2.5.1 has reached its end of support period.
]]>