Internet network endpoint groups overview

Cloud Load Balancing supports proxying traffic to external backends outside Trusted Cloud by S3NS. To define an external backend for a load balancer, you use a resource called an internet network endpoint group (NEG).

You can use this type of deployment when you want to serve content from an external backend, but you want your Trusted Cloud load balancer to be the frontend. This lets you do the following:

• Use Google Edge infrastructure for terminating your user connections.
• Direct the connections to your external backend.
• Deliver traffic to your public endpoint across Google's private backbone, which improves reliability and can decrease latency between client and server.

The sections that follow explain how external backends are used with Cloud Load Balancing. If you want to use an external backend with Cloud Service Mesh, see Cloud Service Mesh with internet network endpoint groups.

Terminology

The following terms are sometimes used interchangeably because they have the same or similar meanings:

• external backend: A backend that resides outside of Trusted Cloud by S3NS and is reachable across the internet. The endpoint in an internet NEG.
• internet network endpoint group (NEG): The Trusted Cloud by S3NS API resource that you use to specify an external backend.
• external endpoint: Same as an external backend.

This document uses the term external backend except when referring to the internet NEG API resource.

Load balancer components

This section describes the load balancing architecture and resources required to configure a load balancer with an external backend. The load balancer requires special configuration only for the backend service. The frontend configuration is the same as any other load balancer.

The following figures show the Trusted Cloud resources required to set up a load balancer with an external backend.

You can only use internet NEGs on the Premium network service tier.

Frontend configuration

No special frontend configuration is required for creating a load balancer with an internet NEG backend. Forwarding rules are used to route traffic by IP address, port, and protocol to a target proxy. The target proxy then terminates connections from clients. Additionally, Envoy-based load balancers require a proxy-only subnet.

Application Load Balancers also use URL maps to set up URL-based routing of requests to the appropriate backend services.

For more details about each of these components, refer to the architecture sections of the respective load balancer:

• External Application Load Balancer overview
• Internal Application Load Balancer overview
• External proxy Network Load Balancer overview
• Internal proxy Network Load Balancer overview

Internet NEG

An internet NEG is a resource used to define an external backend for the load balancer. There are two types of internet NEGs: global internet NEG and regional internet NEG. They differ in the scope (global versus regional) and behavior. The external backend referenced by a global internet NEG must be reachable exclusively over the internet; it cannot be reachable over Cloud VPN or Cloud Interconnect. If the external backend references a Google API or service, the service must be reachable through TCP port 80 or 443 by using the HTTP, HTTPS, or HTTP/2 protocol.

There are two ways to configure the external endpoint referenced by the NEG: INTERNET_FQDN_PORT or INTERNET_IP_PORT. If the INTERNET_IP_PORT format is chosen, only a public internet routable IP address can be used; if the INTERNET_FQDN_PORT format is chosen the FQDN can be resolved to either a public internet routable IP address or to a private IP address depending on the scope of the endpoint: regional or global.

Regional internet NEGs are powered by managed Envoy proxies. Each NEG can have multiple endpoints, and a backend service can include multiple internet NEGs.

For egress traffic, you can configure Cloud NAT gateways to set the source IP addresses. Alternatively, you can route traffic using learned routes within your VPC network. While Cloud NAT isn't required for this routing method, it is supported.

The following table describes how different load balancers support regional internet NEGs.

Load balancers	Endpoint type	Endpoint definition	Scope	Health checks
Regional external Application Load Balancer Regional internal Application Load Balancer Regional external proxy Network Load Balancer Regional internal proxy Network Load Balancer	INTERNET_FQDN_PORT	Either a publicly or privately resolvable fully qualified domain name and an optional port. For example, backend.example.com:443*. The domain name must be resolvable by Cloud DNS. A maximum of 256 endpoints per NEG are allowed.	Regional	Envoy distributed health checks
	INTERNET_IP_PORT	Only a publicly routable IP address and an optional port. For example, 8.8.8.8:4432. The IP address cannot be an RFC 1918 address. A maximum of 256 endpoints per NEG are allowed.

DNS resolution for regional INTERNET_FQDN_PORT endpoints

If your domain is resolvable over the internet, no other configuration is needed to set up DNS. However, if you're resolving private FQDNs, you'll need to configure Cloud DNS to facilitate DNS resolution. The name must be hosted on Cloud DNS or be resolvable using DNS forwarding from Cloud DNS to an on-premises DNS or DNS peering if referencing a Private DNS zone in another VPC network.

Start by creating a Cloud DNS zone to host the DNS records in your project. Then add the DNS records to it. For specific configuration steps, see the Cloud DNS documentation. The Cloud DNS resolution order is detailed at Name resolution order.

If you're using Shared VPC, note the specific network requirements. You can also use other features of Cloud DNS, such as forwarding zones, to fetch records from an on-premises DNS server.

IP address resolution for regional INTERNET_FQDN_PORT endpoints

Regional internet NEGs support domain name resolution using Cloud DNS.

If the DNS server returns multiple IP addresses, Envoy load balances traffic among the returned IP addresses based on the configured load balancing algorithm (round robin, least request, and so on). The list of endpoints is periodically refreshed based on DNS TTL. You can configure retry policies to force Envoy to attempt to connect to another IP address if one fails.

Backend service

Backend services provide configuration information to the load balancer. Load balancers use the information in a backend service to direct incoming traffic to one or more attached backends.

To set up a load balancer with an external backend, you configure the backend service with an internet NEG backend. When you add an internet NEG to a backend service, the following considerations apply, depending on the scope of the NEG:

• The backend service cannot also use other backend types (such as zonal NEGs or instance groups) as backends.

• Number of NEGs per backend service

  • Regional NEGs. You can add up to 50 internet NEGs to the same backend service.

• Number of endpoints per NEG

  • Regional NEGs. You can add up to 256 endpoints per NEG to the same backend service.

  Regional NEGs don't support load balancing modes, such as rate, connection, or utilization. All the endpoints of all the NEGs attached to a backend service are pooled into a single group. Load balancing traffic among this pool of endpoints is handled using Envoy load balancing algorithms. For the load balancing policy algorithms supported, see localityLbPolicy in the regional backend service API documentation.

• Health checks

  • Regional NEGs. The load balancer uses distributed Envoy health checks.

• The backend service's load balancing scheme must match the scheme required by the load balancer you are deploying. For the complete list, see Backend services.

• The backend service protocol must be one of HTTP, HTTPS, or HTTP2.

  We strongly recommend you use either HTTPS or HTTP/2 as the protocol when configuring a backend service with an internet NEG so that communication between the load balancer and the backend is encrypted and authenticated when transiting the public internet.

  Additionally, when using HTTPS or HTTP/2 as the backend protocol, make sure that you use an INTERNET_FQDN_PORT endpoint to create your external backend. This has two advantages:

  • It ensures that the load balancer validates the SSL server certificate presented by the external backend and verifies that the following information is true:

    • The certificate is signed by well-known certificate authorities (CAs).
    • The certificate is not expired.
    • The certificate signature is valid.
    • The configured FQDN matches one of the Subject Alternative Names (SANs) in the certificate.

    If you create the external backend by using an INTERNET_IP_PORT endpoint, SSL server certificate validation is not performed.

  • The SSL Server Name Indication (SNI) extension is only supported with INTERNET_FQDN_PORT endpoints. The configured FQDN is sent an SNI in the client hello during the SSL handshake between the load balancer and the external endpoint. The SNI is not sent when you use an INTERNET_IP_PORT endpoint because IP address literals aren't allowed in the HostName field of an SNI payload.

Health checks

Your health check configuration varies depending on the type of load balancer:

• Regional external Application Load Balancer, regional internal Application Load Balancer, regional external proxy Network Load Balancer, and regional internal proxy Network Load Balancer. Health checks are optional. Health check probes for these load balancers originate from the proxy-only subnet and are then NAT-translated (by using Cloud NAT) to either pre-reserved IP addresses or auto-allocated NAT IP addresses. For details, see Regional NEGs: Use a Cloud NAT gateway.

  Distributed Envoy health checks are created by using the same Trusted Cloud console, gcloud CLI, and API processes as centralized health checks. No other configuration is required.

  Points to note:

  • gRPC health checks are not supported.
  • Health checks with PROXY protocol v1 enabled are not supported.

  • Because the Envoy data plane handles health checks, you cannot use the Trusted Cloud console, the API, or the gcloud CLI to check the health status of these external endpoints. For hybrid NEGs with Envoy-based load balancers, the Trusted Cloud console shows the health check status as N/A. This is expected.

  • Every Envoy proxy assigned to the proxy-only subnet in the region in the VPC network initiates health checks independently. Therefore, you might see an increase in network traffic because of health checking. The increase depends on the number of Envoy proxies assigned to your VPC network in a region, the amount of traffic received by these proxies, and the number of endpoints that each Envoy proxy needs to health check. In the worst case scenario, network traffic because of health checks increases at a quadratic (O(n^2)) rate.

  • Health check logs for distributed Envoy health checks don't include detailed health states. For details about what is logged, see Health check logging. To further troubleshoot connectivity from Envoy proxies to your NEG endpoints, you should also check the respective load balancer logs.

    • Regional external Application Load Balancer logging
    • Internal Application Load Balancer logging
    • Proxy Network Load Balancer logging

Enable the external backend to receive requests

Configure your external backend to allow traffic from Trusted Cloud.

Regional NEGs: Use a Cloud NAT gateway

If you're using a regional internet NEG, you'll first set up a Cloud NAT gateway to allocate a set of IP address ranges from where Trusted Cloud traffic should originate.

The gateway endpoint should be of type ENDPOINT_TYPE_MANAGED_PROXY_LB.

The Cloud NAT gateway can be configured to either automatically allocate external IP addresses based on demand or use a manually pre-reserved set of external IP addresses.

• Automatically allocated IP addresses

  Use automatically allocated IP addresses if your external backend environment doesn't require you to add specific Trusted Cloud IP addresses that can send traffic to the external backend to an allowlist.

• Manually allocated IP addresses

  Use manually allocated IP addresses only if your external backend environment requires you to add specific Trusted Cloud IP addresses to an allowlist. Because each Envoy assigned to your proxy subnet consumes an entire IP address, make sure that the pool of reserved IP addresses is big enough to accommodate all Envoys.

  If you experience connectivity issues at scale, check whether you've reached the Cloud NAT limits. By default, you are limited to 50 manually allocated NAT IP addresses per gateway.

Dynamic port allocation is supported for regional internet NEGs. IP addresses can be shared by proxies, thus fully utilized.

This Cloud NAT configuration applies to the entire proxy-only subnet. Internet traffic associated with all the regional Envoy-based load balancers in the region share the same NAT gateway.

Cloud NAT usage incurs charges for both user traffic and health check traffic. For details about how regional internet NEG pricing works, see Regional internet NEG pricing.

There are certain limitations for NAT gateways configured on proxy-only subnets:

• Only one-to-one NAT translation is performed. IP address sharing is not supported.
• Logging and monitoring are not supported. That is, the --enable-logging and --log-filter flags are not supported.

Authenticate requests to the external backend

This section applies only to Application Load Balancers.

To authenticate requests sent to your external backend, you can do one of the following:

• Set a custom header to indicate that the request came from a Trusted Cloud load balancer by using a custom request header. For example, you can use 16 or more cryptographically random bytes as a shared key.

  Implementing custom header transformations depends on the type of load balancer you're using:

  • Regional external Application Load Balancer and regional internal Application Load Balancer. Custom header transformations can only be configured on the URL map.

    For these Envoy-based load balancers, Host and authority are special keywords reserved by Trusted Cloud. You can't modify these headers for these load balancers. Instead, we recommend that you create other custom headers (for example, MyHost) so that you don't interfere with the reserved header names.

• Enable IAP and verify that the signed JWT in the request header is signed by Google, and that the aud (audience) claim contains the project number where your load balancer is defined.

Logs

Requests proxied to an external backend are logged to Cloud Logging in the same way that requests for other backends are logged.

For more information, see the following:

• Regional external Application Load Balancer logging
• Regional internal Application Load Balancer logging
• Proxy Network Load Balancer logging

Limitations

• Review the backend service section for limitations associated with configuring internet NEGs as backends.
• When you modify a load balancer to change the backend from an internet NEG to any other backend type, or change the backend from any other backend type to an internet NEG, your application experiences a temporary downtime of about 30-90 seconds.

• Review the Cloud NAT gateway section for limitations associated with NAT gateways configured on proxy-only subnets.

Quotas and limits

For information about quotas and limits, see the NEG backends quota table and the Endpoints per NEG quota table.

Pricing

Egress traffic to external internet NEG endpoints is charged at internet egress rates for Premium Tier networking. The source is based on the client location, and the destination is based on the location of your public endpoint.

If you configured a Cloud NAT gateway to map your regional Envoy-based load balancer's proxy-only subnet, you'll incur Cloud NAT charges. Cloud NAT gateways allocated for load balancers incur hourly charges equivalent to a network with more than 32 VM instances. For details, see Cloud NAT pricing.

For more information, see Cloud Load Balancing pricing.

What's next

• Set up a regional external Application Load Balancer with an internet NEG
• Set up a regional external proxy Network Load Balancer with an internet NEG