External proxy Network Load Balancer overview

This document introduces the concepts that you need to understand to configure a Trusted Cloud external proxy Network Load Balancer.

The external proxy Network Load Balancer is a reverse proxy load balancer that distributes TCP traffic coming from the internet to virtual machine (VM) instances in your Trusted Cloud Virtual Private Cloud (VPC) network. When using an external proxy Network Load Balancer, incoming TCP traffic is terminated at the load balancer. A new connection then forwards traffic to the closest available backend.

External proxy Network Load Balancers let you use a single IP address for all users worldwide. The load balancer automatically routes traffic to the backends that are closest to the user.

This load balancer is available to you in the regional mode, and is hereafter referred to as a regional external proxy Network Load Balancer. The load balancer is implemented as a managed service based on the open-source Envoy proxy. The regional mode ensures that all clients and backends are in a specified region. Use this load balancer if you want to serve content from only one geolocation (for example, to meet compliance regulations).

Architecture

The following diagrams show the components of external proxy Network Load Balancers.

Regional

This diagram shows the components of a regional external proxy Network Load Balancer deployment.

Regional external proxy Network Load Balancer components.
Regional external proxy Network Load Balancer components (click to enlarge).

The following are components of external proxy Network Load Balancers.

Proxy-only subnet

The proxy-only subnet provides a set of IP addresses that Google uses to run Envoy proxies on your behalf. You must create one proxy-only subnet in each region of a VPC network where you use load balancers. The --purpose flag for this proxy-only subnet is set to REGIONAL_MANAGED_PROXY. All regional Envoy-based load balancers in the same region and VPC network share a pool of Envoy proxies from the same proxy-only subnet.

Backend VMs or endpoints of all load balancers in a region and a VPC network receive connections from the proxy-only subnet.

Points to remember:

  • Proxy-only subnets are only used for Envoy proxies, not your backends.
  • The IP address of the load balancer is not located in the proxy-only subnet. The load balancer's IP address is defined by its external managed forwarding rule.

Forwarding rules and IP addresses

Forwarding rules route traffic by IP address, port, and protocol to a load balancing configuration that consists of a target proxy and a backend service.

IP address specification. Each forwarding rule references a single IP address that you can use in DNS records for your application. You can either reserve a static IP address that you can use or let Cloud Load Balancing assign one for you. We recommend that you reserve a static IP address. Otherwise, you must update your DNS record with the newly assigned ephemeral IP address whenever you delete a forwarding rule and create a new one.

Port specification. External forwarding rules used in the definition of this load balancer can reference exactly one port from 1-65535. If you want to support multiple consecutive ports, you need to configure multiple forwarding rules. Multiple forwarding rules can be configured with the same virtual IP address and different ports; therefore, you can proxy multiple applications with separate custom ports to the same TCP proxy virtual IP address. For more details, see Port specifications for forwarding rules.

To support multiple consecutive ports, you have to configure multiple forwarding rules. Multiple forwarding rules can be configured with the same virtual IP address and different ports. Therefore, you can proxy multiple applications with separate custom ports to the same TCP proxy virtual IP address.

The following table shows the forwarding rule requirements for external proxy Network Load Balancers.

Load balancer mode Network Service Tier Forwarding rule, IP address, and load balancing scheme Routing from the internet to the load balancer frontend
Regional external proxy Network Load Balancer Premium Tier

Regional external forwarding rule

Regional external IP address

Load balancing scheme: EXTERNAL_MANAGED

 Requests routed to the Envoy proxies in the same region as the load balancer.

Forwarding rules and VPC networks

This section describes how forwarding rules used by external Application Load Balancers are associated with VPC networks.

Load balancer mode VPC network association
Regional external proxy Network Load Balancer

The forwarding rule's VPC network is the network where the proxy-only subnet has been created. You specify the network when you create the forwarding rule.

Target proxies

External proxy Network Load Balancers terminate connections from the client and create new connections to the backends. The target proxy routes these new connections to the backend service.

By default, the target proxy does not preserve the original client IP address and port information. You can preserve this information by enabling the PROXY protocol on the target proxy.

The following table shows the target proxy requirements for external proxy Network Load Balancers.

Load balancer mode Network Service Tier Target proxy
Regional external proxy Network Load Balancer Premium and Standard Tier regionTargetTcpProxies

Backend services

Backend services direct incoming traffic to one or more attached backends. Each backend is composed of an instance group or network endpoint group and information about the backend's serving capacity. Backend serving capacity can be based on CPU or requests per second (RPS).

Each load balancer has a single backend service resource that specifies the health check to be performed for the available backends.

Changes made to the backend service are not instantaneous. It can take several minutes for changes to propagate to GFEs. To ensure minimal interruptions to your users, you can enable connection draining on backend services. Such interruptions might happen when a backend is terminated, removed manually, or removed by an autoscaler. To learn more about using connection draining to minimize service interruptions, see Enabling connection draining.

For more information about the backend service resource, see Backend services overview.

The following table specifies the different backends supported on the backend service of external proxy Network Load Balancers.

Load balancer mode Supported backends on a backend service
Instance groups Zonal NEGs Internet NEGs Serverless NEGs Hybrid NEGs GKE
Regional external proxy Network Load Balancer GCE_VM_IP_PORT type endpoints Regional NEGs only

Backends and VPC networks

For regional external proxy Network Load Balancer backends, the following applies:

  • For instance groups, zonal NEGs, and hybrid connectivity NEGs, all backends must be located in the same project and region as the backend service. However, a load balancer can reference a backend that uses a different VPC network in the same project as the backend service. Connectivity between the load balancer's VPC network and the backend VPC network can be configured using either VPC Network Peering, Cloud VPN tunnels, or Cloud Interconnect VLAN attachments.

    Backend network definition

    • For zonal NEGs and hybrid NEGs, you explicitly specify the VPC network when you create the NEG.
    • For managed instance groups, the VPC network is defined in the instance template.
    • For unmanaged instance groups, the instance group's VPC network is set to match the VPC network of the nic0 interface for the first VM added to the instance group.

    Backend network requirements

    Your backend's network must satisfy one of the following network requirements:

    • The backend's VPC network must exactly match the forwarding rule's VPC network.

    • The backend's VPC network must be connected to the forwarding rule's VPC network using VPC Network Peering. You must configure subnet route exchanges to allow communication between the proxy-only subnet in the forwarding rule's VPC network and the subnets used by the backend instances or endpoints.

  • For all other backend types, all backends must be located in the same VPC network and region.

Backends and network interfaces

If you use instance group backends, packets are always delivered to nic0. If you want to send packets to non-nic0 interfaces (either vNICs or Dynamic Network Interfaces), use NEG backends instead.

If you use zonal NEG backends, packets are sent to whatever network interface is represented by the endpoint in the NEG. The NEG endpoints must be in the same VPC network as the NEG's explicitly defined VPC network.

Protocol for communicating with the backends

When you configure a backend service for an external proxy Network Load Balancer, you set the protocol that the backend service uses to communicate with the backends. The load balancer uses only the protocol that you specify, and does not attempt to negotiate a connection with the other protocol. The external proxy Network Load Balancers only support TCP for communicating with the backends.

Firewall rules

The following firewall rules are required:

  • For regional external proxy Network Load Balancers, an ingress firewall rule to permit traffic from the proxy-only subnet to reach your backends.

  • An ingress allow firewall rule to permit traffic from the health check probe ranges to reach your backends. For more information about health check probes and why it's necessary to allow traffic from them, see Probe IP ranges and firewall rules.

Firewall rules are implemented at the VM instance level, not at the GFE proxies level. You cannot use firewall rules to prevent traffic from reaching the load balancer.

The ports for these firewall rules must be configured as follows:

  • Allow traffic to the destination port for each backend service's health check.
  • For instance group backends: determine the ports to be configured by the mapping between the backend service's named port and the port numbers associated with that named port on each instance group. Port numbers can vary between instance groups assigned to the same backend service.
  • For GCE_VM_IP_PORT NEG zonal NEG backends: allow traffic to the port numbers of the endpoints.

Source IP addresses

The source IP address for packets, as seen by the backends, is not the Trusted Cloud external IP address of the load balancer. In other words, there are two TCP connections.

For the regional external proxy Network Load Balancers:

  • Connection 1, from original client to the load balancer (proxy-only subnet):

    • Source IP address: the original client (or external IP address if the client is behind a NAT gateway or a forward proxy).
    • Destination IP address: your load balancer's IP address.

  • Connection 2, from the load balancer (proxy-only subnet) to the backend VM or endpoint:

    • Source IP address: an IP address in the proxy-only subnet that is shared among all the Envoy-based load balancers deployed in the same region and network as the load balancer.

    • Destination IP address: the internal IP address of the backend VM or container in the VPC network.

Open ports

External proxy Network Load Balancers are reverse proxy load balancers. The load balancer terminates incoming connections, and then opens new connections from the load balancer to the backends. These load balancers are implemented by using Google Front End (GFE) proxies worldwide.

GFEs have several open ports to support other Google services that run on the same architecture. When you run a port scan, you might see other open ports for other Google services running on GFEs.

Running a port scan on the IP address of a GFE-based load balancer isn't useful from an auditing perspective for the following reasons:

  • A port scan (for example, with nmap) generally expects no response packet or a TCP RST packet when performing TCP SYN probing. GFEs send SYN-ACK packets in response to SYN probes only for ports on which you have configured a forwarding rule. GFEs only send packets to your backends in response to packets sent to your load balancer's IP address and the destination port configured on its forwarding rule. Packets that are sent to a different IP address or port aren't sent to your backends.

    GFEs implement security features such as Google Cloud Armor. With Cloud Armor Standard, GFEs provide always-on protection from volumetric and protocol-based DDoS attacks and SYN floods. This protection is available even if you haven't explicitly configured Cloud Armor. You are charged only if you configure security policies or if you enroll in Managed Protection Plus.

  • Packets sent to the IP address of your load balancer can be answered by any GFE in Google's fleet; however, scanning a load balancer IP address and destination port combination only interrogates a single GFE per TCP connection. The IP address of your load balancer isn't assigned to a single device or system. Thus, scanning the IP address of a GFE-based load balancer doesn't scan all the GFEs in Google's fleet.

With that in mind, the following are some more effective ways to audit the security of your backend instances:

  • A security auditor should inspect the forwarding rules configuration for the load balancer's configuration. The forwarding rules define the destination port for which your load balancer accepts packets and forwards them to the backends. For GFE-based load balancers, each external forwarding rule can only reference a single destination TCP port.

  • A security auditor should inspect the firewall rule configuration applicable to backend VMs. The firewall rules that you set block traffic from the GFEs to the backend VMs but don't block incoming traffic to the GFEs. For best practices, see the firewall rules section.

Shared VPC architecture

Regional external proxy Network Load Balancers support deployments that use Shared VPC networks. Shared VPC lets you maintain a clear separation of responsibilities between network administrators and service developers. Your development teams can focus on building services in service projects, and the network infrastructure teams can provision and administer load balancing. If you're not already familiar with Shared VPC, read the Shared VPC overview documentation.

IP address Forwarding rule Target proxy Backend components
An external IP address must be defined in the same project as the load balancer. The external forwarding rule must be defined in the same project as the backend instances (the service project). The target TCP proxy must be defined in the same project as the backend instances.

For regional external proxy Network Load Balancers, the backend VMs are typically located in a service project. A regional backend service and health check must be defined in that service project.

Traffic distribution

When you add a backend instance group or NEG to a backend service, you specify a load balancing mode, which defines a method that measures the backend load and target capacity.

For external proxy Network Load Balancers, the balancing mode can be CONNECTION or UTILIZATION:

  • If the load balancing mode is CONNECTION, the load is spread based on the total number of connections that the backend can handle.
  • If the load balancing mode is UTILIZATION, the load is spread based on the utilization of instances in an instance group. This balancing mode applies to VM instance group backends only.

The distribution of traffic across backends is determined by the balancing mode of the load balancer.

Regional external proxy Network Load Balancer

For regional external proxy Network Load Balancers, traffic distribution is based on the load balancing mode and the load balancing locality policy.

The balancing mode determines the weight and fraction of traffic that should be sent to each backend (instance group or NEG). The load balancing locality policy (LocalityLbPolicy) determines how backends within the group are load balanced.

When a backend service receives traffic, it first directs traffic to a backend (instance group or NEG) according to its balancing mode. After a backend is selected, traffic is then distributed among instances or endpoints in that backend group according to the load balancing locality policy.

For more information, see the following:

Session affinity

Session affinity lets you configure the load balancer's backend service to send all requests from the same client to the same backend, as long as the backend is healthy and has capacity.

External proxy Network Load Balancers offer the following types of session affinity:

  • None

    A session affinity setting of NONE does not mean that there is no session affinity. It means that no session affinity option is explicitly configured.

    Hashing is always performed to select a backend. And a session affinity setting of NONE means that the load balancer uses a 5-tuple hash to select a backend. The 5-tuple hash consists of the source IP address, the source port, the protocol, the destination IP address, and the destination port.

    A session affinity of NONE is the default value.

  • Client IP affinity

    Client IP session affinity (CLIENT_IP) is a 2-tuple hash created from the source and destination IP addresses of the packet. Client IP affinity forwards all requests from the same client IP address to the same backend, as long as that backend has capacity and remains healthy.

    When you use client IP affinity, keep the following in mind:

    • The packet destination IP address is only the same as the load balancer forwarding rule's IP address if the packet is sent directly to the load balancer.
    • The packet source IP address might not match an IP address associated with the original client if the packet is processed by an intermediate NAT or proxy system before being delivered to a Trusted Cloud load balancer. In situations where many clients share the same effective source IP address, some backend VMs might receive more connections or requests than others.

Keep the following in mind when configuring session affinity:

  • Don't rely on session affinity for authentication or security purposes. Session affinity can break whenever the number of serving and healthy backends changes. For more details, see Losing session affinity.

  • The default values of the --session-affinity and --subsetting-policy flags are both NONE, and only one of them at a time can be set to a different value.

Losing session affinity

All session affinity options require the following:

  • The selected backend instance or endpoint must remain configured as a backend. Session affinity can break when one of the following events occurs:
    • You remove the selected instance from its instance group.
    • Managed instance group autoscaling or autohealing removes the selected instance from its managed instance group.
    • You remove the selected endpoint from its NEG.
    • You remove the instance group or NEG that contains the selected instance or endpoint from the backend service.
  • The selected backend instance or endpoint must remain healthy. Session affinity can break when the selected instance or endpoint fails health checks.
  • For Global external proxy Network Load Balancers and Classic proxy Network Load Balancers, session affinity can break if a different first-layer Google Front End (GFE) is used for subsequent requests or connections after the change in routing path. A different first-layer GFE might be selected if the routing path from a client on the internet to Google changes between requests or connections.
All session affinity options have the following additional requirements:

  • The instance group or NEG that contains the selected instance or endpoint must not be full as defined by its target capacity. (For regional managed instance groups, the zonal component of the instance group that contains the selected instance must not be full.) Session affinity can break when the instance group or NEG is full and other instance groups or NEGs are not. Because fullness can change in unpredictable ways when using the UTILIZATION balancing mode, you should use the RATE or CONNECTION balancing mode to minimize situations when session affinity can break.

  • The total number of configured backend instances or endpoints must remain constant. When at least one of the following events occurs, the number of configured backend instances or endpoints changes, and session affinity can break:

    • Adding new instances or endpoints:

      • You add instances to an existing instance group on the backend service.
      • Managed instance group autoscaling adds instances to a managed instance group on the backend service.
      • You add endpoints to an existing NEG on the backend service.
      • You add non-empty instance groups or NEGs to the backend service.

    • Removing any instance or endpoint, not just the selected instance or endpoint:

      • You remove any instance from an instance group backend.
      • Managed instance group autoscaling or autohealing removes any instance from a managed instance group backend.
      • You remove any endpoint from a NEG backend.
      • You remove any existing, non-empty backend instance group or NEG from the backend service.

  • The total number of healthy backend instances or endpoints must remain constant. When at least one of the following events occurs, the number of healthy backend instances or endpoints changes, and session affinity can break:

    • Any instance or endpoint passes its health check, transitioning from unhealthy to healthy.
    • Any instance or endpoint fails its health check, transitioning from healthy to unhealthy or timeout.

Failover

Failover for external proxy Network Load Balancers works as follows:

  • If a backend becomes unhealthy, traffic is automatically redirected to healthy backends within the same region.
  • If all backends are unhealthy, the load balancer drops traffic.

Load balancing for GKE applications

If you are building applications in Google Kubernetes Engine, you can use standalone NEGs to load balance traffic directly to containers. With standalone NEGs, you are responsible for creating the Service object that creates the NEG, and then associating the NEG with the backend service so that the load balancer can connect to the Pods.

For related documentation, see Container-native load balancing through standalone zonal NEGs.

Limitations

  • You can't create a regional external proxy Network Load Balancer in Premium Tier using the Trusted Cloud console. Use either the gcloud CLI or the API instead.

  • Trusted Cloud does not make any guarantees on the lifetime of TCP connections when you use external proxy Network Load Balancers. Clients should be resilient to dropped connections, both due to broader internet issues and due to regularly scheduled restarts of the proxies managing the connections.

What's next