Simulate a zone outage for a regional MIG

To test that your regional managed instance group (MIG) is overprovisioned enough and can survive a zone outage, you can use the following example to simulate a zonal failure.

Before you begin

  • If you want to use the command-line examples in this guide, install the Google Cloud CLI.
  • If you haven't already, set up authentication. Authentication verifies your identity for access to Trusted Cloud by S3NS services and APIs. To run code or samples from a local development environment, you can authenticate to Compute Engine by selecting one of the following options:

    Select the tab for how you plan to use the samples on this page:

    gcloud

    1. Install the Google Cloud CLI, and then sign in to the gcloud CLI with your federated identity. After signing in, initialize the Google Cloud CLI by running the following command: 

      gcloud init
    2. Set a default region and zone.

    REST

    To use the REST API samples on this page in a local development environment, you use the credentials you provide to the gcloud CLI.

      Install the Google Cloud CLI, and then sign in to the gcloud CLI with your federated identity. After signing in, initialize the Google Cloud CLI by running the following command: 

      gcloud init

    For more information, see Authenticate for using REST in the Trusted Cloud authentication documentation.

Use a script to simulate a zone outage

This script stops and starts Apache as the default scenario. If this doesn't apply to your application, replace the commands that stop and start Apache with your own failure and recovery scenario.

  1. Deploy and run this script continuously in every VM in the group. You can do this by adding the script to the instance template or by including the script in a custom image and using the image in the instance template.

    #!/usr/bin/env bash

# Copyright 2016 Google Inc. All Rights Reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
#     http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.

set -o nounset
set -o errexit
set -o pipefail

function GetMetadata() {
  curl -s "$1" -H "Metadata-Flavor: Google"
}

PROJECT_METADATA_URL="http://metadata.google.internal/computeMetadata/v1/project/attributes"
INSTANCE_METADATA_URL="http://metadata.google.internal/computeMetadata/v1/instance"
ZONE=$(GetMetadata "$INSTANCE_METADATA_URL/zone" | cut -d '/' -f 4)
INSTANCE_NAME=$(hostname)

# We keep track of the state to make sure failure and recovery is triggered only once.
STATE="healthy"
while true; do
  if [[ "$ZONE" = "$(GetMetadata $PROJECT_METADATA_URL/failed_zone)" ]] && \
     [[ "$INSTANCE_NAME" = *"$(GetMetadata $PROJECT_METADATA_URL/failed_instance_names)"* ]]; then
    if [[ "$STATE" = "healthy" ]]; then
      STATE="failure"
      # Do something to simulate failure here.
      echo "STARTING A FAILURE"
      /etc/init.d/apache2 stop
    fi
  else
    if [[ "$STATE" = "failure" ]] ; then
      STATE="healthy"
      # Do something to recover here.
      echo "RECOVERING FROM FAILURE"
      /etc/init.d/apache2 start
    fi
  fi
  sleep 5
done

  2. Simulate a zone failure by setting these two project metadata fields:

    • failed_zone: Sets the zone where you want to simulate the outage (limit the failure to just one zone).
    • failed_instance_names: Choose the VMs to take offline by name (to limit the failure to only VM names containing this string).

    You can set this metadata using the gcloud CLI. For example, the following command sets the zone outage to the europe-west1-b zone and affects VMs that have names starting with base-instance-name:

    gcloud compute project-info add-metadata --metadata failed_zone='europe-west1-b',failed_instance_names='base-instance-name-'

  3. After you are done simulating the outage, recover from the failure by removing the metadata keys:

    gcloud compute project-info remove-metadata --keys failed_zone,failed_instance_names

Here are some ideas for failure scenarios you can run using this script:

  • Stop your application completely to see how the MIG responds.
  • Make your VMs return as "unhealthy" on load balancing health checks.
  • Modify iptables to block some of the traffic to and from the VM.
  • Shutdown the VMs. By default, it will be recreated by the regional MIG shortly after but the new incarnation will immediately shutdown itself as soon as the script runs and as long as the metadata values are set. This will result in a crash loop.

What's next