Some or all of the information on this page might not apply to Trusted Cloud by S3NS.

Troubleshooting Google Cloud Armor issues

Use these instructions to troubleshoot issues with Google Cloud Armor security policies.

General issues

Debugging security policies

If you need additional information about what particular event triggers preconfigured rules, read Using request logging, and then enable verbose logging. Cloud Logging records a higher level of detail in your logs that you can use to analyze and debug your policies and rules.

Traffic is allowed despite a deny rule configured in the Google Cloud Armor security policy

To fix this, follow these steps:

Make sure that the Google Cloud Armor security policy is attached to a target backend service. For example, the following command describes all data associated with the backend service BACKEND. The results returned should include the name of the Google Cloud Armor security policy associated with this backend service.
```
gcloud compute backend-services describe BACKEND
```

Review the HTTP(S) logs to determine which policy and rule were matched for your traffic along with the associated action. To view the logs, use Cloud Logging.

The following is a sample log of an allowed request with the interesting fields highlighted. Check for the following fields and make sure that they match the rule that you configured to deny the traffic:

configuredAction should match the action configured in the rule.
name should match the name of the Google Cloud Armor security policy attached to this backend service.
outcome should match configuredAction.
priority should match the priority number of the rule.

  httpRequest:
   remoteIp: 104.133.0.95
   requestMethod: GET
   requestSize: '801'
   requestUrl: http://74.125.67.38/
   responseSize: '246'
   serverIp: 10.132.0.4
   status: 200
   userAgent: curl/7.35.0
     insertId: ajvis5ev4i60
     internalId:
       projectNumber: '895280006100'
     jsonPayload:
       '@type': type.googleapis.com/google.cloud.loadbalancing.type.LoadBalancerLogEntry
       enforcedSecurityPolicy:
         configuredAction: ACCEPT
         name: mydev-policy-log-test1
         outcome: ACCEPT
         priority: 2147483647
       statusDetails: response_sent_by_backend
     logName: projects/mydev-staging/logs/requests
     resource:
       labels:
         backend_service_name: BACKEND_SERVICE_NAME
         forwarding_rule_name: FORWARDING_RULE_NAME
         project_id: PROJECT_ID
         target_proxy_name: TARGET_HTTP_PROXY_NAME
         url_map_name: URL_MAP_NAME
         zone: global
       type: http_load_balancer
     severity: INFO
     timestamp: '2017-04-18T18:57:05.845960288Z'

Review the hierarchy of rules to ensure that the correct rule is matched. It is possible that a higher priority rule with an allow action is matching your traffic. Use the describe command on the security-policies in the Google Cloud CLI to see the contents of the Google Cloud Armor security policy.

For example, the following example shows how a higher priority allow rule (at priority 100) matches traffic coming from the 1.2.3.4 IP address, preventing the lower priority deny rule (at priority 200) from triggering and blocking the traffic.

gcloud compute security-policies describe POLICY_NAME

Output:

  creationTimestamp: '2017-04-18T14:47:58.045-07:00
  description: ''
  fingerprint: Yu5spBjdoC0=
  id: '2560355463394441057'
  kind: compute#securityPolicy
  name: POLICY_NAME
  rules:
  -action: allow
   description: allow high priority rule
   kind: compute#securityPolicyRule
   match:
     srcIpRanges:
     -'1.2.3.4/32'
   preview: false
   priority: 100
  -action: deny
   description: deny lower priority rule
   kind: compute#securityPolicyRule
   match:
     srcIpRanges:
     -'1.2.3.0/24
   preview: false
   priority: 200
  -action: deny
   description: default rule
   kind: compute#securityPolicyRule
   match:
     srcIpRanges:
     -'*'
   preview: false
   priority: 2147483647
   selfLink: http://www.googleapis.com/compute/v1/projects/bigclustertestdev0-devconsole/global/securityPolicies/sp

Preconfigured rule returns false positives

XSS and SQLi detection are based on static signature matching on HTTP request headers and other L7 parameters. These regular expression patterns are prone to false positives. You can use the preconfigured rule for XSS and SQLi detection in preview mode and then check the log for any false positives.

If you find a false positive, you can compare the traffic content with the OWASP CRS rules. If the rule is invalid or not relevant, disable it by using the evaluatePreconfiguredWaf expression, and specify the rule's ID in the exclude ID list argument.

After reviewing the logs and removing all false positives, disable the preview mode.

To add a preconfigured rule in preview mode:

Create a security policy with the preconfigured expression set in preview mode:

gcloud compute security-policies rules create 1000
   --security-policy POLICY_NAME
   --expression "evaluatePreconfiguredWaf('xss-stable')"
   --action deny-403
   --preview

Review the HTTP(S) logs for HTTP request fields such as url and cookie. For example, the requestUrl compares positively to the OWASP CRS rule ID 941180:

httpRequest:
  remoteIp: 104.133.0.95
  requestMethod: GET
  requestSize: '801'
  requestUrl: http://74.125.67.38/foo?document.cookie=1010"
  responseSize: '246'
  serverIp: 10.132.0.4
  status: 200
  userAgent: curl/7.35.0
insertId: ajvis5ev4i60
internalId:
  projectNumber: '895280006100'
jsonPayload:
  '@type': type.googleapis.com/google.cloud.loadbalancing.type.LoadBalancerLogEntry
  enforcedSecurityPolicy:
    configuredAction: ACCEPT
    name: POLICY_NAME
    outcome: ACCEPT
    priority: 2147483647
    preconfiguredExprIds: [ 'owasp-crs-v030001-id941180-xss' ]
  statusDetails: response_sent_by_backend
logName: projects/mydev-staging/logs/requests
resource:
  labels:
    backend_service_name: BACKEND_SERVICE
    forwarding_rule_name: mydev-forwarding-rule
    project_id: mydev-staging
    target_proxy_name: mydev-target-http-proxy
    url_map_name: mydev-url-map
    zone: global
  type: http_load_balancer
severity: INFO
timestamp: '2017-04-18T18:57:05.845960288Z'

Exclude the OWASP CRS rule ID 941180 by updating the rule in the Google Cloud Armor security policy:

gcloud compute security-policies rules update 1000 \
    --security-policy POLICY_NAME \
    --expression "evaluatePreconfiguredWaf('xss-stable', ['owasp-crs-v030001-id941180-xss'])" \
    --action deny-403 \
    --preview

Review the logs again and then disable preview mode to implement the rule.

Clients with denied signatures are not blocked or denied

If you are using Google Cloud Armor with Cloud CDN, security policies are enforced only for requests for dynamic content, cache misses, or other requests that are destined for the CDN origin server. Cache hits are served even if the downstream Google Cloud Armor security policy would prevent that request from reaching the CDN origin server.

Mitigate risk on POST body that exceeds 8 KB when using preconfigured WAF rules

When a preconfigured WAF rule is evaluated in a Google Cloud Armor security policy, up to 8 KB of the POST body is inspected for signature matches against the WAF rules. This approach provides you with low latency layer 7 inspection and protection, while maintaining availability for other Google customers.

You can mitigate the risk from larger POST requests by creating a rule in your security policies to make sure that no uninspected content reaches your backends. Create a rule to deny traffic that exceeds 8 KB (8192 bytes) in POST body size. The following code sample shows how to create this rule:

gcloud compute security-policies rules create 10 \
    --security-policy my-policy \
    --expression "int(request.headers['content-length']) > 8192" \
    --action deny-403 \
    --description "Block requests great than 8KB"

Issues with rate limiting

Traffic is not throttled as expected

You might find that a client IP address is sending high levels of traffic to an application at a rate that exceeds the threshold you set, but the traffic is not throttled as you expect. Follow these steps to investigate the issue.

First, confirm whether a higher-priority rule is allowing traffic from that IP address. Examine the logs to see whether an ALLOW rule was triggered for the IP address. This could be an ALLOW rule on its own, or in another THROTTLE or RATE_BASED_BAN rule.

If you find a higher-priority rule, do one of the following:

Change the priorities to ensure that the rate-limiting rule has a higher priority, by assigning it a lower numerical value.
Exclude the IP address from the matched expression in the rule that has a higher priority.

The issue might also be that the threshold is set incorrectly. If this is the case, the requests are matched accurately, but the conform action is triggered. Examine the logs to confirm that this is the case, then reduce the threshold in your rule.

Lastly, the IP address might not match the throttle or rate-based ban rule. To fix this, check for a mistake in the match condition, then change the rule's match condition to the correct value.

What's next

Get support