Some or all of the information on this page might not apply to Cloud de Confiance by S3NS. See Differences from Google Cloud for more details.

Troubleshoot Cloud Armor issues

Use these instructions to troubleshoot issues with Google Cloud Armor security policies.

General issues

Debugging security policies

If you need additional information about what particular event triggers preconfigured rules, read Using request logging, and then enable verbose logging. Cloud Logging records a higher level of detail in your logs that you can use to analyze and debug your policies and rules.

Traffic is allowed despite a deny rule configured in the Cloud Armor security policy

To fix this, follow these steps:

Make sure that the Cloud Armor security policy is attached to a target backend service. For example, the following command describes all data associated with the backend service BACKEND. The results returned must include the name of the Cloud Armor security policy associated with this backend service.
```
gcloud compute backend-services describe BACKEND
```
Replace BACKEND with the name of the backend service.

Review the HTTP(S) logs to determine which policy and rule were matched for your traffic along with the associated action. To view the logs, use Cloud Logging.

The following is a sample log of an allowed request with the interesting fields highlighted. Check for the following fields and make sure that they match the rule that you configured to deny the traffic:

configuredAction must match the action configured in the rule.
name must match the name of the Cloud Armor security policy attached to this backend service.
outcome must match configuredAction.
priority must match the priority number of the rule.

 httpRequest:
  remoteIp: 104.133.0.95
  requestMethod: GET
  requestSize: '801'
  requestUrl: http://74.125.67.38/
  responseSize: '246'
  serverIp: 10.132.0.4
  status: 200
  userAgent: curl/7.35.0
    insertId: ajvis5ev4i60
    internalId:
      projectNumber: '895280006100'
    jsonPayload:
      '@type': type.googleapis.com/google.cloud.loadbalancing.type.LoadBalancerLogEntry
      enforcedSecurityPolicy:
        configuredAction: ACCEPT
        name: mydev-policy-log-test1
        outcome: ACCEPT
        priority: 2147483647
      statusDetails: response_sent_by_backend
    logName: projects/mydev-staging/logs/requests
    resource:
      labels:
        backend_service_name: BACKEND_SERVICE_NAME
        forwarding_rule_name: FORWARDING_RULE_NAME
        project_id: PROJECT_ID
        target_proxy_name: TARGET_HTTP_PROXY_NAME
        url_map_name: URL_MAP_NAME
        zone: global
      type: http_load_balancer
    severity: INFO
    timestamp: '2017-04-18T18:57:05.845960288Z'

This output includes the following values:

BACKEND_SERVICE_NAME: the name of the backend service
FORWARDING_RULE_NAME: the name of the forwarding rule
PROJECT_ID: the ID of your project
TARGET_HTTP_PROXY_NAME: the name of the target HTTP proxy
URL_MAP_NAME: the name of the URL map

Review the hierarchy of rules to ensure that the correct rule is matched. It is possible that a higher priority rule with an allow action is matching your traffic. Use the describe command on the security-policies in the Google Cloud CLI to see the contents of the Cloud Armor security policy.

For example, the following example shows how a higher priority allow rule (at priority 100) matches traffic coming from the 1.2.3.4 IP address, preventing the lower priority deny rule (at priority 200) from triggering and blocking the traffic.

gcloud compute security-policies describe POLICY_NAME

Replace POLICY_NAME with the name of the security policy.

The output is similar to the following:

creationTimestamp: '2017-04-18T14:47:58.045-07:00
description: ''
fingerprint: Yu5spBjdoC0=
id: '2560355463394441057'
kind: compute#securityPolicy
name: POLICY_NAME
rules:
-action: allow
 description: allow high priority rule
 kind: compute#securityPolicyRule
 match:
   srcIpRanges:
   -'1.2.3.4/32'
 preview: false
 priority: 100
-action: deny
 description: deny lower priority rule
 kind: compute#securityPolicyRule
 match:
   srcIpRanges:
   -'1.2.3.0/24
 preview: false
 priority: 200
-action: deny
 description: default rule
 kind: compute#securityPolicyRule
 match:
   srcIpRanges:
   -'*'
 preview: false
 priority: 2147483647
 selfLink: http://www.googleapis.com/compute/v1/projects/bigclustertestdev0-devconsole/global/securityPolicies/sp

Preconfigured rule returns false positives

XSS and SQLi detection are based on static signature matching on HTTP request headers and other Layer 7 parameters. These regular expression patterns are prone to false positives. You can use the preconfigured rule for XSS and SQLi detection in preview mode and then check the log for any false positives.

If you find a false positive, you can compare the traffic content with the OWASP CRS rules. If you are migrating from an older CRS version, see the WAF rules version comparison for a list of rule changes and renames. If the rule is invalid or not relevant, disable it by using the evaluatePreconfiguredWaf expression, and specify the rule's ID in the exclude ID list argument.

After reviewing the logs and removing all false positives, disable the preview mode.

To add a preconfigured rule in preview mode:

Create a security policy with the preconfigured expression set in preview mode:

 gcloud compute security-policies rules create 1000
     --security-policy POLICY_NAME
     --expression "evaluatePreconfiguredWaf('xss-stable')"
     --action deny-403
     --preview

Replace POLICY_NAME with the name of the security policy.

Review the HTTP(S) logs for HTTP request fields such as url and cookie. For example, the requestUrl compares positively to the OWASP CRS rule ID 941180:

httpRequest:
 remoteIp: 104.133.0.95
 requestMethod: GET
 requestSize: '801'
 requestUrl: http://74.125.67.38/foo?document.cookie=1010"
 responseSize: '246'
 serverIp: 10.132.0.4
 status: 200
 userAgent: curl/7.35.0
insertId: ajvis5ev4i60
internalId:
 projectNumber: '895280006100'
jsonPayload:
 '@type': type.googleapis.com/google.cloud.loadbalancing.type.LoadBalancerLogEntry
 enforcedSecurityPolicy:
   configuredAction: ACCEPT
   name: POLICY_NAME
   outcome: ACCEPT
   priority: 2147483647
   preconfiguredExprIds: [ 'owasp-crs-v042200-id941180-xss' ]
 statusDetails: response_sent_by_backend
logName: projects/mydev-staging/logs/requests
resource:
 labels:
   backend_service_name: BACKEND_SERVICE
   forwarding_rule_name: mydev-forwarding-rule
   project_id: mydev-staging
   target_proxy_name: mydev-target-http-proxy
   url_map_name: mydev-url-map
   zone: global
 type: http_load_balancer
severity: INFO
timestamp: '2017-04-18T18:57:05.845960288Z'

This log includes the following values:

POLICY_NAME: the name of the security policy
BACKEND_SERVICE: the name of the backend service

Exclude the OWASP CRS rule ID 941180 by updating the rule in the Cloud Armor security policy:

 gcloud compute security-policies rules update 1000 \
     --security-policy POLICY_NAME \
     --expression "evaluatePreconfiguredWaf('xss-v422-stable', ['owasp-crs-v042200-id941180-xss'])" \
     --action deny-403 \
     --preview

Replace POLICY_NAME with the name of the security policy.

Review the logs again and then disable preview mode to implement the rule.

Clients with denied signatures aren't blocked or denied

If you are using Cloud Armor with Cloud CDN, security policies are enforced only for requests for dynamic content, cache misses, or other requests that are destined for the CDN origin server. Cache hits are served even if the downstream Cloud Armor security policy can prevent that request from reaching the CDN origin server.

Issues with rate limiting

Traffic isn't throttled as expected

You might find that a client IP address is sending high levels of traffic to an application at a rate that exceeds the threshold you set, but the traffic is not throttled as you expect. Follow these steps to investigate the issue.

First, confirm whether a higher-priority rule is allowing traffic from that IP address. Examine the logs to see whether an ALLOW rule was triggered for the IP address. This could be an ALLOW rule on its own, or in another THROTTLE or RATE_BASED_BAN rule.

If you find a higher-priority rule, do one of the following:

Change the priorities to ensure that the rate-limiting rule has a higher priority, by assigning it a lower numerical value.
Exclude the IP address from the matched expression in the rule that has a higher priority.

The issue might also be that the threshold is set incorrectly. If this is the case, the requests are matched accurately, but the conform action is triggered. Examine the logs to confirm that this is the case, then reduce the threshold in your rule.

Lastly, the IP address might not match the throttle or rate-based ban rule. To fix this, check for a mistake in the match condition, then change the rule's match condition to the correct value.

What's next

Get support