Introduction to Cloud Run troubleshooting

This page describes the common troubleshooting strategies for Cloud Run errors. Personalized Service Health publishes all Cloud Run incidents that stem from the underlying Cloud de Confiance by S3NS infrastructure to identify Cloud de Confiance by S3NS service disruptions impacting your projects. You should also consider setting up alerts on Personalized Service Health events. For information about incidents affecting all Cloud de Confiance by S3NS services, see the Cloud de Confiance by S3NS Service Health dashboard.

See the following sections in the Cloud Run troubleshooting guide for resolving issues related to your Cloud Run resource:

Cloud Run troubleshooting strategies

The following sections explain how you can apply general troubleshooting strategies to resolve your error. If you continue to encounter errors even after following the steps in the troubleshooting guide, see What's next.

Output good logs using Cloud Logging

Troubleshooting your Cloud Run resource is easier if you have good logs for debugging. You should write logs in a way that correlates your container logs with a request log.

With correlated logs, you can identify the request that needs further analysis, find the request trace, and analyze the root cause of the issue. For more information on writing logs, see Write container logs.

Investigate instances using the Logs Explorer

Each request log in Cloud Run contains an instanceId field that identifies an instance that handles your request. Depending on the concurrency value you specify, a single instance can handle multiple requests at the same time.

When you have multiple instances emitting logs at once, you should filter your instances to identify the sequential requests that lead up to an instance crash.

Filtering an instance lets you debug specific performance issues related to cold starts or increased latencies. These issues could also be bound to variables declared in a global scope, when the value is reused in subsequent concurrent requests. An example of this is when you create a single connection pool global object for the instance, and then use it within multiple requests.

To filter a specific instance in the Logs Explorer, follow these steps:

  1. In the Cloud de Confiance console, go to the Logs Explorer page:

    Go to Logs Explorer

  2. Select an existing Cloud de Confiance by S3NS project at the top of the page, or create a new project.

  3. Select the resource Cloud Run Revision for a service, or Cloud Run Job for a job.

  4. Expand a log entry to filter by a specific instance.

  5. Click the instance ID value, and select Show matching entries.

While you investigate instances, you can use Gemini Cloud Assist Investigations to gain additional insights into your logs. For more information about different ways to initiate an investigation by using the Logs Explorer, see Troubleshoot issues with Gemini Cloud Assist Investigations in the Gemini documentation.

Resolve unexpected request latencies

If you encounter issues with latency, do the following:

  1. Check if the latency is affecting all requests to your Cloud Run resource or only a small percentage. Cloud Run is automatically integrated with Cloud Monitoring with no setup or configuration required.

    To see individual request latency metrics, follow these steps:

    1. In the Cloud de Confiance console, go to the Cloud Run page:

      Go to Cloud Run

    2. Select a service or a job from the lists available in left navigation.

    3. Click the METRICS tab to show the Request latencies dashboard.

    To view latency metrics in Cloud Monitoring, select from the Metrics list, Cloud Run Revision > Request_latencies > Request latency.

    For a list of all available Cloud Run metrics and more in-depth details, see Cloud de Confiance by S3NS metrics in Cloud Monitoring.

  2. Identify the request with high latency to understand the source of latency. You can use Cloud Trace or Cloud Logging to understand how long a particular request has taken.

    To identify requests with high latency using Cloud Logging, apply the traceSampled=true filter to correlate logs in Cloud Logging with traces in Cloud Trace. For more information, see Integrate with Cloud Logging.

    Sometimes dependencies such as requests to other services might cause latency issues. To identify such requests, you should have explicit logging targeting the requests. If you don't output such logs, it could appear as a latency issue originating from a Cloud Run service.

    Additionally, you should consider evaluating latency spikes in the context of the chosen time window. A spike's significance is relative; a large spike in a small window might be negligible in a larger window, and vice-versa. Therefore, the time window significantly impacts the interpretation of latency data.

  3. Try increasing the number of minimum instances to reduce latency for incoming requests, and avoid cold starts. You should also consider modifying your source code, and adjust the scaling settings to limit the number of connections to a backing service.

    For more information, see Optimizing performance.

Troubleshoot connectivity issues

If your Cloud Run service is experiencing connectivity issues, consider these strategies and tools to diagnose the problem:

  • PCAP sidecar: For deeper network-level analysis, deploy a PCAP sidecar alongside your Cloud Run service. This sidecar container performs a packet capture using tcpdump within the same network namespace. The sidecar decouples from the main ingress container and doesn't require any modifications to perform a packet capture. Sidecars also use their own resources, which prevents tcpdump from competing with the resources you allocate to the primary service.

  • Network intelligence and connectivity tests for Cloud Run revisions and Cloud Run functions: Perform automated checks on the network path between your Cloud Run resource and an endpoint. This helps you find misconfigurations that might block traffic to or from your Cloud Run resource when connecting to a VM instance, an IP address, or a Google-managed service.

  • Review logs for your Cloud Run resource: Logs show error messages about connection problems, such as failures, timeouts, or refused connections. These logs often reveal if the connection issue is with your application or the network.

What's next

If you can't find a solution to your problem in the Cloud Run documentation, follow these steps: